KR100922544B1 - Live video compositing system by using realtime camera tracking and its method - Google Patents

Abstract

The present invention relates to a system for real-time image synthesis using real-time camera tracking and a method thereof, and selects suitable feature points from two-dimensional feature tracking data to predict the position and angle of a camera in real time through a state predictor. The spatial coordinates of the feature points restored in three dimensions are compared with the urban model information of the DB to estimate the position and angle of the camera in the absolute coordinate system of the urban model space in real time. Can be synthesized and output to ensure accuracy and real-time.

Compositing, camera, motion, prediction

Description

Live video composite system and method using real time camera tracking {LIVE VIDEO COMPOSITING SYSTEM BY USING REALTIME CAMERA TRACKING AND ITS METHOD}

The present invention relates to a real-time camera tracking for predicting the position and rotation of the camera from the image input in a vehicle navigator in real time, and a system and method for synthesizing the image in real time using the same.

As is well known, the navigator shows the surrounding map or the 3D background model only on the graphic screen, and the path to which the driver should move is also shown on the graphic background screen.

This type of navigator is not a problem on highways and outer roads, but it can cause a momentary misunderstanding on the driver's side of the city road, where the roads and intersections are intricately intertwined. Therefore, if the real-time composite screen of the driver's moving direction and surrounding buildings is shown in real-time on the actual front screen instead of the graphic background screen, more accurate and intuitive information can be delivered to the driver. To do this, you need to know the exact position and angle of the camera that captures the live action, and the camera tracking technology that predicts it has been frequently used to synthesize computer graphics (CG) elements and live action images during the post production process of the film production site. come.

In other words, predicting camera movement using only video input is mainly done in non-real time during the post-production process, and it is known that foreign software such as Boujou and Matchmover perform well in this field. However, camera tracking needs to be done in real time in order to synthesize the live image in the navigation system. In the field of real-time camera tracking, Andrew J. Davidson and Ali Azarbayejani have recently proposed a real-time camera tracker with relatively good performance.

Their proposed real-time camera tracker is a state predictor such as Extended Kalman Filter (EKF) with input of two-dimensional tracking data of Natural Feature such as KLT (Kanade Lucas Tomasi, KLT). Estimate the variable values for the camera position and rotation using.

However, in order to implement a system that is driven in real time as in the prior art as described above, an appropriate number of feature points are selected to enable real-time operation by selecting excellent feature points that are less affected by noise among a plurality of feature point data that are automatically detected. The process of map management to keep the set in a state vector is essential, and the data on their proposed real-time camera tracker is not specifically suggested. There is a problem that you have to rely on.

Accordingly, the technical problem of the present invention is to solve the above-described problems, and selects the appropriate feature points from the 2D feature tracking data to predict the position and angle of the camera in real time through a state predictor. In addition, the spatial coordinates of the feature points in three dimensions are compared with the urban model information of the database (DataBase, hereinafter referred to as DB) to estimate the position and angle of the camera in the absolute coordinate system of the urban model space in real time. The present invention provides a real-time image synthesis system and a method using real-time camera tracking that can output a guide and the surrounding information in real-time with the real-time image.

In the photo-realistic image synthesis system using real-time camera tracking according to an aspect of the present invention, only the feature points for which the three-dimensional position is reconstructed among the feature points for the image is a cone generated by connecting the inner boundary of the area on each image plane from the center of the two cameras Feature point prediction and selection unit that selects through the radial inside of the shape, and the extended Kalman Filter (EKF) that inputs two-dimensional feature point tracking data for the selected feature points to predict camera movement information. Compares and matches the camera motion prediction unit, the current position of the camera, and the background geometry model of the surrounding space taken from the urban background geometry model database (DB) to determine the position and angle of the camera calculated in local coordinates. Matching unit for converting to a value in World Coordinate and predicted camera motion information It is characterized in that it comprises a synthesis unit for synthesizing and outputting the actual information of the road guidance and the surrounding information taken from the map and the surrounding information DB.

In addition, according to another aspect of the present invention, the method for synthesizing a real image using real-time camera tracking generates only the feature points whose three-dimensional positions are reconstructed among the feature points of the image by successive boundary lines in the regions on the image planes from two camera centers. Predicting the camera motion information through an extended Kalman filter that inputs two-dimensional feature tracking data for the selected feature points, and selecting the image through the conical radial inner space. Comparing and matching the current position of the camera with the background geometry model of the surrounding space taken from the urban background geometry model database (DB) and converting the position and angle of the camera calculated from the local coordinates into values in world coordinates, and predicting Guidance and surrounding information from the recorded camera movement information from the map and surrounding information DB Characterized in that it comprises the steps of synthesizing and outputting the image.

The present invention selects suitable feature points from the 2D feature point tracking data and predicts the position and angle of the camera in real time through a state predictor. The spatial coordinates of the feature points restored in three dimensions are compared with the urban model information of the DB to estimate the position and angle of the camera in the absolute coordinate system of the urban model space in real time. Synthesis and output in real time has the effect of ensuring accuracy and real-time.

Hereinafter, with reference to the accompanying drawings will be described in detail the operating principle of the present invention. In the following description of the present invention, when it is determined that a detailed description of a known function or configuration may unnecessarily obscure the subject matter of the present invention, the detailed description thereof will be omitted. Terms to be described later are terms defined in consideration of functions in the present invention, and may be changed according to intentions or customs of users or operators. Therefore, the definition should be made based on the contents throughout the specification.

1 is a block diagram for a real-time image synthesis system using real-time camera tracking according to an embodiment of the present invention, the camera initial position predictor 11, the camera motion predictor 13 and the feature point extractor 15 And buffer 17, feature point prediction and screening unit 19, matching unit 21, urban background geometric model DB 23, synthesis unit 25, map and surrounding information DB 27 and output unit 29 It includes.

The camera initial position predictor 11 selects any M of the feature points that are consistently observed during the first appropriate interval N ₁ frame, and performs <Equation 1>.

Where the angled bracket at the top of the vector is the predicted value, x _v is the vector containing the camera's movement, i.e. camera movement, rotation and acceleration, and y _i is the x in the first appearance frame of the i th feature point. where y, y, and depth z are elements, r, q, v, and w are the vectors of movement, rotation, and acceleration components, respectively, where W at the top right is the translation in the World coordinate system, and WR is the frame coordinate system. A transformation that returns to the World coordinate system, where R is the transformation relative to the camera axis in that frame.)

Camera movement by predicting the initial position of a camera (for example, a camera means a camera for a navigation system attached to a vehicle and knowing the internal variable information of the camera) by inputting as a component of a state vector of the camera. The prediction unit 13 and the matching unit 21 are provided respectively. In this case, x and y coordinates in y _i are coordinates in the frame where each feature point first appears, and depth z has no prior information. In addition, it is preferable to repeatedly perform the above-described EKF for N ₂ frames to provide the camera motion predictor 13 with the initial position of the predicted camera only when the depth information z converges.

The camera motion predictor 13 includes a state vector updater 13a, a predictor 13b, a calculator 13c, and an updater 13d, as shown in FIG.

The state vector updater 13a selects the initial position of the camera input from the camera initial position predictor 11 and the two-dimensional feature point tracking data input from the feature point predictor and selector 19 as input. The feature points are used to maintain appropriate level feature points in a state vector, update the feature points off the screen and newly added feature points to the prediction unit 13b, and update them from the update unit 13d. The input final predicted value is arranged and arranged in the state vector and provided to the synthesis unit 25.

The predictor 13b predicts the state vector of the next frame from the current frame using the EKF and provides it to the calculator 13c.

The calculation unit 13c calculates the state vector predicted by the prediction unit 13b as Kalman Gain based on prior knowledge of the process model and provides it to the updater 13d.

The updating unit 13d is represented by Equation 2

은 예측 메트릭스(prediction matrix)이며, K(t)는 Kalman Gain이며, y(t)는 현재 프레임에서의 관찰치이며, h(x(t|t-1))은 이전 프레임에서 예측된 특징점의 3차원 좌표를 현재 프레임의 카메라 사영행렬을 통해 사영시켰을 때 예측되는 좌표를 의미한다.)(Where x (t | t-1) is the state vector predicted in the previous frame,

Is the prediction matrix, K (t) is the Kalman Gain, y (t) is the observation in the current frame, and h ( x (t | t-1)) is 3 of the predicted feature points in the previous frame. The coordinates predicted when projecting the dimensional coordinates through the camera projection matrix of the current frame.)

As described above, the predicted state vector and the calculated value are combined to be updated to the final predicted value and provided to the state vector updater 13a.

The feature point extractor 15 extracts natural features, such as KLT, in real time with respect to the input of the image S1 and provides them to the buffer 17 and the matching unit 21, respectively.

Buffer 17 provides the only feature point to be observed sikimyeo accumulate data during a predetermined period of the _third frame N feature points input from the feature point extraction unit (15), N ₃ or more frames consistent with the feature points and the prediction selection unit (19).

The feature point predictor and selector 19 selects and provides only the feature points whose 3D position is relatively accurately reconstructed among the feature points filtered by the buffer 17 to the camera motion predictor 13.

For example, feature point selection can be understood through FIG. 4. That is, FIG. 4 is a diagram for explaining a scale for evaluating a 3D reconstruction result of a feature point in order to select a feature point to be used for camera motion prediction among 2D feature point tracking data.

Referring to FIG. 4, C ₁ is the center of the camera in the jth frame in which the feature point first appeared, and C ₂ is the center of the camera when the N ₃ frame passes and the feature point finishes the waiting period in the buffer 17. The squares at the bottom left and right represent the image plane at this time, and the dotted lines dotted by the inverted lines after the feature center coordinates observed on the camera center and the image plane in each frame are p 1 and p 2 to be. Here, p 1 and p 2 do not generally intersect due to noise, but by using the Mid Point Algorithm, a coordinate X _i that is an optimal solution in space can be obtained.

By reprojecting the calculated X _i with the predicted camera projection matrix in each frame, we can calculate the RPE (Reprojection Error), which is an error from the actual observation, in the N ₃ interval of the calculated RPE. Calculate the average value and draw the areas R ₁ and R ₂ with this value as the radius. In this case, the centers of R ₁ and R ₂ are the observation coordinates of the feature points obtained through two-dimensional tracking. If the feature point is noisy, the area around the observation where the position may exist can be understood as the area of R ₁ and R ₂ when considering the ground truth that should be actually seen.

Now _, when we connect the boundary lines in the areas R ₁ and R ₂ on the image plane from the camera centers C _{1 and} C ₂ , we can think of conical radial spaces such as S ₁ and S ₂ .

And, <Equation 3>

(Wherein, S ₃ is smaller the more the feature points become narrower can uncertainty region that is present in space the feature points can be considered to be relatively accurately restore the three-dimensional feature point. In such a case two cone S ₁ RPE is small, The width of S ₂ should be narrow and the movement and rotation of the camera moved during the frame of N ₃ should be large. It is desirable to use the volume of S ₃ as a measure of the accuracy of three-dimensional reconstruction of the feature point.)

We can predict the area S ₃ where this feature point is expected to exist in space, and the volume of this space can be used as a measure to compare how accurately the three-dimensional position of the feature point is predicted.

The matching unit 21 is a natural position of the initial position of the camera input from the camera initial position predictor 11 and the natural feature of the image (S1) input from the feature point extractor 15 and the GPS (Global Positioning System) The position of the camera calculated in local coordinates by comparing and matching an approximate current position of the camera received through the receiver S2 with a background geometric model of the surrounding space taken from the urban background geometric model DB 23. And converts the angle into a value in world coordinates and provides it to the combining unit 25.

The synthesizer 25 includes a live image synthesizer 25a and a state vector updater 25b as shown in FIG. 3.

The real picture synthesizing unit 25a is obtained from the map and the surrounding information DB 27 by using the camera motion information which is the final predicted value input from the camera motion predicting unit 13 and the feature points arranged by the state vector updating unit 25b. The synthesized image is provided to the output unit 29 by synthesizing the real-time image of the road guide and the surrounding information, and the real image synthesizing unit 25a may collect natural features of the image S1 input from the feature point extractor 15. The state vector updater 25b is provided.

Based on the matching result of the matching unit 21, the state vector updating unit 25b arranges the feature points that have been left out of the screen and the newly added feature points in the state vector, and repeats the real image synthesizing unit 25a repeatedly until the system is terminated. ), And the state vector updater 25b arranges out-of-screen feature points and newly added feature points in the state vector based on the motion prediction result of the camera motion predictor 13 and continues until the system is terminated. It is repeatedly provided to the live image synthesizer 25a.

The output unit 29 outputs the synthesized result image input from the combiner 29 in real time so that a user can check it as shown in FIG. 5. That is, as shown in the synthesized result image as shown in FIG. 5, it can be seen that the path to be moved on the live screen shown in front of the present, the name of the surrounding buildings, the occupant information, etc. are overlapped for the actual object.

Accordingly, the present invention selects suitable feature points from the 2D feature point tracking data and predicts the position and angle of the camera in real time through the state predictor. The spatial coordinates of the feature points restored in three dimensions are compared with the urban model information of the DB to estimate the position and angle of the camera in the absolute coordinate system of the urban model space in real time. Can be synthesized and output.

Meanwhile, in the detailed description of the present invention, specific embodiments have been described, but various modifications are possible without departing from the scope of the present invention. Therefore, the scope of the present invention should not be limited to the described embodiments, but should be defined not only by the scope of the following claims, but also by those equivalent to the scope of the claims.

1 is a block diagram for a real-time image synthesis system using real-time camera tracking according to an embodiment of the present invention,

2 is a block diagram illustrating in detail the camera motion predictor shown in FIG. 1;

3 is a block diagram illustrating in detail the synthesis unit shown in FIG.

4 is a view for explaining a scale for evaluating a 3D reconstruction result of a feature point in order to select a feature point to be used for camera motion prediction among 2D feature point tracking data according to the present invention;

FIG. 5 is a view illustrating a result image synthesized by the synthesis unit illustrated in FIG.

<Description of the symbols for the main parts of the drawings>

11: Camera initial position predictor 13: Camera motion predictor

15: feature point extraction unit 17: buffer

19: feature point prediction and selection unit 21: matching unit

23: urban background geometric model DB 25: synthesis

27: Map and surrounding information DB 29: Output

Claims (8)

A feature point prediction and selection unit that selects only the feature points whose 3D positions are reconstructed among the image points from the two camera centers through conical radial spaces created by connecting the inner boundary of the region on each image plane; A camera motion predictor for predicting camera motion information through an extended Kalman filter (EKF) for inputting two-dimensional feature point tracking data to the selected feature points; Compares and matches the current position of the camera with the background geometry model of the surrounding space taken from the urban background geometry model database (DB) to determine the position and angle of the camera calculated in local coordinates in world coordinates. A matching unit for converting to Synthesis unit for synthesizing the predicted camera motion information into a live image of the road guidance and surrounding information obtained from the map and the surrounding information DB Photorealistic image synthesis system using real-time camera tracking, including. The method of claim 1, The camera motion predictor, A state vector updating unit which maintains a proper level of feature point variables in a state vector by using the selected feature points and updates the feature points off the screen and newly added feature points; A prediction unit for predicting a state vector of the next frame in the current frame; A calculation unit configured to calculate the predicted state vector using a Kalman gain, An update unit for combining the predicted state vector with the calculated value and updating the final predicted value Photorealistic image synthesis system using real-time camera tracking, including. The method of claim 2, The update unit, Equation

(Where x (t | t-1) is the state vector predicted in the previous frame,

Is the prediction matrix, K (t) is the Kalman Gain, y (t) is the observation in the current frame, and h ( x (t | t-1)) is 3 of the predicted feature points in the previous frame. The coordinates predicted when projecting the dimensional coordinates through the camera projection matrix of the current frame.) Real-time image synthesis system using real-time camera tracking, characterized in that for updating to the final prediction value using. delete The method of claim 1, The system, A camera initial position predictor for predicting an initial position of the camera; A feature point extraction unit for extracting natural features in real time with respect to an image input; Wherein the extracted feature points predetermined length N ₃ sikimyeo accumulate data for a frame buffer to provide only the frame N ₃ or more consistently observed feature points in the feature point selection unit and the prediction Photorealistic image synthesis system using real-time camera tracking further comprising. The method of claim 1, And a current position of the camera is received by a GPS (Global Positioning System). Selecting only the feature points from which the three-dimensional positions of the image points are reconstructed from the two camera centers through conical radial spaces created by connecting the inner boundary of the region on each image plan; Predicting camera motion information through an extended Kalman filter that inputs two-dimensional feature point tracking data to the selected feature points; Comparing and matching the current position of the camera with the background geometry model of the surrounding space obtained from the urban background geometry model database (DB) to convert the position and angle of the camera calculated from the local coordinates into values in the world coordinates; Synthesizing the predicted camera motion information into a live image of a road guide and surrounding information obtained from a map and a surrounding information DB; Photorealistic image synthesis method using real-time camera tracking comprising a. The method of claim 7, wherein The feature points, When connecting the boundary line within the area on each image plane from the center of the two cameras, it forms a conical radial inner space like S ₁ , S ₂ , this is Equation

(In this case, the smaller S ₃ , the narrower the area of uncertainty that this feature may exist in space, and this feature can be regarded as a relatively precise three-dimensional reconstructed feature point, and the RPE (Reprojection Error) is small, resulting in two cones (S _1). , S ₂ ) should be narrow, the camera moved and rotated during the frame period should be large, and the volume of S ₃ is used as a measure of the accuracy of 3D reconstruction of the feature point. Real-time image synthesis method using real-time camera tracking, characterized in that the screening by using.

Images