KR20130084849A - Method and apparatus for camera tracking - Google Patents

Abstract

PURPOSE: A method and an apparatus for tracing the location of a camera are provided to trace first features within multiple frames and to trace second features within a single frame based on the traced first features. CONSTITUTION: A feature extracting unit extracts third features from at least three images in a first frame among multiple frames (410). A feature tracing unit traces the extracted third features to the last frame among the multiple frames (420). A dynamic point detector removes features having a dynamic trace among the traced third features and determines first features (430). A camera location estimating unit estimates the location of each camera within each frame based on the first features (440). [Reference numerals] (410) Extract features from a first frame among multiple frames; (420) Trace the extracted features to the last frame among the multiple frames; (430) Remove features having a dynamic trace among the traced features; (440) Estimate the location of each camera based on remaining features after the removal; (AA) Start; (BB) End

Description

METHOD AND APPARATUS FOR CAMERA TRACKING}

The following embodiments relate to a method and apparatus for tracking the position of a camera.

A method and apparatus are disclosed for tracking the location of each of the cameras based on frames taken by at least three cameras.

Camera tracking is a fundamental problem in computer vision. The purpose of camera tracking may be to automatically recover the movement of the camera from a video sequence. The basic idea of camera tracking is to select scene points that appear within a sequence of frames, and select a three-dimensional dimension of selected scene points based on a set of correspondences of 2D feature points; D) Estimate the position and camera movement at the same time.

There are many applications for camera tracking, such as depth recovery, 3D reconstruction, location recognition and autonomous robot navigation.

In particular, with the widespread use of digital cameras, monocular cameras are already readily accessible, and the price of monocular cameras is becoming lower and lower. Thus, methods for tracking monocular cameras are widely used.

However, three-dimensional information of a dynamic object cannot be restored from an image photographed by the monocular camera. In addition, due to an accumulation error, it is difficult to accurately reconstruct camera motion for large-scale scenes using images captured by a monocular camera.

Several methods have been proposed for reconstructing camera motion and depth maps of an image using stereo cameras. However, when images are taken by a stereo camera, it may be difficult to deal with the occlusion in the image.

An embodiment may provide a method and apparatus for extracting features within the frames based on frames taken by at least three cameras and estimating the position of each of the cameras based on the extracted features. have.

One embodiment provides a method and apparatus for tracking first features in multi-frames and tracking second features corresponding to the first feature in a single-frame based on the tracked first features. can do.

According to one side, a method for tracking the location of the cameras using frames taken by at least three cameras, the method comprising: extracting and tracking one or more first features within multi-frames, the first feature Tracking the location of each of the cameras within each of the multi-frames based on the plurality of frames and within each of the one or more single-frames based on second features of each of the one or more single-frames. Tracking a location of each of the cameras, wherein the one or more single-frames are tracked with less than a threshold second features corresponding to one of the first features of frames subsequent to the multi-frames. A camera positioning method, which is the previous frames of the first frame, is provided.

Tracking the location of each of the cameras within each of the multi-frames comprises: extracting third features from at least three images of the first of the multi-frames, the third features; Tracking to the last frame of the multi-frames, determining the first features by removing features with a dynamic trajectory of third features tracked to the last frame and based on the first features Estimating a position of each of the cameras within each of the multi-frames.

Extracting the third features may include extracting points from at least three images of the first frame, generating Scale Invariant Feature Transform Descriptors (SIFTs), and extracting the extracted points. Matching each other using descriptor comparisons between the generated SIFTs, and generating the third features by concatenating matched points among the points as features.

Extracting the third features may further include removing outliers of the third features using geometric constraints.

The geometric constraint may be one or more of epipolar constraint, re-projection constraint and depth range constraint.

Tracking the location of each of the cameras within each of the single-frames comprises: setting a next frame of the multi-frames as a current frame, a feature of one of the first features within the current frame Extracting the second features corresponding to the step of estimating the position of each of the cameras in the current frame when the number of the second features is greater than or equal to the threshold and the current when the number of the second features is greater than or equal to the threshold. And setting the next frame of the frame as the current frame and extracting the second features.

Tracking the location of each of the cameras within each of the single-frames may include tracking the location of each of the cameras in each of the multi-frames if the number of the second features is less than a threshold. The method may further include performing the step again.

According to another aspect, a camera position tracking device for tracking the position of the cameras using frames taken by at least three cameras, the camera position tracking device, extracting and tracking one or more first features in the multi-frames, A multi-frame processor for tracking the position of each of the cameras within each of the multi-frames based on the first features and the single-based based on the second features of each of the one or more single-frames. A single-frame processor for tracking the position of each of the cameras within each of the frames, wherein the one or more single-frames correspond to one of the first features of frames subsequent to the multi-frames. A camera position tracking device is provided, wherein the second features are previous frames of the first frame tracked less than the threshold.

The multi-frames processor may include a feature extractor configured to extract third features from at least three images of the first frame of the multi-frames, and to track the third features to the last frame of the multi-frames. A feature tracker, a dynamic point detector that determines the first features by removing features having a dynamic trajectory among the third features tracked to the last frame, and within each of the multi-frames based on the first features It may include a camera position estimator for estimating the position of each of the cameras.

The dynamic point detector may calculate a four-dimensional trajectory subspace of each of the third features, and determine whether each of the third features has a dynamic trajectory based on the four-dimensional trajectory subspace.

Extract points from at least three images of the first frame, generate Scale Invariant Feature Transform Descriptors (SIFTs), and compare the extracted points to each other using a descriptor comparison between the generated SIFTs. The third features may be generated by concatenating a match between the two points and associating matched points among the points as features.

The feature extractor may determine the first features by removing outliers of the third features using a geometric constraint.

The single-frame processing unit may include a current frame setting unit configured to set a next frame of the multi-frames as a current frame, and extract the second features corresponding to one of the first features within the current frame. The frame feature estimator and the second feature may include a threshold comparison unit for estimating the position of each of the cameras in the current frame.

When the number of the second features is greater than or equal to the threshold, the current frame setting unit may newly set a next frame of the current frame as the current frame, and the current frame feature estimating unit may set one of the first features from the newly set current frame. The second features corresponding to the features may be extracted.

If the number of the second features is less than a threshold, the multi-frames processor may be re-executed.

A method and apparatus are provided for tracking first features in multi-frames and tracking second features corresponding to the first feature in a single-frame based on the tracked first features.

A method and apparatus are provided for extracting features within the frames based on frames taken by at least three cameras and for estimating the location of each of the cameras based on the extracted features.

1 illustrates an operation of a camera tracking apparatus according to an embodiment.
2 is a structural diagram of a camera tracking device according to an embodiment.
3 is a flowchart of a camera location tracking method, according to an embodiment.
4 is a flowchart of a multi-frames processing step according to an example.
5 illustrates a feature extraction step according to an example.
6 is a flowchart of a feature tracking step according to an example.
7 is a flowchart of a dynamic point detection step according to an example.
8 is a flowchart of a camera position estimation according to an example.
9 is a flowchart of a single-frame processing step according to an example.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. However, the present invention is not limited to or limited by the embodiments. Like reference symbols in the drawings denote like elements.

The three-eye camera may include three synchronized cameras. The three cameras can be configured collinear.

In the following, a sequence may refer to a series of images generated by continuously capturing a scene. The three-eye sequence may refer to three sequences generated by three cameras in the three-eye camera, each successively capturing a scene.

The three-dimensional position M of the scene point may be represented by [ X , Y , Z ] ^T. Here, X , Y and Z may represent the X coordinate, the Y coordinate and the Z coordinate of the scene point, respectively. The two-dimensional image position m of the scene point may be represented by [ u , v ,] ^T.

로 모델링될 수 있다. 여기서, K는 고유 행렬(intrinsic matrix)일 수 있다,

는 투영 행렬(projective matrix)일 수 있다.Camera transformation

Can be modeled as: Here, K may be an intrinsic matrix,

May be a projective matrix.

The eigen matrix may depend on the inherent properties of the camera. The inherent properties of the camera include 1) focal lengths in pixels along the x and y axes of the image, 2) tilt ratio between the x and y axes, and 3) principal points. It can encompass. Here, the principal point may refer to the projection of the camera center to the image plane.

로서 표현될 수 있다. 여기서,

는 스케일 팩터(scale factor)일 수 있다. 운동으로부터 구조의 출현(Structure-from-Motion)은 이미지 측정치들인 m'들로부터 모든 M들 및

을 추정하는 것을 의미할 수 있다.The cross product matrix can model the camera movement including the rotation R and the transformation t . Thus, the projection procedure

. ≪ / RTI > here,

May be a scale factor. Structure-from-Motion emerges from all the Ms from the image measurements m 'and

May mean estimating.

If the selected scene points are all static, the above estimate is very reliable. However, when the scene includes some moving objects, the above estimates can be seriously confused because some points do not have only one 3D position during capture. In order to determine whether a point is static, it can generally be checked whether all pairs of 2D image positions corresponding to the point satisfy the epipolar geometry.

If a stereo rig is available, the 3D positions of the matched points can be calculated more reliably through triangulation. The two images belonging to the same frame and the relative position between the left view and the right view can be derived directly from the stereo rig.

Information about the 3D position can also be useful for tracking features. Tracking the characteristic may be to connect the points of the image corresponding to the same scene point.

1 illustrates an operation of a camera tracking apparatus according to an embodiment.

When a trinocular camera is used that can take a left image, an intermediate image and a right image, it is rare that certain pixels in the intermediate image occlude in both the left and right images.

In addition, when a three-eye camera is used, at each timestamp, three images can be obtained. Thus, based on the three images, to calculate the 3D position of tracked features and to remove outliers, it is more robust than the epipolar geometry. Geometric constraints may be used.

In the configuration of a three-eye camera, the relative positions of each of the three cameras in the three-eye camera may be considered fixed. It can also be assumed that the intrinsic parameters of the three cameras do not change while capturing the image. Thus, input values to the camera positioning apparatus 100 may include not only sequences in three eyes, but also unique matrices and relative camera positions for the three cameras. In addition, the output values of the camera position tracking apparatus 100 may be an extrinsic matrix for each image. Here, the cross product matrix may include a rotation matrix and a translation vector.

및

라면, 상기의 2 개의 카메라들 간의 상대적 위치

는 하기의 수학식 1에 기반하여 계산될 수 있다.The unique parameters and positions of the three cameras can be calibrated by a tool, such as the Camera Calibration Toolbox for Matlab. Each of the two cameras has an extrinsic camera parameter

And

If so, the relative position between the two cameras

May be calculated based on Equation 1 below.

및 좌측 카메라로부터 우측 카메라로의 투영 행렬

이 카메라 위치 추적 장치(100)의 입력 파라미터로서 사용될 수 있다.The three-eye sequence may be input to the camera position tracking apparatus 100. In addition, the unique matrices K _left , K _middle and K _right of each of the three cameras of the _trinocular camera may be used as input parameters of the camera position tracking apparatus 100. Also, of the three cameras, the projection matrix from the left camera to the intermediate camera

And projection matrix from left camera to right camera

This may be used as an input parameter of the camera position tracking device 100.

Once the unique parameters and relative positions of the three cameras have been calculated, the camera position tracking device 100 performs a matched feature point between the images captured by the three cameras via triangulation. It is possible to estimate 3D points of feature points.

Also, once the 3D points of the tracked features are calculated, the camera position tracking device 100 can calculate the positions of the camera reliably and rapidly based on the correspondence between these 3D and 2D.

는 첫 번째 프레임 내에서의 좌측 카메라의 위치를 나타낼 수 있다.

는 마지막 프레임인 F 번째 프레임 내에서의 우측 카메라의 위치를 나타낼 수 있다.
When the three-eye sequence includes F frames, the camera position tracking apparatus 100 may output the positions of the three cameras for each frame. for example,

May represent the position of the left camera within the first frame.

May indicate the position of the right camera within the F-th frame, which is the last frame.

2 is a structural diagram of a camera tracking device according to an embodiment.

The camera position tracking apparatus 100 tracks the positions of the cameras using frames photographed by at least three cameras.

The camera position tracking apparatus 100 may include a multi-frame processor 210 and a single-frame processor 220.

The multi-frames processor 210 can extract and track one or more first features within the multi-frames, and track the location of each of the cameras within each of the multi-frames based on the first features. can do. The multi-frames processor 210 may provide information about the first features to the single-frame processor 220.

The single-frame processor 220 may track the location of each of the cameras within each of the single-frames based on the second features of each of the one or more single-frames. Here, the one or more single-frames may be previous frames of the first frame in which second features corresponding to one of the first features of the frames subsequent to the multi-frames are tracked less than the threshold.

In other words, if the multi-frames processor 210 extracts and tracks the first features within the multi-frames, the single-frames processor 220 may follow the subsequent processing of the frames processed by the multi-frames processor 210. You can process the frames one by one. Here, the processing of the frame may mean tracking features in the frame and tracking the position of each of the cameras in the frame based on the tracked features. That is, if the first features are provided from the multi-frames processor 210 to the single-frame processor 220, then in a subsequent frame of the frames processed by the single-frame processor 220 multi-frames processor 210. The second features can be tracked. Here, each of the second features is a feature corresponding to one of the first features above. Alternatively, each of the second features tracked in the current frame processed by the single-frame processor 220 may be one of the second features tracked within the previous frame processed by the single-frame processor 220 previously. It may be a feature corresponding to one feature.

The number of second features tracked by the single-frame processor 220 may be less than the number of first features. For example, if a scene point corresponding to one of the first features disappears in the sequence of frames, the second feature corresponding to the particular first feature may not be tracked. As the single-frame processor 220 processes the subsequent frames of the frames processed by the multi-frames processor 210 one by one, when the number of tracked second features drops below the threshold, the single-frame processor 220 Tracking the location of each of the cameras may be inappropriate. Thus, if a frame whose second features are tracked less than the threshold of frames subsequent to the multi-frames is found, then the multi-frames including the found frame are again processed by the multi-frames processing unit 210. Can be.

That is, the multi-frames processing unit 210 and the single-frame processing unit 220 may process a sequence of frames alternately with each other. The multi-frames processing unit 210 and the single-frame processing unit 220 may provide each other with information (eg, a frame number) for identifying a frame to be processed.

Since the multi-frames processing unit 210 extracts the first features common in the multi-frames using the multi-frames, the multi-frames processing unit 210 can accurately extract the first features. Since the single-frame processor 220 extracts the second features corresponding to the first features in one frame, the single-frame processor 220 may quickly extract the second features. That is, the multi-frames processing unit 210 and the single-frame processing unit 220 are alternately executed, so that the camera position tracking apparatus 100 may balance accuracy and speed in tracking the position of the camera.

The multi-frames processor 210 may include a feature extractor 211, a feature tracker 212, a dynamic point detector 213, and a camera position estimator 214.

The single-frame processor 220 may include a current frame setter 221, a current frame feature estimator 222, a current frame threshold comparator 223, and a current frame camera position estimator 224.

Specific functions and operating principles of each component will be described in detail with reference to FIGS. 3 to 9 below.

3 is a flowchart of a camera location tracking method, according to an embodiment.

In the following steps 310 and 320, the camera position tracking device 200 uses a subsequence of frames taken by at least three cameras to determine the position of each of the cameras within each frame. Can be traced At least three cameras may be a left camera, an intermediate camera and a right camera, respectively. Each frame may include a left image captured by the left camera, an intermediate image captured by the intermediate camera, and a right image captured by the right camera.

In the multi-frames processing step 310, the multi-frames processor 210 may extract and track one or more first features within the multiple frames. Here, the multi-frames may be two or more consecutive frames. The multi-frames processor 210 may track the position of the cameras in each of the multi-frames based on the tracked first features. Here, the number of multi-frames may be predefined. Hereinafter, the number of multi-frames is denoted by N _f . That is, the multi-frames processor 210 may extract, triangulate, and track features common to the multi-frames composed of N _f frames. The multi-frames processor 210 may detect and remove features corresponding to dynamic scene points among the extracted features. That is, the first features may correspond to N _p static first points successfully tracked up to the N _f th frame of the multi-frame among the features extracted by the multi-frames processing unit 210. The frames processing unit 210 may simultaneously estimate 3N _f camera positions and 3N _p first points positions in the N _f frames using the first features.

In the single-frame tracking step 320, the single-frame processing unit 220 can track the position of each of the cameras in each of the single-frames based on the second features of each of the one or more single-frames. have. Here, the one or more single-frames may be previous frames of the first frame in which the second features corresponding to one of the first features of the frames subsequent to the multiple frames are tracked less than the threshold.

The single-frame processor 220 may search for matching second features with respect to the aforementioned N _p first features within the N _f + 1 th frame, which is the next frame of the multi-frames. Here, the number of second feature is detected match is N _'p. The single-frame processor 220 may acquire 3D positions of the second features. Next, the single-frame processor 220 may estimate the position of each of the cameras in the N _f + 1 th frame based on the correspondence between 2D and 3D. Next, the single-frame processor 220 may search for matching third features for N ′ _p second features within the N _f + 2 th frame. Each of the second features may match one of the first features. Thus, each of the third features may match one of the first features. This procedure may be repeated until the number of features matching the first feature retrieved in the N ′ + n th frame by the single-frame processor 220 goes below the threshold.

The single-frame processing step 320 may be performed much faster than the multi-frames processing step 310. In the single-frame processing step 320, in order to allow more frames to be processed, the single-frame processing unit 220 displays N _f of multiple frames that do not match in a single-frame of the first features. You can project to the first frame. The single-frame processor 320 may compare the local appearance between the points generated by the projection and the features of the window.

After the single-frame processing step 320, the multi-frames processing step 310 may be restarted again. In the restarted multi-frames estimating step 310, the multi-frames processor 210 may re-extract new first features of the N ′ _f frames. In the single-frame processing step 320, the single-frame processing unit 220 may find new second features that match the first features in successive frames.

This two-step procedure can be repeated until all the frames have been processed.

By the two-step procedure described above, features can be extracted and tracked within a substring of frames. The multi-frames processing unit 210 and the single-frame processing unit 220 may automatically remove features representing dynamic points, respectively, and determine the position of each of the cameras in each frame after the removal. Can be used to estimate

By the two-step procedure described above, the camera movement of the input trinocular sequence can be robustly and efficiently recovered, and the tracking efficiency can be improved without reducing the accuracy of the tracking.

A two-step procedure comprising a multi-frame processing step 310 and a single-frame processing step 320 is described in detail with reference to FIGS. 4-9 below.

4 is a flowchart of a multi-frames processing step according to an example.

The multi-frames processing step 310 may include the following steps 410-440.

In the feature extraction step 410, the feature extractor 211 may extract third features from at least three images of the first frame of the multi-frames. Specific operations of the feature extraction step 410 will be described in detail with reference to FIG. 5 below.

In the feature tracking step 420, the feature tracker 212 tracks the extracted third features to the last frame of the multi-frames.

The traditional feature tracking method extracts feature points in the first frame and tracks the extracted feature points frame by frame. If, in a three-eye camera configuration, each feature is tracked from the previous frame to the current frame, more candidates are required to reduce the feature missing problem. Thus, the feature tracker 212 can use a 3D tracking algorithm that does only one tracking while not retaining multiple candidates.

The detailed operation of the feature tracking step 420 is described in detail with reference to FIG. 6 below.

In the dynamic point detection step 430, the dynamic point detector 213 may determine the first features by removing features having a dynamic trajectory among the third features tracked to the last frame. That is, the features remaining after the removal of the third features may be the first features. The detailed operation of the dynamic point detection step 430 is described in detail with reference to FIG. 7 below.

In the camera position estimation step 440, the camera position estimation unit 214 may estimate the position of each of the cameras in each of the multi-frames based on the first features. The detailed operation of the camera position estimation step 440 is described in detail with reference to FIG. 8 below.

5 illustrates a feature extraction step according to an example.

In a point and scale invariant feature transform (SIFT) descriptor generation step 510, the feature extractor 211 may extract points from at least three images of the first frame of the multi-frames. . Here, the point may be a Harris corner point detected by a corner and an edge detector.

Also, the feature extractor 211 may generate Scale Invariant Feature Transform (SIFT) descriptors having a constant scale from three images of the first frame of the multi-frames. Can be. In general, the scale variation between the three images is small.

In the feature generation step 520, the feature extractor 211 may match the extracted points to each other using descriptor comparison between the generated SIFT descriptors, and generate third features by concatenating the matched points with the feature. can do. Here, for acceleration, the feature extractor 211 may use a kd tree for descriptor comparison.

There may be outliers among the features created by the above match. Hereinafter, the third feature generated by the match is denoted by the candidate feature x .

In the outlier removing step 530, the feature extractor 211 may remove the outliers from the third features generated using one or more of the three geometric constraints. Here, the three geometric constraints may be epipolar constraints, re-projection constraints and depth range constraints.

를 뷰 i 및 뷰 j의 상대적 위치들로부터 유도할 수 있다.

는 뷰 i 및 뷰 j 각각의 펀더멘털 행렬일 수 있으며, 뷰 i로부터 뷰 j로의 펀더멘털 행렬일 수 있다. 여기서, 뷰 i는 3 개의 카메라들 중 i 번째 카메라의 뷰일 수 있다.The feature extractor 211 may perform a fundamental matrix based on Equation 2 below.

Can be derived from the relative positions of view i and view j .

May be a fundamental matrix of views i and j respectively, and may be a fundamental matrix from view i to view j . Here, the view i may be a view of the i th camera among the three cameras.

는 3 개의 카메라들 중 제1 카메라로부터 제2 카메라로의 변환 벡터를 나타낼 수 있다.

은 제1 카메라로부터 제2 카메라로의 회전 벡터를 나타낼 수 있다. [t]_x는 벡터 t의 반대칭행렬(skew symmetric matrix)일 수 있다.Here, K may represent unique parameters.

May represent a conversion vector from the first camera to the second camera among the three cameras.

May represent a rotation vector from the first camera to the second camera. [ t ] _x may be a skew symmetric matrix of the vector t .

[ t ] _x may be defined as in Equation 3 below.

Therefore, Equation 4 below can be established for all x . The feature extractor 211 may regard the candidate feature x for which Equation 4 below does not hold as an outlier.

Re-projection validation can be applied as a second geometric test.

에 대해 하기의 수학식 5가 성립할 수 있다.For example, the feature extractor 211 may set the left camera as the reference camera. Projection matrices of the left camera when the left camera is set as the reference camera

Equation 5 below can be established for.

및 우측 카메라의 투영 행렬들

을 계산할 수 있다.By the above set, the feature extraction unit 211 is based on Equations 6 and 7 below, and the projection matrices of the intermediate camera.

And projection matrices of the right camera

Can be calculated.

Next, the feature extractor 211 may triangulate the 3D position M of each of the candidate features x using the positions of the cameras.

Two-view triangulation can be more stable and efficient than three-view triangulation. Thus, the feature extraction unit 211 has the m _right the feature points match in the left camera (that is, the right image), the feature points of m _left and right cameras (or, right image) match in, for example, Sampson lane triangulation We can initialize M using the Sampson suboptimal triangulation algorithm. Here, the feature point may mean a point in the image corresponding to the feature.

After the initialization of M , the feature extractor 211 can improve the M by minimizing the energy function of Equation 8 using m _left , m _middle and m _right by adding the feature point m _middle in the intermediate camera. have.

는 하기의 수학식 9에서와 같이 정의될 수 있다.Where projection function

May be defined as in Equation 9 below.

The minimized value of the energy function may be the re-projection error of candidate feature x .

The feature extractor 211 can use the minimized value of the energy function for the candidate feature x as a criterion for re-projection confirmation.

The feature extractor 211 may recognize candidate features that do not satisfy one of the two geometric constraints as outliers.

를 벗어나는 특징들을 아웃라이어로 간주할 수 있고, 아웃라이어로 간주된 후보 특징들 제거할 수 있다. Although the candidate feature meets both of the two geometric constraints described above, the candidate feature may still be outlier. In such cases, the triangulated depth of the candidate features is usually abnormal. Here, the abnormal depth may mean very small or very large depth. Thus, feature extractor 211 can use depth range constraints to remove outliers. The feature extractor 211 includes a depth range in which a depth value is specified among candidate features.

Features that deviate from may be considered as outliers and candidate features considered to be outliers may be removed.

를 자동으로 계산하기 위해 2 단계의 적응적인 문턱치 선택 방법을 사용할 수 있다.Feature extraction section 211 provides depth range for each view i

The adaptive threshold selection method in two stages can be used to automatically calculate.

및 분산

를 계산할 수 있다.In a first step, the feature extractor 211 may calculate depth values of all features appearing in view i for each view i . The feature extractor 211 may select the lowest depth value of 80% among the calculated depth values. The feature extractor 211 uses the selected depth values to average the depth values for the view i .

And dispersion

Can be calculated.

및 분산

를 사용하여 깊이 범위의 최소 값

및 최대 값

를 계산할 수 있다.The feature extractor 211 may calculate an average depth value based on Equations 10 and 11 below.

And dispersion

Minimum value of depth range using

And maximum values

Can be calculated.

의 값은 5로 세트될 수 있다.Here,

The value of may be set to 5.

의 값은 0에 매우 가까워서, 무익한 것일 수 있다.However, generally calculated

The value of is very close to zero, which can be useless.

를 계산할 수 있다.In the second step, the feature extractor 211 is more accurate

Can be calculated.

Three view pairs can be obtained by given triangulation views. The three view pairs may be expressed as in Equation 12 below.

Since trinocular cameras have almost the same orientation, the trifling rotating component can be neglected. Thus, (X _i, Y _i, Z _i) and ^T is based on the i-th back camera 3D position of the feature point of the feature extraction section 211, equation (14) in equation (13) and to below, x _j and x _j can be calculated. x _j may be the x- coordinate on the focal image plane of the i th view and x _j may be the x- coordinate on the image plane of the focal point of the j th view.

Therefore, the feature extractor 211 may calculate the depth Z _i of the feature point in the i th view based on Equation 15 below.

Here, the feature extractor 211 may calculate dx _ij based on Equation 16 below.

을 선택하는 것은 dx _ij 의 최대 값을 선택하는 것과 동일할 수 있다.As mentioned above, the minimum depth value

Choosing may be equivalent to choosing the maximum value of dx _ij .

The feature extractor 211 may calculate dx _ij of each of the candidate features appearing in both the i th view and the j th view among the candidate features. The feature extractor 211 may collect all the calculated dx _ij values, and collect all the collected dx _ij { d _{x 1} , d _{x 2} ,. You can sort in descending order as shown in}.

로 선택할 수 있다.The feature extractor 211 uses the reference value dx _ij having a value of 80% from the top among the dx _ij arranged in descending order.

.

The feature extractor 211 may calculate a minimum depth value of the i th view based on Equation 17 below.

6 is a flowchart of a feature tracking step according to an example.

In feature tracking step 420, the movement of the third features in the two frames is tracked. The flowchart below describes a method of calculating the position of a feature in the current frame based on the position and positional movement of the feature in the previous frame.

It can be reasonably assumed that the 3D position of the extracted feature will not change much between successive frame coordinate systems. The 3D movement movement V of the feature may be formulated as in Equation 18 below.

Here, M can represent the 3D position within the previous frame of the feature. M ′ may represent a 3D position within the current frame of the feature.

The most basic measure for calculating the position shift may be the similarity between the image patch in the previous frame and the image patch in the current frame.

The feature tracker 212 may minimize the energy function f ( v ) based on Equation 19 below. Here, the energy function f ( v ) may represent the similarity between the image patch in the previous frame and the image patch in the current frame.

는 이전 프레임 내의 i 번째 이미지일 수 있다.

는 현재 프레임 내의 i 번째 이미지일 수 있다. Loc(M, i, j)는 i 번째 이미지 평면 내의, M의 투영이 중심에 위치한 지역 윈도우의 j 번째 픽셀의 위치일 수 있다.here,

May be the i th image in the previous frame.

May be the i th image in the current frame. Loc ( M , i , j ) may be the location of the j th pixel of the local window in which the projection of M is centered within the i th image plane.

The feature tracker 212 may calculate Loc ( M , i , j ) based on Equation 20 below.

Here, v _j may represent an offset from the center of the local window of the j th pixel.

를 단순화하여,

로 표시한다. 여기서,

일 수 있다. 따라서, 수학식 19는 하기의 수학식 21과 같이 단순화될 수 있다.In the following,

By simplifying

. here,

Lt; / RTI > Accordingly, Equation 19 may be simplified as in Equation 21 below.

Equation 21 may be modified as in Equation 22 below.

In addition, Equation 18 may be modified as in Equation 23, and M 'may be defined as in Equation 23 below.

및

는 하기의 수학식 24에서와 같이 근사화될 수 있다.When the three-dimensional motion V is small, the term

And

Can be approximated as in Equation 24 below.

일 수 있다.here,

Lt; / RTI >

를 분해적으로(analytically) 계산할 수 있다.The feature tracking unit 212 is based on a chain rule as shown in Equation 25 below.

Can be calculated analytically.

Here, M _c may be a 3D position in the camera coordinate system. m _i may be a 2D position within the image plane. M _c may be the 2D location of the j th pixel in the local window centered at m _i .

를 계산할 수 있다.Equation 29 below holds as Equations 26, 27, and 28 below. The feature tracking unit 212 is based on Equation 29 below.

Can be calculated.

는 이미지 경사도(gradient)일 수 있다.

는

의 야코비안(jacobian) 행렬일 수 있으며, 하기의 수학식 30에서와 같이 정의될 수 있다.here,

May be an image gradient.

The

It may be a Jacobian matrix of, and may be defined as in Equation 30 below.

By combining Equations 22, 24 and 29, f ( V ) can be approximated as in Equation 31 below.

Here, g _i , _j , T _i and d _i , _j may be defined as in Equations 32, 33, and 34, respectively.

Here, the subscript may indicate a dependency. For example, HR _i may indicate that HR _i only depends on the index of the view and is irrelevant to image patches. Minimization of f ( V ) in Equation 31 may be equivalent to solving the 3 x 3 linear system of Equation 35 below.

The feature tracker 212 can use an iterative scheme to find the solution of the 3D positional shift V.

In the repetition number initialization step 610, the feature tracker 212 may initialize the value of the repetition number k to zero.

In the energy function initialization step 620, the feature tracker 212 may calculate an initial value V ⁽⁰⁾ of the energy function based on Equation 35 above.

In the position initialization step 630, the feature tracker 212 may calculate the initial position M ' ⁽⁰⁾ of the feature according to Equation 36 below.

In the step of increasing the number of repetitions 640, the feature tracker 212 may increase the value of the number of repetitions k by one.

In calculating the energy function step 650, the feature tracking unit 210, and based on the number of times the value k in Equation 37 below to be used, the energy function of the k th iteration f (V ^(k)) to calculate the have.

In the position calculation step 660, the feature tracking unit 210, based on the number of iterations the value k Equation 38 below that use can be determined the position M ^'(k) of the characteristics of the k th iteration.

In the repetition count checking step 670, if the feature tracking unit 210 repetition count value k is less than or equal to a specified threshold, step 640 may be performed again. The feature tracking unit 210 may end the iterative calculation of f ( V ^{( k )} ) and M ′ ^{( k )} when the number of iterations k reaches a specified threshold.

In the feature tracking step 680, the feature tracker 210 may track each of the features in the current frame based on the calculated position. That is, the feature tracker 210 may simultaneously estimate the correspondence between features in successive frames and the 3D position of the frames.

The feature tracker 210 may track each of the third features to the last of the multi-frames by applying steps 610 to 680 sequentially for successive frames of the multi-frame.

7 is a flowchart of a dynamic point detection step according to an example.

In this example, a point may refer to a third feature extracted and tracked in steps 410 and 420.

In order to detect dynamic points, a 2D background subtraction algorithm for free moving cameras, for example proposed by Yaser Sheikh, can be generalized to be applicable to 3D trajectory space. Considering that the three cameras are static, the static points are considered to perform rigid motion accordingly in the opposite direction of the actual movement of the trinocular rig. Thus, the trajectories of the static points can lie within the lower dimension of the subspace.

The trajectory of the point can be defined as the category of 3D coordinates within successive frames.

In steps 410 and 420, when N _p points are extracted and tracked within the N _f multi-frames, the dynamic point detector 213 performs the i th point based on Equation 39 below. Calculate the locus of w _i .

Here, M _i , _j may represent local coordinates in the j- th frame.

행렬 W 내에 배열할 수 있다.The dynamic point detection unit 213 collects all N _p points as shown in Equation 40 below.

Can be arranged in the matrix W.

에 대해서, M _i , _j 는

와 동일할 수 있다. 여기서,

는 4D 전역(world) 동질(homogeneous) 좌표일 수 있다.

는 j 번째 프레임에 관한 강체 운동의 3 x 4 행렬일 수 있다.If all points are static each

For M _i , _j is

≪ / RTI > here,

May be a 4D world homogeneous coordinate.

May be a 3 x 4 matrix of rigid body motion with respect to the j th frame.

Accordingly, W may be factored as in Equation 41 below.

Factorization according to equation (41) indicates that the rank of W is at most 4. Thus, the trajectories of the static point are in subspace spanned from four basic trajectories.

The dynamic point detector 213 may use a random SAmple Consensus (RANSAC) algorithm to robustly calculate the best estimate of the four-dimensional trajectory subspace while identifying the trajectories lying within the subspace. In each iteration, four trajectories, denoted by w ₁ , w ₂ , w _3, and w ₄ , are randomly selected to produce a subspace, and the matrix W ₄ ( W ₄ ^T W ₄ ) ^-1 W ₄ ^T is used to project different trajectories into the subspace. Where W ₄ = [ w ₁ ... w ₄ ]. The dynamic point detector 213 may directly measure the Euclidean distance between the original trajectory and the projected trajectory in order to evaluate the likelihood that the given trajectory will belong to the given subspace.

로 분할(split)할 수 있다. 또한, 동적 포인트 검출부(213)는, 하기의 수학식 42에 기반하여, 투영 오류 f(w _i )를 계산할 수 있다.Indeed, it is difficult to tune the threshold for the Euclidean distance defined within the 3 N _f space to determine whether the trajectory lies in the subspace. Instead, in order to evaluate w _i , the dynamic point detector 213 determines the projected trajectory W ₄ ( W ₄ ^T W ₄ ) ^-1 W ₄ ^T w _i with N _f points.

You can split by. In addition, the dynamic point detector 213 may calculate the projection error f ( w _i ) based on Equation 42 below.

Here, m _i , _j , _k may indicate the position of the i th point on the k th image within the j th frame.

In the trajectory selection step 710, the dynamic point detector 213 selects four trajectories w ₁ , w ₂ , w ₃ and w ₄ for generating a subspace.

In the consensus detection step 720, the dynamic point detector 210 detects a consensus within four trajectories w ₁ , w ₂ , w _3, and w ₄ selected based on the RANSAC algorithm.

In the consensus comparison step 730, if there is sufficient consensus in the data supporting the selected trajectories, the dynamic point detector 213 may end the routine. Otherwise, step 710 is repeated. The dynamic point detector 213 may select four other trajectories until the maximum consensus set is found.

In the dynamic point determination step 740, the dynamic point detector 213 may regard the trajectories that do not belong to the maximum concentration set as the dynamic points.

As described above, the dynamic point detector 210 may calculate a four-dimensional trajectory subspace of each of the third features, and determine whether each of the third features has a dynamic trajectory based on the calculated four-dimensional trajectory subspace. You can judge.

The dynamic point detector 213 may determine the first features by removing features having a dynamic trajectory among the third features tracked to the last frame. That is, the first features may be features that are tracked to the last frame of the third features extracted in the first frame and do not have a dynamic trajectory.

8 is a flowchart of a camera position estimation according to an example.

In the multi-frames processing step 210, if the first features are extracted, the extracted first features are tracked within the N _f frames, and the dynamic points are removed, the camera position estimator 214 is structure- The from-Motion technique can estimate the world coordinates of N _p points and the 3 N _f movements of the cameras simultaneously. The global coordinate of each of the N _p points is represented by M _i . Where i = 1,... , N _f .

으로 세트되었거나, 이전의 추적된 부분열 내에서 추정될 수 있다. 따라서, 카메라 위치 추정부(214)는 현재의 추정된 잇따른 연쇄에 대한 N _f- 1 개의 프레임 리그들만을 추정할 수 있다.In practice, the camera position estimator 214 estimates only N _f- 1 frame rigs for the current tracked subsequence, instead of estimating 3 N _f motions of the cameras. Can be. The relative positions between the three cameras are fixed and known. Also, the first frame rig of these substrings will be

Can be set, or estimated within the previously tracked substring. Accordingly, the camera position estimator 214 may estimate only N _f- 1 frame rigs for the current estimated subsequent chain.

The camera position estimator 214 may set the frame rig to the left camera. The camera position estimator 214 may derive 3 N _f camera positions from the frame rig, based on Equation 43 below.

는 3N _f 개의 카메라 위치들 중 하나의 위치를 나타낼 수 있다. j = 1 … N _f 일 수 있고,

일 수 있다.

는 프레임 리그를 나타낼 수 있다. j = 1 … N _f 일 수 있다.here,

May represent the position of one of the 3 N _f camera positions. j = 1. Can be N _f ,

Lt; / RTI >

May represent a frame rig. j = 1. May be N _f .

In the first frame rig initialization step 810, since N _p points have been triangulated within the first frame of the current substring, the camera position estimator 214 performs inverse transformation with a known frame rig. Through this, world coordinates of N _p points in the first frame can be initialized.

In the remaining frame rig initialization step 820, the camera position estimator 214 performs non-linear optimization using the Levenberg-Marquardt algorithm and (eg, Long Quan, etc.). The camera position estimation method (proposed by) can be used to initialize the rest frame rigs.

In detail, the camera position estimator 214 may perform initialization on the j th frame based on Equation 44 below.

는 j 번째 프레임의 k 번째 이미지로의 i 번째 포인트의 재-투영일 수 있다. N _j 는 j 번째 프레임 내에서의 가시의(visible) 포인트들의 개수일 수 있다.Here, m _i , _j , _k may be 2D measurements of the i th point in the k th image of the j th frame.

May be re-projection of the i th point into the k th image of the j th frame. N _j may be the number of visible points in the j th frame.

After initialization, in step 830, minimizing the re-projection error, the camera position estimator 214 may minimize the re-projection error for all 3D points and camera parameters, based on Equation 45 below. have.

Unlike Equation 44, Equation 45, which is a normal equation, produces a sparse block structure due to a lack of interaction between parameters for different 3D points and cameras. Have

The sparse variant of the Levenberg-Marquart algorithm can benefit from the zero pattern of the regular equation by avoiding storing and operating on "0" elements. The lean block structure can be exploited to obtain tremendous computational benefits by employing the lean variance of the Levenberg-Marquardt algorithm. This application can be called a bundle adjustment and can be used as the standard final step in a Structure-from-Motion system based on almost all features.

에 대한 3 개의 파라미터, 증대하는(incremental) 회전 행렬

의 파라미터화 및 카메라 중심 c에 대한 3 개의 파라미터들을 사용하여

를 파라미터로 나타낼(parameterize) 수 있다.In particular, with respect to Equations 44 and 45, the camera position estimator 214 is based on, for example, a method proposed by Noah Snapley or the like shown in Equation 46 below.

Three parameters for the incremental rotation matrix

Using three parameters for the parameterization of the camera center c and

Can be parameterized.

는 수학식 3에서 정의된 것과 같은 반대칭행렬일 수 있다. R ^init는 초기의 회전 행렬일 수 있다. 카메라 위치 추정부(214)는, 하기의 수학식 47의 함수들에 기반하여, M _i 를 j 번째 프레임 내의 3 개의 뷰들로 투영할 수 있다.here,

May be an antisymmetric matrix as defined in equation (3). R ^init can be an initial rotation matrix. The camera position estimator 214 may project M _i to three views within the j th frame based on the functions of Equation 47 below.

The camera position estimator 214 may decompose the Jacobian matrix on f ( v ) in both Equations 44 and 45 based on the chain rule.

The camera position estimator 214 may calculate the middle view M _middle based on Equation 48 below.

Here, the coordinate of M may be represented with respect to the intermediate camera.

,

및

를 계산할 수 있다.The camera position estimator 214 is based on Equations 49, 50, and 51, respectively.

,

And

Can be calculated.

는 상술된 수학식 30에서 정의된 것과 동일할 수 있다.here,

May be the same as defined in Equation 30 described above.

및 항

은 모든 포인트들에 대해 동일하다. 카메라 위치 추정부(214)는 항

및 항

을 프레임 리그가 갱신될 때 단 1회 선-계산(pre-compute)할 수 있다.term

And terms

Is the same for all points. The camera position estimator 214

And terms

Can be pre-computed only once when the frame rig is updated.

및

는 서로 부호(sign)에 있어서만 상이하며, 2 번 계산될 필요가 없다. 카메라 위치 추정부(214)는 우측 뷰에 대한 야코비안 행렬을 전술된 것과 유사한 방식으로 유도할 수 있다.

And

Are different only in sign from each other and need not be calculated twice. The camera position estimator 214 may derive the Jacobian matrix for the right view in a manner similar to that described above.

In the camera position derivation step 840, the camera position estimator 214 may derive 3 N _f camera positions from the frame rigs, based on Equation 43 described above.

9 is a flowchart of a single-frame processing step according to an example.

The single-frame processing step 320 may include the following steps 910-950.

In single-frame processing step 320, the position of each of the cameras in the current frame can be estimated. The current frame can be initialized by a Linear N-Point Camera Pose Determination method. In addition, the operation of the single-frame processing step 320 may be optimized based on Equation 43 and Equation 44. One of the differences between the multi-frames processing step 310 and the single-frame processing step 320 is that in the single-frame processing step 320, the bundle adjustment can be very local. That is, the points that can be adjusted in the single-frame processing step 320 may be limited to only points that appear within the current frame and the current frame rig. To prevent local optimization, the single-frame processor 220 can also measure the projection of points within previous frames. Thus, camera parameters of previous frames involved may also be used as a constant in equation (45).

The single-frame processing step 320 may be performed after the first features have been extracted and tracked in the multi-frames processing step 310.

In the current frame setting step 910, the current frame setting unit 221 may set the next frame of the multi-frame as the current frame.

In the current frame feature tracking step 920, the current frame feature estimator 222 may extract and track the second features within the current frame. The second features may each correspond to a feature of one of the first features extracted and tracked in the multi-frames processing step 310. That is, the second features may be that the first features extracted in the multi-frames appear continuously in the current frame.

In the threshold comparison unit 930, the current frame threshold comparison unit 223 may check whether the number of the second features extracted in the current frame is greater than or equal to the threshold. If it is above the threshold, step 940 may be performed.

If the number of the extracted second features is less than the threshold, the single-frame processing step 320 may end. After the single-frame processing step 320 ends, for example, there are remaining frames to be processed. When the multi-frames processing step 310 is re-executed, the multi-frames processing unit 210 may be re-executed. It may be more than one consecutive frames.

In the current frame position estimation step 940, the current frame camera position estimator 224 may estimate the position of each of the cameras in the current frame.

In the current frame update step 950, the current frame setting unit 221 may newly set the next frame of the current frame as the current frame. Thereafter, the current frame feature tracking step 920 may be repeatedly performed. That is, the current frame feature estimator may extract the second features within the newly set current frame.

Technical contents according to the embodiment described above with reference to FIGS. 1 to 8 may be applied to the present embodiment as it is. Therefore, more detailed description will be omitted below.

The method according to one embodiment may be implemented in the form of program instructions that may be executed through various computer means and recorded on a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be those specially designed and constructed for the embodiments or may be available to those skilled in the art of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; Magneto-optical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

As described above, the present invention has been described by way of limited embodiments and drawings, but the present invention is not limited to the above embodiments, and those skilled in the art to which the present invention pertains various modifications and variations from such descriptions. This is possible.

Therefore, the scope of the present invention should not be limited to the described embodiments, but should be determined by the equivalents of the claims, as well as the claims.

100: camera position tracking device
210: multi-frames processing unit
220: single-frame processing unit

Claims (16)

A method of tracking the location of cameras using frames taken by at least three cameras, the method comprising:
Extracting and tracking one or more first features within the multi-frames, and tracking the location of each of the cameras within each of the multi-frames based on the first features; And
Tracking the location of each of the cameras within each of the one or more single-frames based on second features of each of the one or more single-frames.
Wherein the one or more single-frames are previous frames of the first frame in which second features corresponding to one of the first features of frames subsequent to the multi-frames are tracked less than a threshold. Location tracking method. The method of claim 1,
Tracking the location of each of the cameras within each of the multi-frames,
Extracting third features from at least three images of the first of the multi-frames;
Tracking the third features to the last of the multi-frames;
Determining the first features by removing features having a dynamic trajectory of third features tracked to the last frame; And
Estimating a position of each of the cameras within each of the multi-frames based on the first features
Including, camera position tracking method. The method of claim 2,
Extracting the third features may include:
Extracting points from at least three images of the first frame and generating Scale Invariant Feature Transform (SIFT); And
Matching the extracted points to each other using descriptor comparison between the generated SIFTs, and generating the third features by concatenating matched points among the points as features.
Including, camera position tracking method. The method of claim 3,
Extracting the third features may include:
Removing outliers of the third features using geometric constraints
Further comprising, the camera position tracking method. 5. The method of claim 4,
Wherein the geometric constraint is one or more of epipolar constraint, re-projection constraint, and depth range constraint. The method of claim 1,
Tracking the location of each of the cameras within each of the single-frames,
Setting a next frame of the multi-frames as a current frame;
Extracting the second features corresponding to one of the first features within the current frame;
Estimating a position of each of the cameras in a current frame when the number of the second features is greater than or equal to a threshold; And
Setting the next frame of the current frame as the current frame and extracting the second features when the number of the second features is greater than or equal to a threshold;
Including, camera position tracking method. The method of claim 6, wherein
Tracking the location of each of the cameras within each of the single-frames,
Re-tracking the location of each of the cameras within each of the multi-frames if the number of the second features is less than a threshold.
Further comprising, the camera position tracking method. A computer-readable recording medium containing a program for performing the method of any one of claims 1 to 7. A camera position tracking device for tracking positions of cameras using frames taken by at least three cameras,
A multi-frames processor for extracting and tracking one or more first features within the multi-frames and tracking the location of each of the cameras in each of the multi-frames based on the first features; And
A single-frame processor for tracking the position of each of the cameras within each of the single-frames based on the second features of each of the one or more single-frames
Wherein the one or more single-frames are previous frames of the first frame in which second features corresponding to one of the first features of frames subsequent to the multi-frames are tracked less than a threshold. Location tracking device. 10. The method of claim 9,
The multi-frames processing unit,
A feature extractor for extracting third features from at least three images of a first frame of the multi-frames;
A feature tracker for tracking the third features to a last frame of the multi-frames;
A dynamic point detector for determining the first features by removing features having a dynamic trajectory among the third features tracked to the last frame; And
A camera position estimator for estimating a position of each of the cameras in each of the multi-frames based on the first features
Comprising a camera position tracking device. The method of claim 10,
The dynamic point detection unit calculates a four-dimensional trajectory subspace of each of the third features, and determines whether each of the third features has a dynamic trajectory based on the four-dimensional trajectory subspace. . The method of claim 10,
The feature extraction unit may extract,
Extract points from at least three images of the first frame, generate Scale Invariant Feature Transform Descriptors (SIFTs), and compare the extracted points to each other using a descriptor comparison between the generated SIFTs. And generate the third features by concatenating matched points among the points as features. The method of claim 12,
And the feature extractor determines the first features by removing outliers of the third features using geometric constraints. The method of claim 13,
And the geometric constraint is one or more of epipolar constraint, re-projection constraint and depth range constraint. 10. The method of claim 9,
The single-frame processing unit,
A current frame setting unit for setting a next frame of the multi-frames as a current frame;
A current frame feature estimator for extracting the second features corresponding to one of the first features in the current frame; And
A threshold comparison unit estimating a position of each of the cameras in a current frame when the number of the second features is greater than or equal to a threshold;
Lt; / RTI >
When the number of the second features is greater than or equal to the threshold, the current frame setting unit newly sets the next frame of the current frame as the current frame, and the current frame feature estimating unit sets one of the first features from the newly set current frame. Extracting the second features corresponding to the feature. 16. The method of claim 15,
And the multi-frames processor is re-executed when the number of the second features is less than a threshold.

Images