KR101985344B1 - Sliding windows based structure-less localization method using inertial and single optical sensor, recording medium and device for performing the method - Google Patents

Abstract

A sliding window based non-structural location recognition method using an inertial and a single optical sensor includes the steps of applying an inertial sensor signal to a continuous time inertial navigation system model and propagating the inertial sensor signal over time to determine an inertial navigation state variable; Estimating navigation information by constructing an error-state Kalman filter (ESKF) using the inertial navigation system model; Extracting a feature point from a newly input image and performing feature point extraction and matching between images in a sliding window matching the extracted feature points with an image input in a previous frame; Updating a sliding window; Estimating a camera state variable through optimization based on a multiple geometry constraint between state variables of a camera and a feature point matching relationship in a sliding window; And correcting the state variable of the inertial navigation system model using the estimated camera state variable as a measurement value. Accordingly, the multi-view constraint minimization-based optical / inertia-coupled position recognition system proposed in the present invention optimizes the three-dimensional position estimation of feature points using the multi-view geometry constraint, The calculation amount is smaller than that in the case of the linear motion, and a stronger estimation is also possible in the case of the linear motion.

Description

TECHNICAL FIELD [0001] The present invention relates to a sliding window-based non-structural location recognition method using inertial and single optical sensors, a recording medium and an apparatus for performing the same,

The present invention relates to a sliding window-based non-structural position recognition method using inertial and single optical sensors, a recording medium and an apparatus for performing the same, and more particularly, And optimizing the constraint based on the multi-viewpoint geometry that does not require three-dimensional position information of the feature points, thereby providing a position and an attitude of the object.

Position recognition is very important in systems such as mobile robots, drones, and unmanned vehicles. This is because it is necessary to know precisely the position of the user in order to determine the next movement in connection with the surrounding environment information. This also applies to virtual reality or augmented reality systems. This is because it is necessary to accurately recognize the wearer or the user's position so that the correct information can be outputted to the screen of the apparatus.

In general, the manner of combining a single optical sensor and an inertial sensor is based on a sliding window approach. The navigation information and the tracked feature point information are stored in the sliding window every time the optical sensor captures an image, and when the length of the predetermined sliding window is reached, the 3D position of the feature point is estimated using the navigation information and the feature point information stored in the sliding window do.

The navigation information stored in the sliding window and the 3D position information of the minutiae are set as initial values and the minutiae point is re-projected to the image using the information to optimize the re-projection error with respect to the minutiae on the actually obtained image . However, there is a disadvantage that the 3D position of the unnecessary feature points must also be obtained in the position recognition system which does not perform the mapping in this process.

A single camera / inertial sensor coupled position recognition system generally performs optimization to minimize re-projection errors. There are various schemes for performing the optimization, but in order to minimize the re-projection error, optimization is performed by considering the position of the camera, the position of the camera, and the position of the minutiae in the sliding window as optimization variables.

The minimization of the redistribution error is based on the location recognition system. First, the inertial sensor information integrated in the system is integrated to calculate the navigation information of position, velocity, and attitude. At the same time, the uncertainty of the navigation information calculated based on the inertial sensor error information is calculated.

Also, whenever the camera acquires an image, it stores position, attitude information, uncertainty information, and extracted feature point information in the sliding window. Repeat the above steps until the sliding window is full, and when the sliding window is full, discard the information of the first window and add the new information to the rear. At this time, it is assumed that the correlation between the incoming minutiae points is acquired.

Based on the geometric constraint between navigation information and minutia information present in the sliding window, navigation information within the sliding window and three-dimensional location of the minutiae are optimized through optimization so that the re-projection error is minimized. The navigation information of the inertial navigation system is corrected using the information in the sliding window obtained through the optimization.

The re-projection error refers to the difference between the position where the 3D position of the feature point shown in the image is combined with the navigation information in the sliding window and the position of the re-projection on the image and the actual feature point extracted from the image as shown in FIG. For example, when M feature point tracking results are used in a sliding window of length N, the number of variables to be optimized is 6xN + 3xM.

However, the position recognition system does not need to obtain specific point information unlike the mapping system. In the existing system, the minutiae information is a variable that must be sought together to obtain navigation information. However, since the number of feature points tracked in an image is generally larger than the length of a sliding window, this greatly affects the amount of computation.

In the case of the optical / inertial coupled position recognition system based on the existing re-projection error minimization, the 3D position of the feature points is also estimated in order to optimize the variables in the sliding window. In the case of minimizing the re-projection error using the three-dimensional position, it is necessary to further estimate the three-dimensional position information which is three times the number of the feature points, and a large amount of calculation is additionally required in this process. In addition, when linear motion is performed, many errors occur in the three-dimensional position estimation and the quality of the optimization result is lowered.

KR 10-2014-0053712 A KR 10-2015-0079098 A

Mourikis, Anastasios I., and Stergios I. Roumeliotis. "A multi-state constraint Kalman filter for vision-aided inertial navigation." Robotics and automation, 2007 IEEE international conference on. IEEE, 2007. Indelman, Vadim. "Bundle adjustment without iterative structure estimation and its application to navigation." Position Location and Navigation Symposium (PLANS), 2012 IEEE / ION. IEEE, 2012.

SUMMARY OF THE INVENTION Accordingly, the present invention has been made in view of the above problems, and it is an object of the present invention to provide a sliding window-based nonlinear optical sensor using an inertial and single optical sensor with reduced computational complexity by performing optimization using constraint conditions based on multi- And to provide a method of recognizing the structure position.

Another object of the present invention is to provide a recording medium on which a computer program for performing a sliding window-based non-structural location recognition method using the inertial and single optical sensors is recorded.

It is still another object of the present invention to provide an apparatus for performing a sliding window-based non-structural location recognition method using the inertial and single optical sensors.

In order to accomplish the object of the present invention, a sliding window-based non-structural position recognition method using an inertial and single optical sensor according to an embodiment of the present invention applies an inertial sensor signal to a continuous time inertial navigation system model, Determining an inertial navigation state variable; Estimating navigation information by constructing an error-state Kalman filter (ESKF) using the inertial navigation system model; Extracting a feature point from a newly input image and performing feature point extraction and matching between images in a sliding window matching the extracted feature points with an image input in a previous frame; Updating a sliding window; Estimating a camera state variable through optimization based on a multiple geometry constraint between state variables of a camera and a feature point matching relationship in a sliding window; And correcting the state variable of the inertial navigation system model using the estimated camera state variable as a measurement value.

In the embodiment of the present invention, the step of predicting the navigation information by constructing the error state variable Kalman filter (ESKF) using the inertial navigation system model may include: determining a state variable as a nominal state variable and an error state variable ; Propagating the nominal state variable using a nonlinear system model; And error state variables may be estimated using a linearization model.

In an embodiment of the present invention, updating the sliding window comprises: storing position and orientation information and uncertainty of the most recent optical sensor signal in time; And discarding information associated with the oldest optical sensor signal in time.

In the embodiment of the present invention, the step of extracting and matching feature points between the images in the sliding window may use FAST (feature from Accelerated Segment Test) corner extractor and KLT (Kanade-Lucas-Tomasi feature tracker).

A computer program for performing a sliding window-based non-structural location recognition method using inertial and single optical sensors is recorded in a computer-readable storage medium according to an embodiment for realizing the above-mentioned object of the present invention .

According to an embodiment of the present invention, a sliding window-based non-structural position recognition apparatus using an inertial sensor and a single optical sensor applies an inertial sensor signal to a continuous-time inertial navigation system model, A state variable determining unit for determining an inertial navigation state variable by propagating the signal; A navigation information predicting unit for predicting the navigation information by constructing an error-state Kalman filter (ESKF) using the inertial navigation system model; A matching relation extracting unit for extracting a feature point from a newly input image and performing feature point extraction and matching between images in a sliding window matching the extracted feature points with an image input in a previous frame, A sliding update unit updating the sliding window; A state variable estimator for estimating camera state variables through optimization based on multiple geometry constraints between camera state variables and feature point mappings in a sliding window; And a state variable correction unit for correcting the state variable of the inertial navigation system model using the estimated camera state variable as a measurement value.

In an embodiment of the present invention, the navigation information predicting unit divides the state variable into a nominal state variable and an error state variable, propagates the nominal state variable using a nonlinear system model, Can be estimated using a model.

In an embodiment of the present invention, the updating unit may store position and orientation information and uncertainty of the most recent optical sensor signal in time, and discard information associated with the oldest optical sensor signal in time.

In an embodiment of the present invention, the matching relation extracting unit may use a FAST (Features from Accelerated Segment Test) corner extractor and a KLT-tracker (Kanade-Lucas-Tomasi feature tracker).

In an embodiment of the present invention, the inertial sensor signal may include accelerometer data and gyro data of a three-dimensional vector.

According to the sliding window based non-structural position recognition method using inertia and single optical sensor, since the optimization is performed without estimating the three-dimensional position using the multi-view geometry constraint, , And a stronger estimation is also possible in the case of linear motion.

FIG. 1 is a view for explaining the conventional concept of re-projection error.
2 is a diagram of a position recognition system in accordance with the present invention.
3 is a block diagram of a sliding window-based non-structural location sensing device using inertial and single optical sensors in accordance with an embodiment of the present invention.
FIG. 4 is a graphical representation of a state variable estimation based on minimization of re-projection error and a state variable estimation based on minimization of a multiple geometry constraint error according to the present invention.
FIG. 5 is a satellite image of a position recognition result for an unmanned vehicle according to a sliding window-based non-structural position recognition method using inertia and single optical sensor of the present invention.
6 is a flow diagram of a sliding window-based non-structural location recognition method using inertial and single optical sensors in accordance with an embodiment of the present invention.

The following detailed description of the invention refers to the accompanying drawings, which illustrate, by way of illustration, specific embodiments in which the invention may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention. It should be understood that the various embodiments of the present invention are different, but need not be mutually exclusive. For example, certain features, structures, and characteristics described herein may be implemented in other embodiments without departing from the spirit and scope of the invention in connection with an embodiment. It is also to be understood that the position or arrangement of the individual components within each disclosed embodiment may be varied without departing from the spirit and scope of the invention. The following detailed description is, therefore, not to be taken in a limiting sense, and the scope of the present invention is to be limited only by the appended claims, along with the full scope of equivalents to which such claims are entitled, if properly explained. In the drawings, like reference numerals refer to the same or similar functions throughout the several views.

Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the drawings.

2 is a diagram of a position recognition system in accordance with the present invention. 3 is a block diagram of a sliding window-based non-structural location sensing device using inertial and single optical sensors in accordance with an embodiment of the present invention.

The multi-view constraint minimization-based optical / inertia coupled position recognition system (1) proposed in the present invention optimizes the constraint conditions based on the multi-view geometry which does not require the three-dimensional position information of the feature points, There is no need to obtain, and the number of variables needed for optimization is reduced.

Also, the re-projection method is known to be vulnerable when the optical sensor takes a forward movement, but constraints based on multiple view geometry are known to be robust to forward movement. Therefore, by using the present invention, it is possible to overcome the limitations of the existing system in terms of calculation amount and stability.

2, a multi-view constrained minimization based optical / inertial coupled position recognition system 1 includes an inertial sensor 10, an optical sensor 30 and a signal from the inertial sensor 10 and the optical sensor 30 And a sliding window based non-structural location recognition device 50 (hereinafter referred to as a device) using an inertial and single optical sensor that receives and recognizes a location.

The geometric relationship between the inertial sensor 10 and the optical sensor 30 can be obtained in advance through a calibration process. The inertial sensor 10 and the optical sensor 30 may be configured separately from the device 50 or may be attached to the device 50 or some module.

The signal of the inertial sensor 10 is composed of accelerometer data and gyro data, and is a three-dimensional vector. The inertial sensor 10 may be a single sensor, and the optical sensor 30 may be a camera. The camera may be a single camera at multiple points of view or a plurality of cameras at a single point of view.

The device 50 may be mobile or stationary. The device 50 may be in the form of a server or an engine and may be a device, an apparatus, a terminal, a user equipment (UE), a mobile station (MS) a wireless device, a handheld device, and the like.

The device 50 may execute or produce various software based on an operating system (OS), i.e., a system. The operating system is a system program for allowing software to use the hardware of a device. The operating system includes a mobile computer operating system such as Android OS, iOS, Windows Mobile OS, Sea OS, Symbian OS, Blackberry OS, MAC, AIX, and HP-UX.

Referring to FIG. 3, the apparatus 50 includes a state variable determining unit 510, a navigation information predicting unit 520, a matching relation extracting unit 530, a sliding updating unit 540, a state variable estimating unit 550, And a state variable correcting unit (not shown).

The state variable determining unit 510 determines an inertial navigation state variable by propagating with time according to a continuous time inertial navigation system model using an inertial sensor signal.

The inertial navigation system of the state variable determining unit 510 determines the inertial navigation state variable by propagating with time according to the continuous time inertial navigation system model of Equation (1) using the inertial sensor signal.

[Equation 1]

여기서

,

는 각각 기준좌표계인 {G}에서 표현한 동체 좌표계 {B}의 중심인 IMU의 위치 및 속도를 나타내며,

는 기준좌표계 대비 동체좌표계의 자세를 나타내는 단위 쿼터니언(unit quaternion)을 나타낸다.

는 동체좌표계에서 표현된 벡터를 기준좌표계로 변환하는 좌표변환 행렬인

와 유사하게 동체좌표계에서 표현된 순수 쿼터니언(pure quaternion)의 좌표변환에 사용될 수 있다.

here

,

Represents the position and velocity of the IMU, which is the center of the body coordinate system {B} expressed in {G}, which is the reference coordinate system,

Represents a unit quaternion representing the attitude of the body coordinate system relative to the reference coordinate system.

Is a coordinate transformation matrix for transforming a vector expressed in a body coordinate system into a reference coordinate system

Can be used for coordinate transformation of a pure quaternion expressed in the body coordinate system.

,

는 각각 동체좌표계에서 표현된 비력(specific force)과 각속도를 나타내며,

는 기준좌표계에서 표현된 중력벡터를 나타낸다. 가속도계 및 자이로스코프의 출력은 바이어스를 가지며, 시간에 따라 바이어스가 변하는 편류(drifit)가 생기게 된다.

,

는 각각 가속도계와 자이로스코프의 바이어스를 의미하며,

,

는 가속도계와 자이로스코프의 바이어스 편류를 랜덤워크로 모델링할 때의 잡음을 나타낸다.
관성항법 상태변수는 아래의 수학식 2와 같다. 아래의 수학식 3은 관성 센서 신호의 일반적인 모델이다.Also

,

Represent the specific force and angular velocity expressed in the body coordinate system, respectively,

Represents the gravity vector expressed in the reference coordinate system. The output of the accelerometer and gyroscope has a bias, and a drift occurs in which the bias varies with time.

,

Respectively denote the biases of the accelerometer and the gyroscope,

,

Represents the noise when modeling the bias drift of the accelerometer and the gyroscope as a random walk.
The inertial navigation state variable is shown in Equation 2 below. Equation 3 below is a general model of the inertial sensor signal.

&Quot; (2) "

&Quot; (3) "

,

은 각각 가속도계와 자이로스코프 센서 출력을 의미하고,

는 기준좌표계에서 표현된 선형가속도 벡터를 나타낸다. 또한

,

는 각각 가속도계와 자이로스코프의 센서 출력에 존재하는 영평균 가우시안 잡음을 나타낸다.
상기 항법 정보 예측부(520)는 상기 관성 항법 시스템 모델을 이용하여 오차 상태 변수 칼만필터(error-state Kalman filter, ESKF)를 구성하여 항법 정보를 예측한다.In Equation (3)

,

Respectively denote accelerometer and gyroscope sensor outputs,

Represents the linear acceleration vector expressed in the reference coordinate system. Also

,

Represents the zero mean Gaussian noise present in the sensor output of the accelerometer and gyroscope, respectively.
The navigation information predicting unit 520 constructs an error-state Kalman filter (ESKF) using the inertial navigation system model to predict navigation information.

The navigation information prediction of the navigation information predicting unit 520 is performed by constructing an error-state Kalman filter (ESKF) using the inertial navigation system model of Equation (1). The error state parameter Kalman filter is an indirect structure of the extended Kalman filter (EKF), which is a nonlinear estimator.

를 명목(nominal) 상태 변수와 오차(error) 상태 변수

로 나눈 후, 명목 상태 변수는 아래의 수학식 4의 비선형 시스템 모델

을 이용하여 전파하고, 오차 상태 변수는 아래의 수학식 5의 선형화 모델을 이용하여 추정한다. Error state variable The Kalman filter is a state variable

The nominal state variable and the error state variable

, The nominal state variable is the nonlinear system model

And the error state variable is estimated by using the linearization model of Equation (5) below.

&Quot; (4) "

&Quot; (5) "

는 시간 t_k 에서의 시스템 행렬이며, w _k는 시스템에 존재하는 영평균 가우시안 잡음을 나타낸다.
오차 상태 변수는 초기 값으로 0을 가지기 때문에 실제로는 오차 상태 변수의 불확실성만을 전파한다. 측정치 보정을 통하여 오차 상태 변수가 추정되면, 아래의 수학식 6과 같이 이를 명목 상태 변수에 보정한다. 이후 오차 상태 변수는 다시 0으로 초기화된다.Here, the subscript k denotes the value at time t _k from the discrete-time system. Also

Is the system matrix at time t _k , and w _k is the zero mean Gaussian noise present in the system.
Since the error state variable has an initial value of 0, it actually propagates only the uncertainty of the error state variable. If an error state variable is estimated through measurement correction, it is corrected to a nominal state variable as shown in Equation (6) below. The error state variable is then reset to zero.

&Quot; (6) "

Navigation information in the prediction block to propagate uncertainty in error state variables, such as Equation (7) below, using the covariance Q _k of w _k linearized model and the system error of the expression (5).

&Quot; (7) "

는 측정치 보정이 된 공분산을 의미한다.Where P _{k +1} is the covariance propagated using only the inertial sensor signal without measurement correction,

Means a covariance that has been subjected to measurement correction.

When the optical sensor signal is newly inputted, the matching point extracting unit 530 extracts the minutiae from the image newly entered, updates the minutiae point relation between the images in the sliding window by matching the minutiae point with the image input in the previous frame, Discard old video information.

Extraction and matching of feature points can be performed in various ways according to the user's environment. In the embodiment of the present invention, feature point extraction and matching are performed using FAST (features from accelerated segment test) corner extractor and KLT tracker (Kanade-Lucas-Tomasi feature tracker).

In addition, when the optical sensor signal newly comes in and the minutiae extraction unit 530 extracts and associates the minutiae point, the sliding window of the sliding update unit 540 is updated. According to the first input first output (FIFO) method, the update method separately stores position and attitude information and uncertainty when the latest optical sensor signal is input, and discards information associated with the oldest optical sensor signal.

는 아래의 수학식 8 및 수학식 9와 같다.

는 시간 i에서의 카메라 상태변수로서, 카메라의 기준좌표계 대비 위치와 자세를 나타내는 쿼터니언으로 이루어져 있다.Therefore, the information in the sliding window whose length is N at the current time k

Is expressed by the following equations (8) and (9).

Is a camera state variable at time i and consists of a quaternion representing the position and attitude of the camera relative to the reference frame of reference.

&Quot; (8) "

&Quot; (9) "

The state variable estimating unit 550 estimates the camera state variable through optimization based on the multiple geometry constraint between the state variables of the camera and the feature point matching relationship in the sliding window.

The multi-geometry constraint error minimization-based state variable estimation according to the present invention is performed using the state variable of Equation (9) and the feature point matching relationship in the sliding window. FIG. 4 is a graphical representation of state variable estimation based on minimization of re-projection error and state variable estimation based on minimization of multiple geometric constraint error.

Referring to FIG. 4, gray circles represent three-dimensional positions of feature points, orange squares represent camera state variables, black squares represent image feature points, and pink diamonds represent multiple geometry constraints.

As can be seen from FIG. 4, using the multiple geometry constraints, camera state variables can be estimated through optimization based on constraints between camera state variables, regardless of the three-dimensional position of the feature points.

The minimization of the multi-geometry constraint error-based state variable estimation can be performed by a Maximum-A-Posteriori (MAP) estimator as shown in Equation (10) below.

&Quot; (10) "

는 카메라 상태변수의 모음을 의미하고, Z는 모든 특징점 연관관계이다.here,

Denotes a collection of camera state variables, and Z denotes all feature point associations.

&Quot; (11) "

는 다중기하제약조건으로 묶일 수 있는 측정치의 조합을 의미하며,

는 관련된 상태변수의 조합을 의미한다. 여기서, N은 슬라이딩 윈도우의 길이, N_k 는 다중 기하 제약 조건의 조합의 수이다.Equation (10) is decomposed and modeled as shown in Equation (11), and then a cost function as shown in Equation (12) below is set using Negative Log-likelihood. In Equation (11)

Means a combination of measurements that can be grouped into multiple geometry constraints,

Means a combination of related state variables. Where N is the length of the sliding window and N _k is the number of combinations of multiple geometry constraints.

&Quot; (12) "

Here, the function h is Equation (13) which is a multi-geometry constraint error model.

&Quot; (13) "

는 시간 k에서의 카메라에서 특징점까지의 line-of-sight (LOS) 벡터이며,

은 시간 k의 카메라 좌표계 중심에서 시간 l의 카메라 좌표계 중심으로서의 이동 벡터이다.
수학식 12와 같이 비용 함수를 설정하면, 수학식 10의 MAP 문제의 해는 비선형 최소 자승법(nonlinear least square)을 이용하여 구할 수 있다.Here, the function h is Equation (13) which is a multi-geometry constraint error model. Each observation time is k, l, m,

Is a line-of-sight (LOS) vector from camera to feature at time k,

Is the motion vector as the center of the camera coordinate system at time l at the center of the camera coordinate system at time k.
If the cost function is set as shown in Equation (12), the solution of the MAP problem in Equation (10) can be obtained by using a nonlinear least squares method.

If the camera state variable is estimated optimally through the above procedure, the state variable of the inertial navigation system is corrected using this information as a measurement value. This process uses the update of the measurement of the Kalman filter with the error state variable. The state variable correcting unit (not shown) may be implemented as part of the state variable determining unit 510 and the state variable estimating unit 550 .

에 추가로, 시간 k와 s 사이에 존재하는 상대 움직임에 존재하는 거리환선계수

와 시간 s에서의 카메라 상태변수(위치, 자세)를 추정해야 하며, 이에 대한 측정치 모델은 수학식 14와 같다.Since the information obtained from the image system is all relative, the relative motion information between the camera state variables estimated above is extracted. Then, a measurement value update is performed using the nonlinear measurement model of Equation (14) below.
To this end,

In addition, the distance between the time k and the motion s

And the camera state variable (position, posture) at time s should be estimated, and the measured value model is expressed by Equation (14).

&Quot; (14) "

와

는 시간 k와 s 사이의 상대 위치 및 자세 차이를 의미하고, s는 0에서 N-1까지의 정수로 슬라이딩 윈도우에서의 위치를 의미한다.here,

Wow

Denotes a relative position and posture difference between time k and s, and s denotes an integer from 0 to N-1, which indicates a position in the sliding window.

Each parameter in the equation (14) is expressed by the following equation (15).

&Quot; (15) "

는 시간 k에서 측정된 기준좌표계에서의 카메라 위치이며,

는 시간 k에서 측정된 기준좌표계 대비 카메라의 자세를 의미하고,

는 IMU와 카메라 사이의 위치 차이를 동체좌표계에서 나타낸 것이다.
본 발명은 무인자동차에 응용 가능하다. 무인자동차에 장착된 단일 광학 센서와 내장되어 있는 관성 센서 시스템을 결합하여 GPS의 사용이 제한된 고층빌딩 지역에서도 정밀한 위치 인식이 가능하다. here,

Is the camera position in the reference coordinate system measured at time k,

Denotes the posture of the camera with respect to the reference coordinate system measured at time k,

Is the positional difference between the IMU and the camera in the body coordinate system.
The present invention is applicable to unmanned vehicles. By combining a single optical sensor mounted on an unmanned vehicle and an inertial sensor system, accurate positioning is possible even in high-rise building areas where the use of GPS is limited.

In order to verify the performance of the present invention, the IMU data of the KITTI Raw dataset and the monochrome image were used. KITTI dataset is a data set provided by Toyota Motor Corporation (TI) and KIT (Karlsruhe Institute of Technology) in Germany for performance benchmarking of image-based algorithms applicable to automobiles.

In this embodiment, performance verification is performed for data acquired through a city driving. In order to obtain the feature point information from the stored images, edges were extracted using FAST (Features from Accelerated Segment Test) technique and feature points were tracked in the sliding window using KLT (Kanade-Lucas-Tomasi) tracker. The tracked multiple feature point information is divided and stored in the sliding window, and at the same time, the position and attitude information calculated from the inertial sensor system is received and stored in the sliding window.

After the sliding window is complete, the position recognition is performed by performing optimization using multiple viewpoint geometry constraints at each moment. The implementation and analysis are implemented using Matlab of Mathworks. The result of the position recognition using the high-precision GPS and the system of the present invention is shown in FIG. 5.

As can be seen from FIG. 5, it can be confirmed that the actual data indicated by the green line and the location recognition result (indicated by the red line) according to the present invention are almost the same. These results demonstrate that the accuracy of the optical / inertial coupled position recognition based on the minimization of the multi-view constraint according to the present invention is high.

The multi-view constraint minimization-based optical / inertial coupled position recognition system proposed in the present invention performs optimization without estimating the three-dimensional position using the multi-view geometry constraint, so that the computation amount is smaller than the existing re- And a stronger estimation is also possible in the case of linear motion.

6 is a flow diagram of a sliding window-based non-structural location recognition method using inertial and single optical sensors in accordance with an embodiment of the present invention.

The sliding window-based non-structural location recognition method using inertial and single optical sensors according to the present embodiment may proceed in substantially the same configuration as the apparatus 50 of FIG. Therefore, the same components as those of the device 50 of FIG. 3 are denoted by the same reference numerals, and a repeated description thereof is omitted. Further, it can be executed by software (application) for performing sliding window-based non-structural position recognition using inertial and single optical sensors according to the present embodiment.

Referring to FIG. 6, a sliding window based non-structural position recognition method using an inertial and single optical sensor according to the present embodiment applies an inertial sensor signal to a continuous time inertial navigation system model, (Step S10).

Then, an error-state Kalman filter (ESKF) is constructed using the inertial navigation system model to predict navigation information (step S30).

The error state parameter Kalman filter is an indirect structure of the extended Kalman filter (EKF), which is a nonlinear estimator. The error state variable Kalman filter divides the state variable into a nominal state variable and an error state variable and then propagates the nominal state variable using the nonlinear system model of Equation (4) Estimate using the linearization model of Equation 5.

When the optical sensor signal is input, feature points are extracted from the newly input image, and feature points are extracted and matched between the images in the sliding window matching the extracted feature points with the image input in the previous frame (step S50).

Extraction and matching of feature points can be performed in various ways according to the user's environment. In the embodiment of the present invention, feature point extraction and matching can be performed using a FAST (feature from accelerated segment test) corner extractor and a KLT tracker (Kanade-Lucas-Tomasi feature tracker).

Further, when the optical sensor signal newly comes in and the minutiae point extraction and association are made, the sliding window is updated (step S70).

According to the first input first output (FIFO) method, the update method separately stores position and attitude information and uncertainty when the latest optical sensor signal is input, and discards information associated with the oldest optical sensor signal.

The camera state variable is estimated through optimization based on the multi-geometry constraint between the state variables of the camera and the feature point matching in the sliding window (step S70).

The multi-geometry constraint error minimization-based state variable estimation according to the present invention is performed using the state variable of Equation (9) and the feature point matching relationship in the sliding window.

Using multiple geometry constraints, it is possible to estimate camera state variables through optimization based on constraints between camera state variables, regardless of the 3D location of feature points.

The state variable of the inertial navigation system model is corrected using the estimated camera state variable as a measurement value (step S90). This process uses the update of the measurement of the Kalman filter with the error state variable.

The multi-view constraint minimization-based optical / inertial coupled position recognition system proposed in the present invention performs optimization without estimating the three-dimensional position using the multi-view geometry constraint, so that the computation amount is smaller than the existing re- And a stronger estimation is also possible in the case of linear motion.

Such sliding window based non-structural location recognition methods using inertial and single optical sensors may be implemented in applications or implemented in the form of program instructions that may be executed through various computer components and recorded on a computer readable recording medium have. The computer-readable recording medium may include program commands, data files, data structures, and the like, alone or in combination.

The program instructions recorded on the computer-readable recording medium are those specially designed and constructed for the present invention and may also be those known and used by those skilled in the computer software field.

Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tape, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like.

Examples of program instructions include machine language code such as those generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware device may be configured to operate as one or more software modules for performing the processing according to the present invention, and vice versa.

It will be apparent to those skilled in the art that various modifications and variations can be made in the present invention without departing from the spirit or scope of the present invention as defined by the following claims. You will understand.

The present invention relates to the construction of a location recognition system, and provides a location and an attitude of an object in the progress of technology development related to virtual reality and augmented reality. This technology can be used not only for autonomous navigation devices but also for all industries such as movie, advertisement, animation, game, medicine, etc., and can be directly used with virtual reality and augmented reality as commercialization progresses It is expected.

1: Multi-view constraint minimization based optical / inertial coupled position recognition system
10: inertia sensor
30: Optical sensor
50: Sliding window based non-structural position recognition device using inertial and single optical sensor
510: state variable determining unit
520: navigation information prediction unit
530: matching relation extracting unit
540: Sliding update unit
550: state variable estimating unit

Claims (10)

Applying the inertial sensor signal to a continuous time inertial navigation system model and propagating it over time to determine an inertial navigation state variable;
Estimating navigation information by constructing an error-state Kalman filter (ESKF) using the inertial navigation system model;
Extracting a feature point from a newly input image and performing feature point extraction and matching between images in a sliding window matching the extracted feature points with an image input in a previous frame;
Updating a sliding window;
Estimating a camera state variable through optimization based on a multiple geometry constraint between state variables of a camera and a feature point matching relationship in a sliding window; And
And using the estimated camera state variable as a measure to correct the state variable of the inertial navigation system model.
The method of claim 1, wherein the step of predicting the navigation information comprises constructing an error-state variable Kalman filter (ESKF) using the inertial navigation system model,
Dividing the state variable into a nominal state variable and an error state variable;
Propagating the nominal state variable using a nonlinear system model; And
A sliding window based non-structural location recognition method using inertial and single optical sensors, comprising the step of estimating error state variables using a linearization model.
2. The method of claim 1, wherein updating the sliding window comprises:
Storing position and attitude information and uncertainty of the latest optical sensor signal in time; And
And discarding information associated with the oldest optical sensor signal in time. ≪ RTI ID = 0.0 > A < / RTI >
2. The method of claim 1, wherein the extracting and matching of feature points between images in the sliding window comprises:
A sliding window based non-structural location recognition method using an inertial and single optical sensor using a FAST (Correction Segment Test) corner extractor and a Kanade-Lucas-Tomasi feature tracker.
A computer-readable recording medium on which a computer readable recording medium for performing a sliding window-based non-structural location recognition method using inertial and single optical sensors according to any one of claims 1 to 4.
A state variable determining unit that applies an inertial sensor signal to a continuous time inertial navigation system model and propagates the signal based on time to determine an inertial navigation state variable;
A navigation information predicting unit for predicting the navigation information by constructing an error-state Kalman filter (ESKF) using the inertial navigation system model;
A matching relation extracting unit for extracting a feature point from a newly input image and performing feature point extraction and matching between images in a sliding window matching the extracted feature points with an image input in a previous frame,
A sliding update unit updating the sliding window;
A state variable estimator for estimating camera state variables through optimization based on multiple geometry constraints between camera state variables and feature point mappings in a sliding window; And
And a state variable correction unit for correcting the state variable of the inertial navigation system model using the estimated camera state variable as a measurement value.
7. The navigation system according to claim 6,
Inertial and single optical sensors, which divide state variables into nominal and error state variables, nominal state variables propagate using a nonlinear system model, and error state variables are estimated using a linearization model, A sliding window based non - structural location - aware device.
7. The apparatus of claim 6,
A sliding window based non-structural position sensing device using an inertial and single optical sensor that stores position and orientation information and uncertainty of the most up-to-date optical sensor signal in time and discards information associated with the oldest optical sensor signal in time.
7. The apparatus according to claim 6, wherein the matching-
A sliding window-based non-structural location-aware device using an inertial and single optical sensor, using FAST (Corresponding Accelerated Segment Test) corner extractor and KLT tracker (Kanade-Lucas-Tomasi feature tracker).
The method according to claim 6,
Wherein the inertial sensor signal comprises accelerometer data of a three-dimensional vector and gyro data.

Images