KR102226846B1 - System for Positioning Hybrid Indoor Localization Using Inertia Measurement Unit Sensor and Camera - Google Patents

Abstract

The hybrid indoor positioning system using the IMU sensor and camera combines the position results obtained from the IMU sensor and the camera to reduce the IMU sensor error that occurs in the IMU sensor-based positioning by the camera pose, which is the camera-based positioning. By reducing the azimuth error that occurs as a result of the IMU positioning, there is an effect of minimizing the position error in indoor positioning.

Description

System for Positioning Hybrid Indoor Localization Using Inertia Measurement Unit Sensor and Camera}

The present invention relates to an indoor positioning system, and in particular, by combining an IMU sensor and a position result obtained from a camera, an IMU sensor error occurring in an IMU sensor-based positioning is reduced by a camera pose, which is a camera positioning result, and in the camera-based positioning. The present invention relates to a hybrid indoor positioning system using an IMU sensor and a camera that can reduce the azimuth error that occurs as a result of IMU positioning.

Personal Navigation System (PNS) refers to all systems for finding a person's location. GPS is mainly used as one of the representative methods of the personal navigation system, but it becomes impossible to use GPS alone in an indoor environment or in a signal shaded area such as a tunnel.

One of the methods to solve this problem, the PDR (Pedestrian Dead-Reckoning) system, is a speculative navigation system developed under the assumption that the position is changed by the walking of a pedestrian.

The PDR system can know how far the pedestrian has moved from the initial position by using the step information generated when walking. At this time, step information such as stride length, direction, posture, and height is obtained using an inertia measurement unit (IMU).

However, the IMU-based position measurement system provides a certain level of position accuracy, but the accuracy varies depending on the magnitude of sensor error caused by external electromagnetic noise or sensor drift, accelerometer accumulated error, gyroscope drift error, There is a problem in that the position error increases due to an external magnetic field that affects the magnetometer.

Korean Patent Registration No. 10-1160132

In order to solve such a problem, the present invention reduces the IMU sensor error occurring in the IMU sensor-based positioning by combining the position result obtained from the IMU sensor and the camera by the camera pose, which is the camera positioning result, and An object of the present invention is to provide a hybrid indoor positioning system using an IMU sensor and a camera that can reduce the azimuth error that occurs as a result of IMU positioning.

Hybrid indoor positioning system according to a feature of the present invention for achieving the above object,

By using sensor signals received from IMU (Inertia Measurement Unit) sensors including a 3-axis accelerometer, a 3-axis magnetometer, and a 3-axis gyroscope, the step and direction angle information of the pedestrian is calculated, and the step and direction angle information are used. A first indoor positioning device for calculating a first position;

Detects a plurality of feature points in a plurality of image frames photographed by a camera, performs matching of a pre-stored key frame, estimates posture change information of the camera using the matching result, and calculates a second posture change information based on the posture change information. A second indoor positioning device for positioning a position; And

And a sensor fusion unit for calculating a final position of a pedestrian by combining the first position calculated from the first indoor positioning device and the second position calculated from the second indoor positioning device using a linear Kalman filter. do.

The first indoor positioning device detects a pedestrian's step by analyzing a change in magnitude of acceleration and angular velocity using each sensor signal of the 3-axis accelerometer and the 3-axis gyroscope, and uses the detected step to detect the pedestrian's step. The step length is estimated, direction angle information is calculated by fusing the sensor signals of the 3-axis magnetometer and the 3-axis gyroscope, and the first position of the pedestrian is calculated based on the calculated direction angle information and the step length. Characterized in that.

The second indoor positioning device receives a plurality of image frames photographed from the camera, detects a plurality of feature points in a frame unit, matches the detected feature points with a key frame, and determines each of the matched feature points. Accumulating in a key frame to update the key frame, calculating posture change information of the camera through feature point tracking between the key frame and the updated key frame, and positioning the second position based on the posture change information It is characterized by that.

The sensor fusion unit performs position correction by combining the first position and the second position by the linear Kalman filter, and then calculates the position of the pedestrian on the map by matching the current position of the pedestrian to the map. Characterized

According to the above-described configuration, the present invention combines the position results obtained from the IMU sensor and the camera to reduce the IMU sensor error occurring in the IMU sensor-based positioning by the camera pose, which is the camera positioning result, and is generated in the camera-based positioning. By reducing the azimuth error as a result of the IMU positioning, there is an effect of minimizing the position error in indoor positioning.

1 is a block diagram showing the configuration of a hybrid indoor positioning system using an IMU (Inertia Measurement Unit) sensor and a camera according to an embodiment of the present invention.
2 is a diagram showing a result of IMU-based positioning.
3 is a diagram showing a camera-based positioning result.
4 is a view showing the experimental results obtained in the positioning of the hybrid indoor positioning system of the present invention.
5 is a diagram showing an RMSE of an IMU-based positioning approach.
6 is a diagram illustrating an RMSE of a camera-based positioning approach.
7 is a diagram showing the RMSE of the positioning approach of the hybrid indoor positioning system of the present invention.
8 is a diagram showing IMU-based positioning, camera-based positioning, and a CDF of hybrid positioning of the present invention.

Hereinafter, the present invention will be described in detail by describing a preferred embodiment of the present invention with reference to the drawings.

Throughout the specification of the present invention, when a certain part "includes" a certain component, it means that other components may be further included rather than excluding other components unless otherwise stated.

1 is a block diagram showing the configuration of a hybrid indoor positioning system using an IMU (Inertia Measurement Unit) sensor and a camera according to an embodiment of the present invention.

The hybrid indoor positioning system 100 according to an embodiment of the present invention largely includes a first indoor positioning device 110 using an IMU sensor, a second indoor positioning device 120 using a camera, and a sensor fusion unit 130. .

The first indoor positioning device 110 of the present invention performs a function of measuring inertia data according to the movement of a moving object.

The first indoor positioning device 110 includes IMU sensors 111, a posture determination unit 112, a step detection unit 113, a step length estimation unit 114, a magnetometer azimuth calculation unit 115, a gyroscope azimuth calculation unit 116, a direction filter unit 117, and a current position calculation unit 118.

The IMU sensors 111 include an accelerometer 111a, a magnetometer 111b, and a gyroscope 111c.

The accelerometer 111a measures any sensor, circuit, and/or other motion and/or acceleration of the mobile computing device along each of the three-dimensional axes.

The accelerometer 111a measures three-axis acceleration data of a pedestrian.

The magnetometer 111b is configured to measure the angular direction for any sensor, circuit and/or predefined coordinate system.

The gyroscope 111c may measure a 3-axis roll, a 3-axis pitch, and/or a 3-axis yaw of a pedestrian.

The gyroscope 111c measures the three-axis angular velocity data of the pedestrian.

The posture determination unit 112 receives the 3-axis acceleration data from the accelerometer 111a, and calculates the pitch angle and the roll angle of the received 3-axis acceleration data by Equation 1 below.

The posture determination unit 112 receives the 3-axis angular velocity data from the gyroscope 111c, and calculates the pitch angle and the roll angle of the received 3-axis angular velocity data by Equation 1 below.

The posture determination unit 112 calculates the fusion pitch angle and the fusion roll angle by combining the pitch angle and roll angle of the three-axis acceleration data and the pitch angle and the roll angle of the three-axis angular velocity data using a Kalman filter.

The posture determination unit 112 may remove a cumulative error of the accelerometer 111a and a drift error of the gyroscope 111c by combining the pitch angle and roll angle of acceleration data, and the pitch angle and roll angle of angular velocity data. For this reason, the accelerometer 111a can maintain a limited error range.

The Kalman filter is used to combine the pitch angle and roll angle of the accelerometer 111a and the gyroscope 111c.

는 Pitch의 각,

는 Roll의 각,

는 IMU 센서의 3축 가속도 데이터, g는 중력 가속도를 나타낸다.here,

Is the angle of the pitch,

Is the angle of the roll,

Is the 3-axis acceleration data of the IMU sensor, and g is the gravitational acceleration.

The expression used for calculating the pitch and roll angles from the accelerometer 111a is valid only in situations where the acceleration or angular velocity is sufficiently small.

When considering the pitch of the gyroscope 111c and the angle of the roll, errors are accumulated according to the numerical integration process. In order to remove the error of the accelerometer 111a and the gyroscope 111c, a sensor fusion algorithm in which the first indoor positioning device 110 and the second indoor positioning device 120 are mixed is proposed.

The step detection unit 113 detects a step (step) by analyzing the fusion pitch angle obtained by combining the pitch angle of acceleration data and the pitch angle of angular velocity data from the posture determination unit 112. That is, the step detection unit 113 detects a step by analyzing a change in magnitude of acceleration and angular velocity.

The step length estimating unit 114 receives the fusion pitch angle obtained by combining the pitch angle of acceleration data and the pitch angle of angular velocity data from the step detection unit 113, and uses the received fusion pitch angle as a linear regression model of Equation 2 below. The step length SL is estimated based on.

는 degree 단위의 피치 진폭이고, 상수 a와 b는 각 회귀선에 맞는 매개 변수이다.here,

Is the pitch amplitude in degrees, and the constants a and b are parameters for each regression line.

The parameter used to estimate the position of the IMU sensor is the azimuth angle estimation. The azimuth estimation is the process of determining the pedestrian's moving direction. The magnetometer 111b and the gyroscope 111c are used for azimuth estimation.

If the magnetometer 111b is used, high positioning accuracy can be obtained through a long experiment. However, since the azimuth angle of the magnetometer 111b is easily affected by an external magnetic field, reliability is poor.

Estimation of the azimuth angle of the gyroscope 111c is used in a short time experiment because accuracy is poor in a long time experiment due to a drift error.

The present invention shows better performance in short and long experiments by using complementary functions of the magnetometer 111b and the gyroscope 111c.

에서의 자력과 글로벌(Glbal)

좌표계 간의 관계는 다음의 수학식 3과 같이 표현된다.Device

Magnetic and Global (Glbal)

The relationship between the coordinate systems is expressed as Equation 3 below.

는 Pitch의 각,

는 Roll의 각을 나타낸다.here,

Is the angle of the pitch,

Represents the angle of the roll.

)을 다음의 수학식 4와 같이 표현한다.Magnetometer azimuth calculation unit 115 is the azimuth (

) Is expressed as in Equation 4 below.

Here, Hx and Hy represent the azimuth magnetic force of the global coordinate system.

The magnetometer azimuth calculation unit 115 receives direction information for the pedestrian's coordinate system from the magnetometer 111b, and the fusion pitch angle obtained by combining the pitch angle of acceleration data and the pitch angle of angular velocity data from the attitude determination unit 112, and angular velocity data. The fusion roll angle that combines the roll angle and the roll angle of angular velocity data is received and used as a parameter when calculating the absolute azimuth angle.

는 바디 프레임 X축 자이로스코프(111c)의 출력,

는 바디 프레임 Y축 자이로스코프(111c)의 출력,

는 바디 프레임 Z축 자이로스코프(111c)의 출력을 나타낸다.The azimuth angle of the gyroscope 111c has a lower azimuth error when compared with the magnetometer 111b through a short time experiment.

Is the output of the body frame X-axis gyroscope 111c,

Is the output of the body frame Y-axis gyroscope 111c,

Represents the output of the body frame Z-axis gyroscope 111c.

The gyroscope azimuth calculation unit 116 receives the three-axis angular velocity data from the gyroscope 111c and uses the pitch angle and the roll angle to display the direction in which the pedestrian is located in the three-dimensional space. Calculate the angle of.

는 각 Roll, Pitch, Yaw,

는 각 x, y,z 바디 프레임 축의 출력을 나타낸다.here,

Is each Roll, Pitch, Yaw,

Represents the output of each x, y, z body frame axis.

The gyroscope azimuth calculation unit 116 updates the quaternion according to the Euler angle by using the value of the quaternion to the calculated Euler angle as shown in Equation 6 below.

는 각 Roll, Pitch, Yaw의 오일러 각도,

는 사원수의 값을 나타낸다.here,

Is the Euler angle of each Roll, Pitch, Yaw,

Represents the value of the quaternion.

The direction filter unit 117 combines the absolute azimuth angle calculated by the magnetometer azimuth angle calculation unit 115 and the angular velocity information calculated by the gyroscope azimuth angle calculation unit 116 by the Kalman filter of Equation 7 below, Calculate the direction information (Heading _gyro).

The direction filter unit 117 may calculate the azimuth angle of the gyroscope from the quaternion obtained through the Euler angle. In other words, the direction filter unit 117 calculates the final direction of the pedestrian through the sum of the directions of the magnetometer 111b and the directions of the gyroscope 111c.

는 업데이트된 사원수의 값을 나타낸다.here,

Represents the updated quaternion value.

In order to compensate for the sensor error of the magnetometer 111b and the gyroscope 111c, the gyroscope azimuth angle is combined with the magnetometer azimuth angle to provide better results compared to the results of the individual sensor azimuth angles. The Kalman filter is used for azimuth fusion of the magnetometer 111b and the gyroscope 111c.

The azimuth fusion result and step length information are used for position estimation. The current position of the pedestrian is estimated by the azimuth angle from the previously known position, step length information, and step interval. The initial position of the pedestrian is defined as 0, and the step length and azimuth information estimate the current position of the pedestrian based on the initial value.

The current position calculation unit 118 receives the final direction information of the pedestrian from the direction filter unit 117 and receives the step length SL estimated from the step length estimation unit 114.

The current position calculation unit 118 calculates the current position of the pedestrian by using the final direction information of the pedestrian, the step length, and the initial position value, as shown in Equation 8 below.

Here, Xt-1 and Yt-1 are the initial position values, Xt and Yt are the pedestrian's position values, SL is the step length, and Heading _fusion is the azimuth angle of the fusion of the magnetometer and the gyroscope.

The second indoor positioning device 120 of the present invention includes a smartphone camera 121, a tracking unit 122, a place recognition unit 123, a map database unit 124, and a local mapping unit ( Local Mapping 125 and Loop Closing 126 are included.

The second indoor positioning device 120 relates to a method for real-time location recognition and mapping of pedestrians based on an image, and applies an Oriented Fast Rotated BRIEF (ORB)-Simultaneous Localization and Mapping (SLAM) system.

The Simultaneous Location Recognition and Mapping (SLAM) method is a technology that restores the surrounding environment of a pedestrian with a sensor such as a camera into a 3D model and estimates the pedestrian's position in a 3D space at the same time.

The smartphone camera 121 acquires a plurality of image frames by photographing at 10 frames per second (FPS) at a predetermined size using a lens.

The tracking unit 122 extracts Oriented Fast Rotated BRIEF (ORB) keypoints from the new image of each frame.

The tracking unit 122 receives a plurality of frames and detects a feature point in units of frames.

The tracking unit 122 tracks the feature points by matching the feature points of the previous frame and the feature points of the current frame. In order to track the feature points between frames, the tracking unit 122 matches the feature points in the current frame when they exist within a window of a predetermined size from the coordinates of the previous frame feature points.

The tracking unit 122 matches the detected feature point and the key frame, and updates the key frame by accumulating the matched feature point in the key frame.

The tracking unit 122 sets the feature points of the first frame as a key frame, and when the feature points of the frames following the first frame match, accumulates the matched feature points in the key frame and repeatedly executes the key frame to be updated. Frames can be extracted.

The tracking unit 122 calculates a posture change by tracking feature points between the key frame or the updated key frame and the second frame or the subsequently input frame.

When the number of feature points tracked from the previous frame is less than a certain number, the tracking unit 122 adds a new key frame for tracking new feature points.

The tracking unit 122 estimates the camera position in every frame and controls input of a new key frame.

Afterwards, BA (Motion only Bunle Adjust) is used to optimize the camera pose using the initial characteristics that match the previous frame.

The map database unit 124 stores map points or map points of a 3D ORB. Each map point corresponds to a patch of texture planes triangulated in different views and modified with bundle adjustments.

The map point corresponds to the ORB image of the image, so it is a triangulation of the FAST edge. Map points can be associated with ORB functions in multiple keyframes.

Keyframes are snapshots that summarize actual visual information. Each keyframe stores all ORB functions in a frame and a camera pose.

The visibility graph determines a local map or improves the performance of the location recognition unit 123.

In the visibility graph, when there are map points in which two key frames are common, each node representing a key frame and an edge is stored.

For each visual word, the location recognition unit 123 includes a vocabulary in which a corresponding key frame is displayed and a reversal index in which the weight of the key frame is stored. Accordingly, it is possible to perform a database query very efficiently by checking only general visual words. have.

After the local mapping process is completed, the location recognition unit 123 inserts a new key frame into the map database unit 124 by a loop closure operation.

When querying the database, the location recognition unit 123 calculates a similarity score between all keyframes that share the query image and the visual word.

The location recognition unit 123 queries the database and then calculates camera poses and similarity transformations, respectively, and uses them as geometric validity checks.

The location recognition unit 123 determines that the two consecutive keyframes are consistent and there is an edge in the visibility graph connecting the two consecutive keyframes.

When the initialization of the camera pose is completed through the feature point tracking, the tracking unit 122 searches for a local map in the map database unit 124 using a covisibility graph having a key frame. Afterwards, it matches the local map points found by reprojection, and finally optimizes the camera pose using all matches.

The local mapping unit 125 obtains an optimal reconstruction related to a camera pose by performing a local BA using a new key frame.

The local mapping unit 125 stores a new keyframe received from the tracking unit 122, and generates new map points by triangulating ORB feature points in different keyframes.

The local mapping unit 125 triangulates the ORB feature points in N adjacent keyframes in the visibility graph sharing most of the map points.

The local mapping unit 125 performs local bundle optimization for optimizing the currently processed keyframes, all keyframes connected in the visibility graph, and all map points visible in the keyframes (Local Bundle Adjustment).

The loop closing unit 126 retrieves the keyframe processed by the local mapping unit 125, searches for a loop with a new keyframe on the map, and closes the loop from the current keyframe to the keyframe of the loop. A similarity transform that provides information on the accumulated drift of is calculated.

In other words, the loop closing unit 126 compares the previous keyframes with the current keyframes in the map database unit 124 to search for a keyframe having a degree of similarity greater than a predetermined threshold. When a key frame with a similarity greater than a predetermined threshold value is detected, the loop closing unit 126 sets them as loop closures, optimizes the graph of the camera pose based on the posture change information of the set loop closure, and optimizes the graph. Based on this, location recognition and map creation (SLAM) are performed.

The indoor positioning system 100 of the present invention combines the first indoor positioning device 110 and the second indoor positioning device 120 to minimize the position measurement error. It is to reduce the sensor error and the azimuth error of the second indoor positioning device 120 based on the camera sensor.

The sensor fusion unit 130 combines the first position measurement result of the first indoor positioning device 110 based on the IMU sensor and the second position measurement result of the second indoor positioning device 120 based on the camera sensor, Calculate location information.

The sensor error of the first indoor positioning device 110 is compensated by the position measurement result of the second indoor positioning device 120, and the azimuth error of the second indoor positioning device 120 is the position of the first indoor positioning device 110 It is supplemented with the measurement result to improve the location measurement accuracy.

The sensor fusion unit 130 combines the first position measurement result of the first indoor positioning device 110 and the second position measurement result of the second indoor positioning device 120 using a linear Kalman filter.

The sensor fusion unit 130 uses a Kalman filter algorithm having a movement model of Equation 9 and an observation model of Equation 10 below.

는 보행자 위치의 2차원 좌표이고,

는 시스템에 입력되는 IMU 센서의 x,y값(보행자의 2차원 좌표 정보), A와 B는 단위 행렬이고, W는 zero mean을 갖는 동작 모델이고, 공분산 행렬 Q의 가우시안 잡음을 나타낸다.Equation 9 is a function of the transition state of the sensor fusion,

Is the two-dimensional coordinate of the pedestrian's position,

Is the x,y values (two-dimensional coordinate information of the pedestrian) of the IMU sensor input to the system, A and B are the identity matrices, W is the motion model with zero mean, and represents the Gaussian noise of the covariance matrix Q.

는 zero mean과 공분산 행렬 R을 갖는 IMU 기반의 제1 실내 측위 장치(110)의 위치 측정 결과의 가우시안 잡음을 나타낸다. 센서 융합은 직접적인 관측 문제이기 때문에 H를 항등 행렬로 사용한다.Equation 10 can be obtained through the position measurement result of the first indoor positioning device 110 as an observation model,

Denotes Gaussian noise of a position measurement result of the IMU-based first indoor positioning device 110 having a zero mean and a covariance matrix R. Since sensor fusion is a direct observation problem, we use H as the identity matrix.

The linear Kalman filter is a set of mathematical equations that provide an efficient means of calculation by minimizing positional errors to track the state of a process. The linear Kalman filter has excellent signal-to-noise separation and can prevent instantaneous position jumps, and is used to stably detect accurate position, speed, and time.

The linear Kalman filter algorithm consists of two processes: a time update equation for prediction and a measurement update equation.

The time update process of the linear Kalman filter forwards the current state estimation result to a prediction step (Predict) of the Kalman filter that predicts the current state in advance in advance of time.

The time update process of the linear Kalman filter includes a linear estimation equation of Equation 11 below and a Prior tracking error covariance equation of Equation 12 below.

는 시스템에 입력되는 IMU 센서의 x,y값(보행자의 2차원 좌표 정보),

는 예측값을 나타낸다.Where A and B are identity matrices,

Is the x,y value of the IMU sensor input to the system (two-dimensional coordinate information of the pedestrian),

Represents the predicted value.

는 공분산 행렬, T는 전치행렬,

는 에러 공분산 값을 나타낸다.here,

Is the covariance matrix, T is the transposed matrix,

Represents the error covariance value.

The measurement update process of the linear Kalman filter is a correction step of the Kalman filter that corrects the estimated state values transmitted by the actual measurement at the corresponding time.

The measurement update process of the linear Kalman filter includes the Kalman gain generation equation of Equation 13 below, the updated estimate generation equation of Equation 14 below, and the Posteriori tracking error covariance equation of Equation 15 below.

Here, H is an identity matrix, and R is a covariance matrix.

는 관찰 함수를 나타낸다.here,

Represents the observation function.

All variables used in the algorithm of the linear filter are summarized in Table 1 below.

The sensor fusion unit 130 may calculate the position of the pedestrian on the map by matching the current position of the pedestrian with a map after performing position correction through a linear Kalman filter.

In the hybrid indoor positioning system 100 of the present invention, data was collected in IT1 Building 512 of Kyungpook National University by using an IMU sensor and a camera in order to evaluate the performance and accuracy of indoor positioning. In order to collect data, the user held a smartphone and an IMU sensor in his hand and conducted an experiment at the experiment site.

This experiment was conducted with an Exynos 7420 processor and 3GB of RAM on the Android 5.0.2 Lollipop platform mounted on the Samsung Galaxy S6 Edge smartphone.

A Biscuit Programmable Wi-Fi 9-Axis absolute orientation sensor was used for IMU positioning. The length of the test area is 3.1m and the width is 6m.

FIG. 2 is an IMU-based positioning result, FIG. 3 is a camera-based positioning result, and FIG. 4 shows an experimental result obtained from positioning of the hybrid indoor positioning system 100 of the present invention.

As shown in Fig. 4, it can be seen that the influence of sensor error is significantly reduced in the hybrid positioning result of the present invention. In the IMU-based positioning result, there is a cumulative error of the accelerometer 111a and a deflection error of the gyroscope 111c.

The camera-based positioning result has a problem that it cannot be accurately estimated when a pedestrian changes direction.

The hybrid indoor positioning system 100 of the present invention shows higher accuracy when comparing the IMU method and the camera method.

The positioning accuracy of the hybrid indoor positioning system 100 of the present invention is evaluated using a Root Mean Square Error (RMSE) and a cumulative error distribution function of the positioning errors.

5 to 7 show IMU-based positioning, camera-based positioning, and RMSE results of the hybrid positioning method of the present invention.

5 shows an RMSE of an IMU-based positioning approach. The maximum error of the IMU-based positioning approach is 0.7467m.

6 shows a positioning error in a camera-based positioning approach. The camera-based positioning approach has a positioning error of up to 1m compared to the actual position value.

7 shows the RMSE for the hybrid positioning approach of the present invention. The maximum positional error of the hybrid positioning approach of the present invention is 0.4924m compared to the actual position. In the positioning error results, the proposed hybrid positioning approach shows better results than the IMU-based and camera-based positioning approaches.

Table 2 below shows the performance analysis of different positioning approaches.

In Table 2, the hybrid positioning method of the present invention has fewer positioning errors compared to the IMU and camera-based positioning method. The hybrid positioning method provides higher positioning accuracy compared to other positioning approaches. The accuracy of the hybrid approach of the present invention is analyzed by Cumulative Distribution Functions (CDF) of positioning errors. 8 shows IMU-based positioning, camera-based positioning, and a CDF of hybrid positioning of the present invention.

Looking at the CDF graph, the hybrid positioning method of the present invention can reduce the positional error and obtain better performance compared to the IMU and camera-based positioning method.

The IMU-based positioning method is superior to the camera-based positioning method.

As a result of error analysis, the camera-based positioning method is affected by camera pose errors and the indoor positioning accuracy is degraded. In all experimental results analysis, the hybrid positioning approach of the present invention provides reasonable positioning accuracy for indoor applications.

The present invention proposes a hybrid indoor positioning using an IMU sensor and a smartphone camera 121. The IMU sensor error occurring in the IMU sensor-based positioning is reduced by the camera pose, which is a camera positioning result, and the azimuth error occurring in the camera-based positioning is reduced by the IMU positioning result.

The hybrid positioning method of the present invention further improves the positioning result compared to the individual positioning method. The present invention proposes a sensor fusion framework based on a Kalman filter for hybrid positioning.

Although the embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements by those skilled in the art using the basic concept of the present invention defined in the following claims are also provided. It belongs to the scope of rights.

100: hybrid indoor positioning system 110: first indoor positioning device
111: IMU sensors 111a: accelerometer
111b: magnetometer 111c: gyroscope
112: posture determination unit 113: step detection unit
114: step length estimation unit 115: magnetometer azimuth calculation unit
116: gyroscope azimuth calculation unit 117: direction filter unit
118: current position calculation unit 120: second indoor positioning device
121: smartphone camera 122: tracking unit
123: location recognition unit 124: map database unit
125: local mapping unit 126: loop closing unit
130: sensor fusion unit

Claims (16)

By using sensor signals received from IMU (Inertia Measurement Unit) sensors including a 3-axis accelerometer, a 3-axis magnetometer, and a 3-axis gyroscope, the step and direction angle information of the pedestrian is calculated, and the step and direction angle information are used. A first indoor positioning device for calculating a first position;
Detects a plurality of feature points in a plurality of image frames photographed by a camera, performs matching of a pre-stored key frame, estimates posture change information of the camera using the matching result, and calculates a second posture change information based on the posture change information. A second indoor positioning device for positioning a position; And
And a sensor fusion unit for calculating a final position of a pedestrian by combining the first position calculated from the first indoor positioning device and the second position calculated from the second indoor positioning device using a linear Kalman filter. Hybrid indoor positioning system. The method of claim 1,
The first indoor positioning device detects a pedestrian's step by analyzing a change in magnitude of acceleration and angular velocity using each sensor signal of the 3-axis accelerometer and the 3-axis gyroscope, and uses the detected step to detect the pedestrian's step. The step length is estimated, direction angle information is calculated by fusing the sensor signals of the 3-axis magnetometer and the 3-axis gyroscope, and the first position of the pedestrian is calculated based on the calculated direction angle information and the step length. Hybrid indoor positioning system, characterized in that. The method of claim 1,
The second indoor positioning device receives a plurality of image frames photographed from the camera, detects a plurality of feature points in frame units, matches the detected plurality of feature points and key frames, and determines each of the matched feature points. Accumulating in a key frame to update the key frame, calculating posture change information of the camera through feature point tracking between the key frame and the updated key frame, and positioning the second position based on the posture change information Hybrid indoor positioning system, characterized in that. The method of claim 1,
The sensor fusion unit performs position correction by combining the first position and the second position by the linear Kalman filter, and then calculates the position of the pedestrian on the map by matching the current position of the pedestrian to the map. Hybrid indoor positioning system characterized by. The method of claim 1,
The first indoor positioning device receives 3-axis acceleration data from the accelerometer, calculates a pitch angle and a roll angle of the received 3-axis acceleration data by Equation 1 below, and calculates 3-axis angular velocity data from the gyroscope. After receiving, calculate the pitch angle and roll angle of the received three-axis angular velocity data by the following equation (1), and the pitch angle and roll angle of the three-axis acceleration data and the pitch angle and roll angle of the three-axis angular velocity data are first Kalman Hybrid indoor positioning system, characterized in that it further comprises a posture determination unit for calculating the fusion pitch angle and the fusion roll angle by combining using a filter.
[Equation 1]

here,

Is the angle of the pitch,

Is the angle of the roll,

Is the 3-axis acceleration data of the IMU sensor, and g is the gravitational acceleration. The method of claim 5,
The first indoor positioning device includes: a magnetometer azimuth angle calculator configured to receive direction information on a coordinate system of a pedestrian from the magnetometer, and receive the fusion pitch angle and the fusion roll angle from the posture determination unit to calculate an absolute azimuth angle; And
In order to receive the 3-axis angular velocity data from the gyroscope and display the direction in which the pedestrian is located in a 3D space, a gyroscope azimuth calculation unit calculates the Euler angle of Equation 2 below using the pitch angle and the roll angle. Hybrid indoor positioning system, characterized in that.
[Equation 2]

here,

Is each Roll, Pitch, Yaw,

Is the output of each x, y, z body frame axis. The method of claim 6,
The gyroscope azimuth angle calculator performs an update of a quaternion according to the Euler angle by using the calculated Euler angle as a quatern value according to Equation 3 below.
[Equation 3]

here,

Is the Euler angle of each Roll, Pitch, Yaw,

Is the value of the number of employees. The method of claim 6,
_{The first indoor positioning device calculates the final direction information (Heading gyro} ) of the pedestrian by combining the absolute azimuth angle calculated by the magnetometer azimuth angle calculation unit and the angular velocity information calculated by the gyroscope azimuth angle calculation unit by a second Kalman filter. Hybrid indoor positioning system, characterized in that it further comprises a directional filter unit. The method of claim 8,
The first indoor positioning device further comprises a direction filter unit for calculating _{the final direction information (Heading gyro} ) of the pedestrian by combining the absolute azimuth angle and the angular velocity information by Equation 4 below. .
[Equation 4]

here,

Is the value of the updated number of employees. The method of claim 5,
The first indoor positioning device receives a fusion pitch angle obtained by combining a pitch angle of the acceleration data and a pitch angle of angular velocity data, and uses the received fusion pitch angle to step length based on the linear regression model of Equation 5 below. Hybrid indoor positioning system, characterized in that it further comprises a step length estimating unit for estimating (SL).
[Equation 5]

here,

Is the pitch amplitude in degrees, and the constants a and b are parameters for each regression line. The method of claim 5,
The first indoor positioning device includes: a magnetometer azimuth angle calculator configured to receive direction information on a coordinate system of a pedestrian from the magnetometer and receive the fusion pitch angle and the fusion roll angle from the posture determination unit to calculate an absolute azimuth angle;
A gyroscope azimuth calculation unit for receiving 3-axis angular velocity data from the gyroscope and calculating angular velocity information indicating a direction in which a pedestrian is located in a 3D space; And
Further comprising a direction filter _{unit for calculating the final direction information (Heading gyro} ) of the pedestrian by combining the absolute azimuth angle calculated by the magnetometer azimuth angle calculation unit and the angular velocity information calculated by the gyroscope azimuth angle calculation unit by a second Kalman filter. Hybrid indoor positioning system, characterized in that. The method of claim 11,
The first indoor positioning device includes: a step length estimating unit receiving a fusion pitch angle obtained by combining a pitch angle of the acceleration data and a pitch angle of angular velocity data, and estimating a step length SL using the received fusion pitch angle; And
The final direction information of the pedestrian is received from the direction filter unit, the step length (SL) estimated from the step length estimation unit is received, and the final direction information of the pedestrian, the step length, and initial stage by Equation 6 below. Hybrid indoor positioning system, characterized in that it further comprises a current position calculator for calculating the current position of the pedestrian using the position value.
[Equation 6]

Here, Xt-1, Yt-1 are the initial position values, Xt, Yt are the pedestrian's position values, SL is the step length, and Heading _fusion is the azimuth angle of the fusion of the magnetometer and the gyroscope. The method of claim 1,
The second indoor positioning device compares the previous keyframes with the current keyframes and searches for a keyframe having a degree of similarity greater than a predetermined threshold value. When a key frame with a similarity greater than a predetermined threshold is detected, the loop closing unit sets them as loop closures, optimizes the camera pose graph based on the posture change information of the set loop closure, and optimizes the graph. Hybrid indoor positioning system, characterized in that to perform location recognition and map creation based on. The method of claim 1,
The sensor fusion unit combines the first position and the second position using a Kalman filter algorithm having a movement model of Equation 7 below and an observation model of Equation 8, which is a transition state function of sensor fusion. Hybrid indoor positioning system.
[Equation 7]

here,

Is the two-dimensional coordinate of the pedestrian's position,

Is the x,y value (two-dimensional coordinate information of the pedestrian) of the IMU sensor input to the system, A and B are the identity matrices, W is the motion model with zero mean, and represents the Gaussian noise of the covariance matrix Q.
[Equation 8]

Equation 8 can be obtained through the position measurement result of the first indoor positioning device,

Is Gaussian noise of the position measurement result of the first indoor positioning device having a zero mean and a covariance matrix R, and H is an identity matrix. The method of claim 14,
The Kalman filter algorithm transfers the current state estimation result in the forward direction to a prediction step (Predict) of the Kalman filter that predicts the current state in advance of time. A hybrid indoor positioning system, characterized in that it is a time update equation for prediction consisting of a prior tracking error covariance equation.
[Equation 9]

Where A and B are identity matrices,

Is the input of the system,

Is the predicted value.
[Equation 10]

here,

Is the covariance matrix, T is the transposed matrix,

Is the error covariance value. The method of claim 15,
The Kalman filter algorithm corrects the estimated state values transmitted by actual measurement over time, and the Kalman gain generation equation of Equation 11 below, the updated estimation value generation equation of Equation 12 below, and Posteriori of Equation 13 below. A hybrid indoor positioning system, characterized in that it is a measurement update equation consisting of a tracking error covariance equation.
[Equation 11]

Here, H is the identity matrix and R is the covariance matrix.
[Equation 12]

here,

Is an observation function.
[Equation 13]

Images