KR20220107886A - Apparatus for PDR Based on Deep Learning using multiple sensors embedded in smartphones and GPS location signals and method thereof - Google Patents

Abstract

The present invention relates to a deep learning-based pedestrian dead reckoning (PDR) positioning device using a smartphone multi-sensor, a global positioning system (GPS) location signal, and a method thereof. According to the present invention, the deep learning-based PDR positioning device comprises: a location-setting unit receiving a GPS signal lastly received outside a building from a user-owned smartphone and setting a location coordinate value extracted from the GPS signal as a starting point; a data-acquiring unit receiving a sensor value measured whenever the user walks indoors after a point in time at which the GPS signal is acquired and generating input data by preprocessing the sensor value; a learning unit for forecasting a GPS longitude change by entering the input data to a first learning model that has been previously learned and forecasting a GPS latitude change by entering the input data to a second learning model; and a location positioning unit positioning a moving path of the user by applying the forecast GPS longitude change and GPS latitude change to the starting point. According to the present invention, by accumulating data according to a smartphone location (during a call, texting, in a pocket, etc.) that the user mainly uses while walking, indoor location positioning can be performed without considering a separate calculation method according to the various smartphone locations. An effective estimation can be performed when external communications are difficult, such as in a disaster environment, because additional communication equipment, such as a Wi-Fi access point (AP) used in the existing indoor positioning, is not required. The purpose of the present invention is to provide the deep learning-based PDR positioning device and the method thereof, which are for positioning a user location indoors by using the data combining the sensor value acquired when walking with the smartphone and the GPS signal.

Description

A deep learning-based PDR positioning device and method using multiple sensors and GPS location signals of a smartphone {Apparatus for PDR Based on Deep Learning using multiple sensors embedded in smartphones and GPS location signals and method thereof}

The present invention relates to a deep learning-based PDR positioning device and method using multiple sensors and GPS location signals of a smartphone, and more specifically, a combination of a sensor value that can be obtained when walking with a smartphone and a GPS signal It relates to a deep learning-based PDR positioning device and method for positioning a user indoors using data.

Pedestrian Dead Reckoning (hereinafter referred to as “PDR”) technology uses various sensors to identify movement information such as speed, acceleration, direction, and distance that a person moves, and calculates the relative position from the starting point. means technology.

The conventional pedestrian positioning (PDR) is updated according to Equation 1 below.

는 스마트폰 사용자의 초기 위치를 나타내고,

은 m번째 걸음마다 추정된 사용자의 보폭을 나타내고,

은 m번째 걸음마다 추정된 기기의 방향각을 나타낸다.here

represents the initial location of the smartphone user,

represents the estimated user's stride for every m-th step,

denotes the direction angle of the device estimated for every m-th step.

)은 하기 수학식2를 이용하여 추정된다. And, the stride length (

) is estimated using Equation 2 below.

는 보정 상수를 나타내고,

은 가속도계의 최대값이고,

은 가속도계의 최소값을 나타낸다. here,

represents the correction constant,

is the maximum value of the accelerometer,

is the minimum value of the accelerometer.

)은 하기 수학식 3을 이용하여 추정되며, 잡음에 영향을 받는다. Also, the direction angle (

) is estimated using Equation 3 below and is affected by noise.

Conventional pedestrian positioning (PDR) has a problem of estimating a user's step length, and a problem of sensor drift and accumulation error occurring according to a walking path.

Therefore, in order to solve the above problems, a lot of research has been done on a technique for estimating a user's current location by combining a location positioning (PDR) using a sensor of a smart phone and an access point (AP) inside a building. .

However, the location estimation method combining the access point (AP) had the inconvenience of having to readjust the location one by one according to the user's stride length, drift, and accumulation error. In addition, if the access point (AP) cannot be used, the exact location cannot be determined, and the unique biomechanical information that appears every time each user walks cannot be reflected in the stride length, so it was difficult to estimate the exact location indoors. .

The technology that is the background of the present invention is disclosed in Korean Patent No. 10-1452373 (Announcement on October 13, 2014).

As described above, according to the present invention, a deep learning-based PDR positioning device and method for positioning a user's location indoors using data combining a GPS signal and a sensor value that can be obtained when walking with a smartphone, and a method therefor it is for

According to an embodiment of the present invention for achieving this technical task, in the deep learning-based PDR positioning device using multiple sensors and GPS location signals of a smartphone, the GPS signal last received from the outside of the building from the smartphone possessed by the user a position setting unit for receiving and setting the position coordinate value extracted from the GPS signal as a starting point, receiving the measured sensor value every time the user walks indoors after the time of acquiring the GPS signal, and pre-processing the sensor value a data acquisition unit generating input data by inputting the input data into a pre-learned first learning model to predict a change in GPS longitude, and a learning unit for predicting a change in GPS latitude by inputting the input data into a second learning model and a positioning unit configured to position the moving path of the user by applying the predicted change in GPS longitude and change in GPS latitude to the starting point.

Further comprising a database for collecting and storing sensor values and GPS signal position coordinate values obtained using at least one sensor among acceleration, gyroscope, and geomagnetism installed in the smartphone,

The learning unit pre-processes the sensor values stored in the database to obtain input data indicating the location and direction of the smartphone according to the change in GPS latitude and longitude of the previous step and the current step, and use the obtained input data as a first learning model and input to a second learning model to output a change in GPS longitude through the first learning model, and to learn to output a change in GPS latitude through the second learning model.

The input data may include at least one of a change amount of acceleration, an average amount of the sum of acceleration magnitudes, a geomagnetic value, and a change amount of a gyroscope.

The user's current GPS coordinate value estimated by applying the predicted change in GPS longitude and change in GPS latitude may be expressed in units of seconds using the following equation.

Here, Degree indicates degrees, Minutes indicates minutes, and C indicates GPS coordinate values, and cut() indicates rounding off decimal places.

The first learning model and the second learning model are implemented in the form of a Multi-Layer Perceptron (MLP) algorithm, and when a user walks while sending a text using Tensor Flow (texting mode) , the nth mode corresponding to any one of a case of walking while talking (talking mode), a case of walking while shaking a part of the body (swing mode), and a case of walking with a smartphone in a pocket (pocket mode) By normalizing the sensor value measured for each step to a characteristic value, the GPS longitude change amount and the GPS latitude change amount at the nth step can be predicted, respectively.

In addition, according to an embodiment of the present invention, in the deep learning-based PDR positioning method using the PDR positioning device, the GPS signal received last from the outside of the building is received from the smartphone possessed by the user, and the location extracted from the GPS signal Setting a coordinate value as a starting point, receiving a sensor value measured every time a user walks indoors after the point at which the GPS signal is acquired, and pre-processing the sensor value to generate input data; 1 inputting the input data into a learning model to predict a change in GPS longitude, inputting the input data to a second learning model to predict a change in GPS latitude, and setting the predicted change in GPS longitude and change in GPS latitude as the starting point and positioning the user's moving path by applying to the .

As described above, according to the embodiment of the present invention, the drift problem of the sensor, which is a problem in the existing pedestrian positioning (PDR), is corrected with the GPS signal, so that it is strong against the effect of the accumulation error of the sensor drift and more accurate indoor positioning is possible. . In addition, according to the embodiment of the present invention, since additional equipment is not required to estimate the stride length, an economical pedestrian positioning (PDR) system can be implemented.

In addition, according to the embodiment of the present invention, by accumulating data according to the location of the smartphone mainly used by the user when walking (during a call, texting, in the pocket, etc.), a separate calculation method according to various smartphone locations is not considered. Indoor positioning is possible without the need for additional communication equipment such as Wi-Fi access point (AP) used in existing indoor positioning, so it is possible to effectively estimate in situations where external communication is difficult, such as in a disaster environment.

1 is a configuration diagram for explaining a deep learning-based PDR positioning apparatus according to an embodiment of the present invention.
2 is a flowchart for explaining a method for positioning a user's location indoors using a deep learning-based PDR positioning device according to an embodiment of the present invention.
3 is an exemplary diagram illustrating a general location of a smartphone of a user while walking.
4 is a diagram schematically illustrating a first learning model and a second learning model according to an embodiment of the present invention.

Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the accompanying drawings. In this process, the thickness of the lines or the size of the components shown in the drawings may be exaggerated for clarity and convenience of explanation.

In addition, the terms to be described later are terms defined in consideration of functions in the present invention, which may vary according to intentions or customs of users and operators. Therefore, definitions of these terms should be made based on the content throughout this specification.

Hereinafter, a deep learning-based PDR positioning apparatus according to an embodiment of the present invention will be described in more detail with reference to FIG. 1 .

1 is a configuration diagram for explaining a deep learning-based PDR positioning apparatus according to an embodiment of the present invention.

As shown in FIG. 1, the deep learning-based PDR positioning device 100 according to an embodiment of the present invention includes a position setting unit 110, a data acquisition unit 120, a learning unit 130, and a position positioning unit ( 140 ) and a database 150 .

First, the location setting unit 110 sets a starting point for positioning the user. In other words, the location setting unit 110 receives the location coordinate value extracted from the GPS signal from the user's smart phone. In this case, the GPS signal indicates the last signal received when the user is located outside the building. Then, the position setting unit 110 sets the received position coordinate value as a starting point.

The data acquisition unit 120 receives sensor values measured through a plurality of sensors built into the smart phone after the GPS signal is received, and pre-processes the received sensor values to generate input data.

Here, the plurality of sensors includes at least one sensor among an acceleration sensor, a gyroscope sensor, and a geomagnetic sensor, and the input data indicates the location and direction of the smartphone according to the change in GPS latitude and longitude of the previous step and the current step.

The learning unit 130 builds two learning models, inputs the sensor values obtained through a plurality of sensors and the position coordinate values of the GPS signals to the built two learning models, and outputs the GPS longitude change and the GPS latitude change, respectively. learn to do

In other words, the learning unit 130 learns to output the GPS longitude variation by inputting input data to the first model implemented in the form of a multi-layer perceptron (MLP) algorithm.

In addition, the learning unit 130 learns to output the GPS latitude change by inputting input data to the second model implemented in the form of a multi-layer perceptron (MLP) algorithm.

The location positioning unit 140 applies the GPS longitude change and the GPS latitude change obtained from the first model and the second model, respectively, to the starting point to position the moving path of the user.

Finally, the database 150 collects a sensor value measured using a smart phone and a location coordinate value of a GPS signal. In detail, the database 150 collects sensor values measured through a plurality of sensors built into the smart phone whenever the user normally walks, and GPS location coordinate values acquired as the user's location changes.

Hereinafter, a method of positioning a user's position indoors using the deep learning-based PDR positioning apparatus 100 according to an embodiment of the present invention will be described in more detail with reference to FIGS. 2 to 4 .

2 is a flowchart for explaining a method for positioning a user's position indoors using a deep learning-based PDR positioning device according to an embodiment of the present invention, and FIG. 3 is a general smartphone location of the user while walking It is also an example.

As shown in FIG. 2 , the deep learning-based PDR positioning device 100 according to an embodiment of the present invention collects sensor values and GPS location coordinates measured according to a user's steps ( S210 ).

The deep learning-based PDR positioning device 100 acquires a sensor value and a GPS location coordinate value measured every time a user walks from a smartphone possessed by the user.

In other words, since the smartphone includes at least one sensor among an acceleration sensor, a gyroscope sensor, and a geomagnetic sensor, when a user walks using a plurality of sensors, information on motion information, direction, and movement speed according to the location of the smartphone can be obtained.

As shown in Figure 3, the location of the smartphone is when walking while sending a text (texting mode), when walking while making a call (talking mode), when walking while shaking a part of the body (swing mode) and pocket It is different depending on any one mode among the cases of walking with a smartphone in the pocket mode, and the sensor value according to the location of the smartphone is measured differently.

Therefore, the deep learning-based PDR positioning device 100 receives the sensor values and GPS position coordinate values obtained from point A and point B respectively when the user moves one step from point A to point B, and the received sensor value and the GPS location coordinate values are stored in the database 150 .

Next, the learning unit 130 learns the first learning model and the second learning model by using the sensor values and GPS location coordinate values stored in the database 150 ( S220 ).

In other words, the learning unit 130 builds a first learning model and a second learning model implemented in the form of a multi-layer perceptron (MLP) algorithm.

A multi-layer perceptron (MLP) has a structure of a neural network including one or more intermediate layers between an input layer and an output layer, and a value transmitted from the input layer is transmitted to the output layer through the hidden layer. At this time, the multilayer perceptron (MLP) compares the result value output through the output layer with the actual value, and determines and applies a weight value that minimizes the error between the result value and the actual value. And, in the multi-layer perceptron (MLP), if the error between the result value of the output layer and the actual value falls within the allowable range, learning is terminated.

Accordingly, the learning unit 130 inputs the input data obtained by pre-processing the sensor values stored in the database 150 to the first learning model, and obtains the GPS longitude change amount through the first learning model. Then, the learning unit 130 compares the GPS longitude change amount obtained through the first learning model with the GPS location coordinate value stored in the database, and then the error between the GPS longitude change amount and the GPS location coordinate value stored in the database is within the allowable range. Repeatedly supervised learning until appropriate.

In addition, the learning unit 130 inputs the input data obtained by pre-processing the sensor values stored in the database 150 to the second learning model, and obtains the GPS latitude change through the second learning model. Then, the learning unit 130 compares the GPS latitude change obtained through the second learning model with the GPS location coordinate value stored in the database, and then the error between the GPS latitude change amount and the GPS location coordinate value stored in the database is within the allowable range. Repeatedly supervised learning until appropriate.

Here, the input data input to the first learning model and the second learning model includes at least one of a change amount of acceleration, an average amount of the sum of acceleration magnitudes, a geomagnetic value, and a change amount of a gyroscope.

4 is a diagram schematically illustrating a first learning model and a second learning model according to an embodiment of the present invention.

As shown in FIG. 4 , the first learning model and the second learning model according to an embodiment of the present invention are implemented using Tensor Flow. In other words, the sensor values stored in the database 150 are expressed in different measurement units. Therefore, as shown in FIG. 4 , the learning unit 130 provides a batch normalization layer such as the Mul and Add layers shown in black, and uses the provided batch normalization layer layer for every nth step. The measured sensor values are normalized to the characteristic values.

When estimating the location of the user who entered the room in the state that step S220 is completed, the location setting unit 110 sets a starting point using the GPS signal received from the user's smartphone (S230).

When the user enters the room, the smartphone in the user's possession can no longer obtain the user's location coordinates through the GPS. Accordingly, the location setting unit 110 receives the last GPS signal received from the outside of the building from the smartphone, and sets the location coordinate value obtained from the received GPS signal as a starting point.

Next, the data acquisition unit 120 obtains input data by pre-processing sensor values measured through a plurality of sensors built into the smartphone ( S240 ).

In other words, the data acquisition unit 120 receives the sensor value generated after the last GPS signal is received. Then, the data acquisition unit 120 pre-processes the received sensor value to acquire input data reconstructed as data usable for positioning.

Here, the input data includes at least one of a change amount of acceleration, an average amount of the sum of acceleration magnitudes, a geomagnetic value, and a change amount of the gyroscope.

To explain this in more detail, when the user moves one step from point A to point B, a plurality of sensors embedded in the smartphone acquires at least one sensor value among acceleration, gyroscope, and geomagnetism, and dips the acquired sensor value. It is transmitted to the learning-based PDR positioning device (100).

Then, the data obtaining unit 120 obtains an average amount of acceleration change and acceleration according to the user's stride length, and obtains a geomagnetic value indicating the moving direction of the user and a change amount of the gyroscope.

Next, the positioning unit 140 inputs input data including the acquired acceleration change amount, the average amount of acceleration magnitude, the geomagnetic value, and the gyroscope change amount into the first learning model and the second learning model, respectively. A GPS longitude change amount and a GPS latitude change amount are obtained from the first learning model and the second learning model (S250).

Meanwhile, in general, latitude coordinate values and longitude coordinate values according to GPS signals are expressed using decimal points. The difference between the latitude and longitude changes measured as the user moves one step using the latitude and longitude coordinate values displayed to the decimal place is too small, so it is difficult to locate the user's location.

Accordingly, in the embodiment of the present invention, the GPS longitude change amount and the GPS latitude change amount are expressed in units of seconds using Equation 4 below.

Here, Degree indicates degrees, Minutes indicates minutes, and C indicates GPS coordinate values, and cut() indicates rounding off decimal places.

Next, the positioning unit 140 applies the GPS longitude change amount predicted from the first learning model and the GPS latitude change amount predicted from the second learning model to the starting point to determine the user's moving path ( S260 ).

For example, the GPS location coordinate value corresponding to the user's starting point is 37° 29' 45.6"N, 126° 57' 20.6"E, and the GPS longitude change predicted from the first learning model is +1 second (+1" ), and it is assumed that the GPS latitude change amount predicted from the second learning model is +1 second (+1").

Then, the location positioning unit 140 may obtain the current user's location coordinates by correcting the amount of change predicted from the first learning model and the second learning model. That is, the location positioning unit 140 can predict the user's movement distance and direction angle using the obtained location coordinates and sensor values, so that the user's movement path can be determined indoors.

As described above, the deep learning-based PDR positioning device according to an embodiment of the present invention corrects the drift problem of the sensor, which is a problem in the existing pedestrian positioning (PDR), with a GPS signal, so it is strong against the effect of the accumulation error of the sensor drift and more Accurate indoor positioning is possible. In addition, according to the embodiment of the present invention, since additional equipment is not required to estimate the stride length, an economical pedestrian positioning (PDR) system can be implemented.

In addition, the deep learning-based PDR positioning device according to an embodiment of the present invention accumulates data according to the smartphone location (during a call, texting, in a pocket, etc.) mainly used when the user walks, so that it is located in various smartphone locations. Therefore, indoor positioning is possible without considering a separate calculation method, and there is no need for additional communication equipment such as Wi-Fi access point (AP) used in existing indoor positioning, so it is effective in situations where external communication is difficult, such as in a disaster environment. It is possible to estimate

Although the present invention has been described with reference to the embodiment shown in the drawings, this is merely exemplary, and it will be understood by those skilled in the art that various modifications and equivalent other embodiments are possible therefrom. will be. Therefore, the true technical protection scope of the present invention should be determined by the technical spirit of the following claims.

100: PDR positioning device
110: position setting unit
120: data acquisition unit
130: study department
140: positioning unit
150 : database

Claims (10)

In a deep learning-based PDR positioning device using multiple sensors and GPS location signals of a smartphone,
A location setting unit that receives the last GPS signal received from the outside of the building from the user's smartphone, and sets the location coordinate value extracted from the GPS signal as a starting point;
A data acquisition unit that receives a sensor value measured every time a user walks indoors after acquiring the GPS signal, and pre-processes the sensor value to generate input data;
A learning unit for predicting a change in GPS longitude by inputting the input data into a pre-learned first learning model, and predicting a change in GPS latitude by inputting the input data into a second learning model, and
A deep learning-based PDR positioning device including a positioning unit for positioning a user's moving path by applying the predicted change in GPS longitude and change in GPS latitude to the starting point. According to claim 1,
Further comprising a database for collecting and storing sensor values and GPS signal position coordinate values obtained using at least one sensor among acceleration, gyroscope, and geomagnetism installed in the smartphone,
The learning unit,
The sensor values stored in the database are pre-processed to obtain input data representing the location and direction of the smartphone according to the change in GPS latitude and longitude of the previous and current steps, and the obtained input data is used for the first learning model and the second learning A deep learning-based PDR positioning device that inputs to a model, outputs a change in GPS longitude through the first learning model, and learns to output a change in GPS latitude through the second learning model. 3. The method of claim 1 or 2,
The input data is
A deep learning-based PDR positioning device including at least one of an amount of change in acceleration, an average amount of the sum of the magnitudes of acceleration, a geomagnetic value, and a change in a gyroscope. 3. The method of claim 1 or 2,
The current GPS coordinate value of the user estimated by applying the predicted change in GPS longitude and change in GPS latitude is,
A deep learning-based PDR positioning device expressed in units of seconds using the following equation:

Here, Degree indicates degrees, Minutes indicates minutes, and C indicates GPS coordinate values, and cut() indicates rounding off decimal places. 3. The method of claim 1 or 2,
The first learning model and the second learning model,
Implemented in the form of a Multi-Layer Perceptron (MLP) algorithm, and using Tensor Flow, when a user walks while sending a text message (texting mode), when walking while talking on a call (talking mode), Normalizes the sensor value measured at every nth step to a characteristic value in response to any one mode of walking while shaking a part of the body (swing mode) and walking with a smartphone in a pocket (pocket mode) A deep learning-based PDR positioning device that predicts the change in GPS longitude and change in GPS latitude at the nth step, respectively. In a deep learning-based PDR positioning method using a PDR positioning device,
Receiving the last GPS signal received from outside the building from the user's smartphone, and setting the location coordinate value extracted from the GPS signal as a starting point;
Receiving a sensor value measured every time the user walks indoors after the GPS signal is acquired, and pre-processing the sensor value to generate input data;
Predicting a change in GPS longitude by inputting the input data into a pre-trained first learning model, and predicting a change in GPS latitude by inputting the input data into a second learning model, and
Deep learning-based PDR positioning method comprising the step of applying the predicted change in GPS longitude and change in GPS latitude to the starting point to determine the user's moving path. 7. The method of claim 6,
Collecting the sensor value and the position coordinate value of the GPS signal obtained by using at least one sensor among acceleration, gyroscope, and geomagnetism included in the smartphone, and
The sensor value is preprocessed to obtain input data indicating the location and direction of the smartphone according to the change in GPS latitude and longitude of the previous step and the current step, and input the obtained input data to the first learning model and the second learning model to output the GPS longitude change amount through the first learning model, and learning to output the GPS latitude change amount through the second learning model. 8. The method of claim 6 or 7,
The input data is
A deep learning-based PDR positioning method comprising at least one of an amount of change in acceleration, an average amount of the sum total of magnitudes of acceleration, a geomagnetic value, and a change in a gyroscope. 8. The method of claim 6 or 7,
The current GPS coordinate value of the user estimated by applying the predicted change in GPS longitude and change in GPS latitude is,
Deep learning-based PDR positioning method expressed in units of seconds using the following equation:

Here, Degree indicates degrees, Minutes indicates minutes, and C indicates GPS coordinate values, and cut() indicates rounding off decimal places. 8. The method of claim 6 or 7,
The first learning model and the second learning model,
It is implemented in the form of a Multi-Layer Perceptron (MLP) algorithm, and uses Tensor Flow to send texting mode, talk mode, swing mode, and pocket mode. ), a deep learning-based PDR positioning method that predicts the change in GPS longitude and latitude at the nth step by normalizing the sensor value measured every nth step to a characteristic value in response to any one mode.

Images