KR102350593B1 - Apparatus and method for classifying gait pattern based on multi modal sensor using deep learning ensemble - Google Patents

Abstract

The present invention relates to a multimodal sensor-based gait pattern classification apparatus using deep learning ensembles, which collects gait information by using a sensor of a shoe insole and accurately identifying a gait pattern by using a combination of convolutional neural network (CNN) and a recurrent neural network (RNN), and a method thereof. According to the present invention, the multimodal sensor-based gait pattern classification apparatus comprises: a data collection unit collecting gait data from a smart insole; a data generation unit defining a section of a unit step in gait information and adjusting the size of all unit steps to a standard length to generate a data set; a CNN learning classification unit and an RNN learning classification unit independently trained by using the data set generated by the data generation unit and applying an average ensemble model to provide one final prediction by performing CNN learning classification and RNN learning classification, respectively; and a gait classification result output unit using a value output from a fully connected network through the CNN learning classification unit and the RNN learning classification unit to determine and output a gait type.

Description

Apparatus and method for classifying gait pattern based on multi modal sensor using deep learning ensemble}

The present invention relates to gait pattern classification. Specifically, gait information is collected using a sensor of a shoe insole, and a gait pattern can be accurately identified using a combination of CNN (Convolutional Neural Network) and RNN (Recurrent Neural Network). It relates to an apparatus and method for gait pattern classification based on a multimodal sensor using a deep learning ensemble.

Walking is one of the representative movements of humans, and analysis of gait patterns can be utilized in many applications such as bionics, rehabilitation medicine, and healthcare.

As one of the sub-fields of gait analysis, gait type classification is being studied for the diagnosis of Parkinson's disease, sports analysis, and development of walking aids for the elderly based on gait data acquired using sensors.

Gait patterns have characteristics that are easily affected by factors such as physical differences due to physical characteristics, differences in speed and topography, and these characteristics have a negative effect on the performance of gait type classification by making large variations within the same gait type. .

For example, even for the same gait motion, there is a difference in the pattern of walking on flat ground and walking on a hill, and this variation makes it difficult to extract features for classifying a gait type called 'walking'.

The walking type classification system consists of a sensor module that acquires sensor data and an application module that calculates classification results based on the acquired data. , an accelerometer, and a gyro sensor.

However, most sensors have limitations in measuring gait data only in a limited environment due to limitations such as the size of the sensor and the inconvenience of installation.

Recent developments in wearable sensor technology have led to weight reduction and simplification of equipment used for measuring gait data.

That is, various wearable devices such as smart watches, sports bands, and smart insoles are being developed as sensor modules are miniaturized and low-power sensor technologies are developed.

Since the use of wearable sensors has fewer environmental restrictions for data collection, data can be collected relatively easily in daily life. It has the advantage of low processing load.

In addition, techniques for collecting gait information using a smart insole with various types of sensors, performing neural network analysis for each sensor type, and classifying gait types using the analyzed information have been proposed. 's gait pattern classification technique has limitations in terms of accuracy.

Therefore, the development of a new technology capable of increasing the accuracy of gait pattern classification is required.

Republic of Korea Patent Publication No. 10-2020-0063795 Republic of Korea Patent Publication No. 10-2020-0019051 Republic of Korea Patent Publication No. 10-2019-0136324

The present invention is to solve the problems of the gait pattern classification technology of the prior art, and by using a deep learning algorithm to check the gait information collected through a wearable device to increase the accuracy of gait pattern classification, a multi-using deep learning ensemble An object of the present invention is to provide an apparatus and method for classifying a gait pattern based on a modal sensor.

The present invention collects gait information using a sensor of a shoe insole, and uses a combination of CNN (Convolutional Neural Network) and RNN (Recurrent Neural Network) to accurately identify gait patterns. Multimodal using a deep learning ensemble An object of the present invention is to provide an apparatus and method for sensor-based gait pattern classification.

The present invention accurately detects the human gait cycle from the collected raw gait information, and designs an ensemble model that uses CNN and RNN in a cooperative and complementary way to improve identification accuracy by utilizing multimodal sensing. An object of the present invention is to provide an apparatus and method for gait pattern classification based on a multimodal sensor using a deep learning ensemble.

The present invention provides an environment for wearing shoes with insoles, registering gait information in the system, and using a deep learning ensemble to identify gait patterns with high accuracy from gait information of one step. An object of the present invention is to provide an apparatus and method for pattern classification.

Other objects of the present invention are not limited to the above-mentioned objects, and other objects not mentioned will be clearly understood by those skilled in the art from the following description.

An apparatus for classifying a gait pattern based on a multi-modal sensor using a deep learning ensemble according to the present invention for achieving the above object is a data collection unit that collects gait data from a smart insole; Section of a unit step in information about gait A data generator that defines and adjusts the size of all unit steps to a standard length to generate a data set; Each is independently trained using the data set generated by the data generator, and performs CNN learning classification and RNN learning classification, respectively. A CNN learning classifier and RNN learning classifier that apply an average ensemble model to provide one final prediction; a gait type is determined using the value output from a fully connected network through a CNN learning classifier and an RNN learning classifier. and a gait classification result output unit to output.

A method for classifying a gait pattern based on a multi-modal sensor using a deep learning ensemble according to the present invention for achieving another object includes: collecting information about a user's gait from a smart insole; section of a unit step in gait-related information defining a data set; scaling all unit steps to a standard length using spline interpolation, and concatenating unit steps of the same form for both feet to generate a data set; Using the CNN learning classification and RNN learning classification, input to a fully connected network capable of classifying the type of gait, and determining the type of gait using the value output from the fully connected network; characterized in that it comprises; do.

The apparatus and method for classifying a gait pattern based on a multimodal sensor using a deep learning ensemble according to the present invention as described above have the following effects.

First, the deep learning algorithm is used to check the gait information collected through the wearable device to increase the accuracy of gait pattern classification.

Second, it collects gait information using the sensor of the shoe insole, and uses a combination of CNN (Convolutional Neural Network) and RNN (Recurrent Neural Network) to accurately identify gait patterns.

Third, it is possible to accurately detect the human gait cycle from the collected raw gait information, and to design an ensemble model that uses CNN and RNN in a cooperative and complementary way to increase the identification accuracy by utilizing multimodal sensing.

Fourth, it provides an environment in which shoes with insoles are worn, gait information is registered in the system, and gait patterns can be identified with high accuracy in gait information of one step.

1 is a block diagram of a device for gait pattern classification based on a multi-modal sensor using a deep learning ensemble according to the present invention;
2 is a flowchart illustrating a method for classifying a gait pattern based on a multi-modal sensor using a deep learning ensemble according to the present invention.
3 is a block diagram of a foot logger according to the present invention;
4 is a block diagram showing a gait cycle of a person;
5 is a convolution graph of the average and standard deviation of eight pressure values (blue line) and a Gaussian function;
6 is a block diagram showing a data standardization procedure
7 is a configuration diagram of a network model according to an embodiment of the present invention;
8 is a schematic diagram of partitioning data into training and test data sets.
9 is a schematic diagram showing a t-SNE plot for the output of a fully connected layer with 256 units in CNN and RNN architectures.
10 is a configuration diagram showing the identification accuracy of CNNs, RNNs and ensemble networks using tri-modal detection
11 is a block diagram showing the identification accuracy of CNNs, RNNs and ensemble networks using bi-modal detection.
12A and 12B are graphs comparing classification accuracy using an apparatus for classifying a gait pattern based on a multi-modal sensor using a deep learning ensemble according to the present invention.

Hereinafter, a preferred embodiment of an apparatus and method for classifying a gait pattern based on a multimodal sensor using a deep learning ensemble according to the present invention will be described in detail as follows.

Features and advantages of the apparatus and method for multimodal sensor-based gait pattern classification using a deep learning ensemble according to the present invention will become apparent through detailed description of each embodiment below.

1 is a block diagram of a device for gait pattern classification based on a multimodal sensor using a deep learning ensemble according to the present invention.

The present invention is to provide a framework for identifying an individual in gait information, collecting gait information using a sensor of a participant's shoe insole, and using a combination of CNN (Convolutional Neural Network) and RNN (Recurrent Neural Network) It may include a configuration for classifying a gait pattern.

An apparatus for classifying a walking pattern based on a multi-modal sensor using a deep learning ensemble according to the present invention includes a pressure sensor, an acceleration sensor, and a gyro sensor, and a smart insole 10 that collects information about a user's walking, and a smart insole A data collection unit 20 that collects gait data from (10), defines the section of a unit step in information about gait, and adjusts the size of all unit steps to a standard length using spline interpolation, The data generation unit 30, which generates a data set by connecting the unit steps of the same form for both feet, and the data generation unit 30 are each independently trained using the data set, respectively, CNN learning classification and RNN learning A CNN learning classifier 40 and an RNN learning classifier 50 that apply an average ensemble model to classify and provide one final prediction, a CNN learning classifier 40 and an RNN learning classifier 50 and a gait classification result output unit 60 that determines and outputs the type of gait using the value output from the fully connected network.

Here, the data generator 30 defines a section of a unit step in information about walking, adjusts the size of all unit steps to a standard length using spline interpolation, and detects three types of pressure, acceleration, and rotation. It includes a sensing mode separating unit 31 that separates into modes, and a data set generating unit 32 that connects unit steps of the same form for both feet and generates a standardized data set.

2 is a flowchart illustrating a method for classifying a gait pattern based on a multimodal sensor using a deep learning ensemble according to the present invention.

In the method for classifying a gait pattern based on a multi-modal sensor using a deep learning ensemble according to the present invention, first, a data collection unit collects information about a user's gait from a smart insole including a pressure sensor, an acceleration sensor, and a gyro sensor. (S201)

Next, a section of a unit step is defined in the information about walking. (S202)

Then, by using spline interpolation, the size of all unit steps is adjusted to a standard length, and the unit steps of the same form are connected for both feet and a data set is generated (S203).

Next, CNN learning classification and RNN learning classification are performed using the data set, input to a fully connected network capable of classifying the type of gait, and the gait type is determined using the value output from the fully connected network. (S204 )

Hereinafter, each step of the method for classifying a gait pattern based on a multimodal sensor using a deep learning ensemble according to the present invention will be described in detail.

The collected data consists of a time series measured by a pressure sensor, a 3-axis accelerometer, and a 3-axis gyroscope installed on the shoe insole, and the gait cycle includes a stance phase and a swing phase.

The gait cycle refers to the movement from the moment one foot touches the ground to the moment it leaves the ground and then touches the ground again.

The gait cycle consists of a total of 7 steps (heel strike, foot flat, mid stance, heel off, toe off, mid swing, late swing).

The heel strike stage is a stage indicating the start of the gait cycle, and the heel of one foot is in contact with the ground, and the foot flat stage is a stage in which the sole of one foot is in contact with the ground. In the mid stance phase, the leg of one foot is perpendicular to the ground and is stopped, and at this time, the leg of the other foot moves forward. The heel-off stage refers to a time when one heel starts to be lifted off the floor, and the toe-off stage is a stage in which the front toe of one foot falls off the floor and the sole of the other foot touches the ground at the same time. The mid swing stage is a stage in which one foot moves forward with one foot off the ground, and the late swing stage refers to the stage just before the start of the next gait cycle.

In general, based on one foot, heel strike, foot flat, mid stance, heel off, and toe off phases in which the foot is in contact with the ground are defined as the stance phase, and the mid swing and late swing phases in which the foot is away from the ground is defined as the swing phase.

The swing phase is the total time the foot is in the air, so the pressure value during the swing phase should be zero for that foot.

Given this nature of the gait cycle, it should be possible to divide the original time series into a series of separate units (i.e., steps) in order to use the data more efficiently.

However, the pressure sensor mounted on the insole frequently reports a non-zero value during the swing phase due to high temperature or interference between the sensors.

In the present invention, the unit step is determined using Gaussian filtering to overcome the potential error of the pressure sensor. Then, we design an ensemble network using CNN and RNN together with multi-modal sensing data to identify gait patterns in continuous unit step data.

Data sets are generated by selecting single, double, or triple modal inputs from a pressure sensor, accelerometer, or gyroscope.

Then, the CNN and RNN are independently trained using the same training data set, and in the test phase, the individual's softmax scores are calculated by taking the average of the softmax scores from the CNN and RNN.

With such a configuration, an identification accuracy of about 99% can be achieved using tri-modal detection using only a single unit step.

Hereinafter, data preprocessing procedures for converting raw gait information into a standard format, convolutional neural networks, recurrent neural networks, and the design of ensemble models used for identification are described.

Gait information is being analyzed to diagnose mental and physical diseases.

Changes in gait patterns may be indicative of intellectual disability, dementia, depression and deformation of joint diseases, and gait information may reflect the mental health of the elderly.

In particular, poor mental health can make your gait more asymmetrical. Gait analysis can also be used to estimate age and gender.

And the gait information is collected as a time series vector and represents several consecutive unit steps.

In the present invention, the data source may be obtained as follows.

3 is a block diagram of a foot logger according to the present invention, and FIG. 4 is a block diagram illustrating a gait cycle of a person.

A commercial insole (FootLogger) can be used to collect gait information.

As shown in Figure 1, FootLogger includes 8 pressure sensors, a 3-axis accelerometer, and a 3-axis gyroscope for each foot. The sampling rate is 100 Hz for each device, and the pressure sensor categorizes the pressure intensity into one of three levels: 0, 1, or 2.

As such, FootLogger has a built-in pressure sensor array consisting of 8 pressure sensors, a 3-axis acceleration sensor array, and a 3-axis gyro sensor array. measure

In the case of the pressure sensor, 0, 1, and 2 values are output according to the strength of the pressure. 0 is the state without pressure, that is, the swing phase (the state where the foot is off the ground), and 1 and 2 are the stance phase (the state where the foot is on the ground). It means the strength of the pressure in the standing state).

The accelerometer and gyroscope measure acceleration and rotation respectively, both recorded as integers between -32,768 and 32,768 in 3D space.

및

로 표시한다.(univariate) time series and multivariate time series, respectively

and

indicated as

, 가속도는

, 회전의 경우

이다. 다른 참가자 식별은 아래 첨자 번호, 즉 id = i에 대해

를 사용하여 표현된다.Other sensing modalities are expressed using superscript characters. That is, the pressure is

, the acceleration is

, in the case of rotation

to be. Another participant identification is for a subscript number, i.e. id = i

is expressed using

And it improves performance by converting the data collected through data preprocessing into a fixed standard format.

In the present invention, in terms of identification accuracy. Gaussian filtering can be used to transform the data into a standard format.

To determine the unit step more consistently and accurately, a convolution operation is performed using the average of the eight pressure values and the Gaussian function.

는 8개의 압력 값의 평균이고,

여기서

라고 하면, 컨볼루션 연산은 수학식 1에서와 같이 정의된다.

is the average of eight pressure values,

here

, the convolution operation is defined as in Equation 1.

5 is a convolution graph of the average and Gaussian function of the average of eight pressure values (blue line) and the standard deviation s = 0.2s (red line).

및

의 예는 각각 파란색 선과 빨간색 선으로 표시된다.in Figure 5

and

Examples of are indicated by blue and red lines, respectively.

A unit step is formally defined to establish a standard format.

이고, 여기서, 모든 시간 t에 대해

및

이 되는

이다.Sorted list for each foot

, where, for every time t

and

becoming

to be.

으로 정의하고, 여기서,

이다.time series of unit steps

is defined as, where,

to be.

으로 정의되므로

이고, 여기서

모든 참가자의 두 발의 단위 스텝이다.For discrete variables, the sampling rate of the insole is 100 Hz and the standard length is

because it is defined as

and where

It is a unit step of all participants' two feet.

Then, using spline interpolation, all unit steps are scaled to a standard length d, and unit steps of the same form are connected for both feet.

으로 표시되고, 여기서

이고, 해당 치수는

및

이다. 실험에서는 d = 87로 설정한다.In summary, the i-th unit step detecting m is

is indicated as, where

, and the dimensions are

and

to be. In the experiment, d = 87 is set.

6 is a block diagram illustrating a data standardization procedure.

Figure 6 (a) shows the original format, (b) shows that the length of the model is fixed at d = 87, and (c) shows that the same sensing method of both feet is connected in a standard format.

6 shows a procedure to intuitively show data standardization, and the original time series of two feet is converted into a standard format according to the following procedure.

(a) The original time series is divided into unit fragments using the start time and end time of the two unit steps independently.

(b) All unit pieces are scaled using spline interpolation with standard unit step spacing d. It is then separated into three sensing modes: pressure, acceleration and rotation.

(c) For each sensing mode, the sized pieces of the two feet are joined together.

연속 단위 스텝을 하나의 샘플로 결합하여 참조 데이터 세트를 생성한다.Also,

Consecutive unit steps are combined into one sample to create a reference data set.

6 shows the effect of the amount of information of a unit step on classification accuracy using a reference data set, and the amount of information is proportional to the k-value.

를 전체

에 관한 새 레이블

로 다시 레이블을 지정한다. 여기서,

이다.Specifically, a series of unit steps

full

new label about

label it again with here,

to be.

로 표현할 수 있다.So the dimensions of the data set are

can be expressed as

where w = 8 for pressure and w = 3 for acceleration and rotation.

The network design will be described as follows.

The data set contains time series measured using tri-modal sensing.

In the present invention, since the pressure between one foot and different positions on the ground was measured in the insole during the same gait cycle, eight pressure values correlated with each other are used. Similarly, acceleration values in 3D space are correlated, and rotation values in 3D space are also correlated.

Considering these characteristics, we design a predictive network model using CNN and RNN.

,가속도

및 회전

의 단위 스텝이며 모델의 출력은 소프트 맥스 확률 u의 벡터로 다음과 같이 정의된다.The input of the network model according to the invention is a pressure in standard form

,acceleration

and rotation

is a unit step of and the output of the model is a vector of soft max probability u, defined as

, RNN 만 활성화 된 경우

, CNN과 RNN이 모두 활성화 된 경우

표기법을 사용한다.If only CNN is enabled

, if only RNN is active

, when both CNN and RNN are enabled

use the notation.

In addition, the proposed network model is designed to be applicable to single-mode and dual-mode detection.

,

,

, 바이 모달 감지 모델의 경우

,

,

,

,

,

으로 표기한다.In case of uni-modal detection model,

,

,

, for the bimodal sensing model

,

,

,

,

,

marked with

A conceptual diagram of the network model is shown in FIG. 7 .

7 is a configuration diagram of a network model according to an embodiment of the present invention.

A convolutional neural network applied to the present invention will be described as follows.

A convolutional neural network consists of a feature extractor and a fully connected layer. The feature extractor consists of three steps: a filter layer, a non-linearity activation function layer, and a feature pooling layer.

The CNN applied to the present invention is composed of the same layer independent but for each detection mode, and the network includes three consecutive one-dimensional (1D) convolutional layers.

인 32개의 필터가 포함되고, 두 번째 계층에는 크기가 20 × 32인 64개의 필터가 포함되며, 세 번째 계층에는 크기가 20 × 64인 128개의 필터가 포함된다.The first convolutional layer has a size

32 filters are included, the second layer contains 64 filters of size 20 × 32, and the third layer contains 128 filters of size 20 × 64.

의 너비와 같다.For the first layer, the filter width is in the standard input format

equal to the width of

For the second and third layers, the filter width is equal to the number of filters in the previous convolutional layer.

The convolution operation between the filter and the input produces a single scalar value and concatenates a series of scalar values to provide a feature vector. The filtering stride for all convolutional layers is set to 1 and the padding size is determined to keep the output the same height as the input.

Therefore, the dimensions of the feature map after each of the three convolutional layers are 87×32, 87×64, and 87×128.

인 피쳐 벡터를 생성하기 위해 평면화된다.After the third convolutional layer, the feature map is

It is flattened to create an in-feature vector.

In the case of bi-modal and tri-modal sensing, feature vectors of different sensing modes are connected to one vector that becomes the input of a fully connected layer, and Rectifier Linear Unit (ReLU) is used as an activation function to avoid the vanishing gradient phenomenon. use

A recurrent neural network will be described as follows.

Similar to CNN, RNN consists of the same layer for each detection mode, and the network model includes two consecutive Long Short-Term Memory (LSTM) layers.

Unlike other RNN models, LSTM can be applied to long-term time series data by utilizing an internal memory device designed to overcome the 'error back-flow problem'.

In the network according to the present invention, there are 64 memory units in which forget, input and output gates are activated using a hard sigmoid function in each LSTM layer. Set the iterative dropout ratio to 0.2 to avoid overfitting problems.

이다.The input dimension to the first LSTM layer is

to be.

For a given row of input data, the first LSTM layer is designed to produce a single scalar value per unit of memory, and then concatenates a series of scalar values to provide an output vector.

이다.Thus, the output dimension of the first LSTM layer is the input to the second LSTM layer.

to be.

Unlike the first LSTM layer, the second LSTM layer only returns the last scalar value of the output vector per unit of memory. Therefore, the output dimension of the second LSTM layer is 64.

In the case of dual-mode and tri-mode sensing, the output vectors of different sensing modes are concatenated into one vector to become the input of a fully connected layer.

The overall connected network will be described as follows.

A fully connected network includes a fully connected layer, a dropout layer, and a soft max layer. The fully connected layer contains 256 nodes, and the activation function of the node is ReLU.

The individual nodes are kept with a probability of 0.7, so a reduced layer remains, and is indicated as a dropout layer in FIG. 5 .

This 'dropout' procedure is used to alleviate the over-fitting problem and improve the regularization performance.

The soft max layer includes n nodes, where n is the number of user identifiers. In this way, a user identifier can be selected by calculating a normalized probability using the soft max function.

The averaging ensemble model will be described as follows.

The network model according to the present invention selects a user identifier by independently utilizing CNN and RNN.

We propose an average ensemble model to aggregate the predictions of CNNs and RNNs and provide one final prediction. Since both CNN and RNN have the same soft max layer at the end, we can compute the average probability of CNN and RNN as follows:

In order to confirm the performance of the apparatus and method for multimodal sensor-based gait pattern classification using the deep learning ensemble according to the present invention, a unique sensing method (single, double and triple) and network design (CNN, RNN) according to the present invention and Ensemble), the results of comparing the identification accuracy are as follows.

Gait information was collected from 30 adults aged 20-30 years, and the data were measured using an insole while the participants walked for approximately 3 minutes. Data collected during the 3-minute walk included an average of about 160 step steps, while the full data set included a total of 4750 step steps.

,

,

이고, 각각

이다.Let d = 87 in the experiment, and the dimensions of the data set for pressure, acceleration and rotation are

,

,

and each

to be.

8 is a block diagram of dividing data into training and test data sets.

As shown in Figure 8, training and test data sets are generated independently for three types of Monte Carlo Cross-Validation (MCCV) methods: MCCV (30%), Sub-MCCV (50%), and MCCV (50%). did.

A sub-MCCV set was generated to verify the performance of the system with a limited number of samples from the training data set. MCCV used 100% of the samples in the training or test data set, whereas Sub-MCCV used only about 84% of the total sample.

The first data set was generated by selecting 30% of the total sample as test data and the remaining data as training data, denoted as MCCV (30%).

The second data set was generated from a subset of the total sample by selecting the same number of samples for the test and training data, and we denote this set as Sub-MCCV (50%).

The third data set was generated by selecting 50% of the total sample as test data and the rest as training data, and denotes this set as MCCV (50%).

Table 1 shows the number of samples for the various k-values and validation methods. Following the procedure described above, the generation of three types of data sets was repeated 20 times. For each data set, the proposed network was independently trained and tested, and then the average identification accuracy was summarized for each type of MCCV method.

The latent space and training time are as follows.

9 is a t-dispersion stochastic neighbor embedding (t-SNE) plot of a unit feature vector, where t-SNE performs non-linear dimensionality reduction. The feature vector is the output of the fully connected layer with 256 units in the CNN and RNN architectures of FIG.

Each color in Fig. 9 represents a participant, and dots of the same color correspond to the participant's unit step.

All unit steps of each participant are clearly grouped together and unit steps of all participants are clearly clustered.

9 is a schematic diagram showing a t-SNE plot for the output of a fully connected layer with 256 units in CNN and RNN architectures.

For k = 1 and trimodal detection in the experiment, the average training time of the ensemble model is 28.48 minutes.

Using the generated data set for identification accuracy verification, we evaluate the performance of the proposed framework by considering three scenarios: single, double and triple sensing schemes in pressure, acceleration and rotation data.

We train CNN and RNN networks independently for unique k values of 1–4 using a given training data set generated by one of three different MCCV methods.

The prediction result of the ensemble network was determined by averaging the soft max scores of the pre-trained CNN and RNN networks.

10 shows the identification accuracy of the proposed method using tri-modal sensing for different k-values, (a) MCCV (30%), (b) Sub-MCCV (50%), (c) MCCV (50%).

The first scenario considered in tri-modal sensing is that all forms of pressure, acceleration and rotation data can be utilized. 10 shows the identification accuracy of CNNs, RNNs and ensemble networks using tri-modal sensing.

Detailed information on the identification accuracy is shown in Table 2.

The scenario considered in bimodal sensing is that a combination of the two approaches can be utilized in pressure, acceleration and rotation data.

11 shows the identification accuracy of CNNs, RNNs and ensemble networks using bi-modal sensing. is accuracy.

Table 3 summarizes the identification accuracy for k = 1.

12A and 12B are graphs comparing classification accuracy using an apparatus for classifying a gait pattern based on a multi-modal sensor using a deep learning ensemble according to the present invention.

It can be seen that the present invention can accurately detect a person's gait cycle even if the sensor noise is included in the original gait information by using the pressure data.

It is the use of ensemble models based on CNNs and RNNs that use different sensing data individually or together to identify individuals using data from a single gait cycle. shows the highest performance compared to

The apparatus and method for gait pattern classification based on multi-modal sensor using a deep learning ensemble according to the present invention described above are to provide a framework for identifying an individual in gait information, and to walk using a sensor of a participant's shoe insole. It collects information and classifies gait patterns using a combination of Convolutional Neural Network (CNN) and Recurrent Neural Network (RNN).

As described above, it will be understood that the present invention is implemented in a modified form without departing from the essential characteristics of the present invention.

Therefore, the specified embodiments are to be considered in an illustrative rather than a restrictive point of view, the scope of the present invention is indicated in the claims rather than the foregoing description, and all differences within the scope equivalent thereto are included in the present invention. will have to be interpreted.

10. Smart guide 20. Data collection unit
30. Data standardization unit 40. CNN learning classification unit
50. RNN learning classification unit 60. gait classification result output unit

Claims (13)

a data collection unit that collects gait data from the smart insole;
a data generator defining a section of a unit step in information about walking and adjusting the size of all unit steps to a standard length to generate a data set;
A CNN learning classifier and RNN learning classifier that are each independently trained using the data set generated by the data generator, and apply an average ensemble model to provide one final prediction by performing CNN learning classification and RNN learning classification, respectively. ;
A gait classification result output unit that determines and outputs the type of gait using the value output from the fully connected network through the CNN learning classification unit and the RNN learning classification unit;
The CNN learning classifier and the RNN learning classifier are independently trained using the same training data set, and individual softmax scores in the test phase are calculated by taking the average of the softmax scores in the CNN and RNN. A device for gait pattern classification based on multimodal sensors using a deep learning ensemble of The method of claim 1, wherein the data generator comprises:
A detection mode separation unit that defines the section of a unit step in information about walking, uses spline interpolation to adjust the size of all unit steps to a standard length, and separates it into three sensing modes: pressure, acceleration, and rotation; ,
A device for classifying a gait pattern based on a multimodal sensor using a deep learning ensemble, characterized in that it includes a data set generator that connects unit steps of the same form for both feet and generates a data set. delete The method according to claim 1, wherein the univariate time series and the multivariate time series are each

and

indicated as,
Perform a convolution operation using the average of the 8 pressure values and the Gaussian function to determine the unit step,

is the average of eight pressure values,

here

, the convolution operation is

An apparatus for classifying a gait pattern based on a multimodal sensor using a deep learning ensemble, characterized in that it is performed as 5. The method of claim 4, wherein the unit step,
sorted list for each paw

, where, for every time t

and

becoming

in case of,
time series of unit steps

is defined as, where,

ego,
For discrete variables, the sampling rate of the insole is 100 Hz and the standard length is

because it is defined as

and where

is a device for multimodal sensor-based gait pattern classification using a deep learning ensemble, characterized in that it is the unit step of both feet of all participants. The method of claim 5, wherein the i-th unit step sensing m is

is displayed as
here

, and the dimensions are

and

Apparatus for classification of gait patterns based on multimodal sensors using deep learning ensembles, characterized in that The method of claim 6, wherein the input of the CNN learning classifier and the RNN learning classifier,
standard form of pressure

,acceleration

and rotation

is the unit step of the model

The output of is a vector of soft max probabilities u,

is defined as

the output of the model when given as input

Apparatus for multimodal sensor-based gait pattern classification using a deep learning ensemble, characterized in that is expressed as a vector u representing the probability for each class calculated by the softmax activation function. The deep learning ensemble according to claim 1, wherein the fully connected network includes a fully connected layer, a dropout layer, and a soft max layer, and the fully connected layer includes 256 nodes and the activation function of the node is ReLU. A device for gait pattern classification based on a multimodal sensor using The method of claim 1, wherein the CNN learning classifier and the RNN learning classifier aggregate the predictions of the CNNs and RNNs and build an average ensemble model to provide one final prediction,

calculated as,
Here, if only CNN is enabled

, if only RNN is enabled

, when both CNN and RNN are enabled

Apparatus for classification of gait patterns based on multimodal sensors using deep learning ensembles, characterized in that collecting information about the user's gait from the smart insole in the data collection unit;
defining a section of a unit step in information about walking in a data generator;
adjusting the sizes of all unit steps to standard lengths by using spline interpolation in the data generating unit, and generating a data set by connecting unit steps of the same form for both feet;
Using the generated data set, the CNN learning classification unit and the RNN learning classification unit perform CNN learning classification and RNN learning classification, and input to a fully connected network that can classify the type of gait, and the fully connected network in the gait classification result output unit Including; determining the type of gait using the value output from
For CNN learning classification and RNN learning classification in CNN learning classifier and RNN learning classifier, CNN and RNN are trained independently using the same training data set, and individual softmax scores in the test phase are And A method for multimodal sensor-based gait pattern classification using a deep learning ensemble, characterized in that it is calculated by taking the average of the soft max scores in the RNN. 11. The deep learning of claim 10, wherein the collected data consists of a time series measured by a pressure sensor, a 3-axis accelerometer, and a 3-axis gyroscope installed on a shoe insole, and the gait cycle includes a stance phase and a swing phase. A method for classification of gait patterns based on multimodal sensors using ensembles. 12. The method of claim 11, wherein the unit step is determined using Gaussian filtering to overcome the error of the pressure sensor;
Multimodal sensor-based gait pattern classification using deep learning ensemble, which is characterized by identifying gait patterns from continuous unit step data by constructing an ensemble network using CNN and RNN together with multi-modal sensing data. way for.

delete

Images