KR102302719B1 - Apparatus and method for classification of gait type by performing neural network analysis for various detection information - Google Patents

Abstract

The present invention is a technology for collecting gait information using a smart insole equipped with various types of sensors, performing neural network analysis for each sensor type, and classifying the type of gait using the analyzed information, a pressure sensor and an acceleration sensor. , an information collection unit that receives information about a user's gait from an insole including two or more of the gyro sensors, and performs a deep learning model calculation for each type of sensor among the information on gait received from the information collection unit to determine the type of gait It includes a classification unit that classifies

Description

Apparatus and method for classifying gait types by performing neural network analysis for each detection information

The present invention relates to an apparatus and method for classifying gait types by performing neural network analysis for each detection information, and more particularly, to collect gait information using a smart insole equipped with various types of sensors, and to analyze neural networks for each sensor type It is a technology that classifies the type of walking using the analyzed information.

Walking is one of the most important behaviors in daily life. When a problem occurs in the gait pattern, it can cause not only musculoskeletal diseases such as joint deformation, but also mental diseases such as intellectual disability, dementia, and depression. It is an important factor in overall physical health. Accordingly, many studies on gait have been conducted, and in particular, a method for classifying gait types is receiving a lot of attention in the field of medical diagnosis and various types of health care.

In the meantime, various methods have been proposed for measuring gait patterns and automatically classifying gait types based on the data. Paper A practical gait analysis system using gyroscopes proposed a method to distinguish straight and curved gait using gyroscope sensors and pressure sensors. The thesis Assessment of walking features from foot inertial sensing uses a pressure sensor and an IMU sensor to calculate the movement angle of the ankle joint on a treadmill, and based on a certain threshold value, 4 types of walking cycles (stance, heel-off) , swing, heel-strike) was proposed. However, this method judges the gait cycle based on the amount of change in the movement angle of the ankle joint. In this method, since the user arbitrarily sets a threshold, which is a criterion for determining the amount of angular change, there is a limitation in that the gait cycle can be distinguished only in a state in which a specific constraint condition is set. The paper Activity classification using realistic data from wearable sensors attaches various types of sensors (accelerometer, gyroscope, humidity meter, etc.) to 8 body parts and collects behavioral data that can be performed at home, bus, restaurant, and library. , proposed a method for recognizing gait types and behaviors using decision trees and artificial neural networks. However, there are disadvantages in that it is inconvenient to attach too many sensors to the body to collect data and the complexity of the model defined for classifying the data is high.

The paper Statistical analysis of parkinson disease gait classification using artificial neural network proposed a method of classifying the gait patterns of normal people and patients with Parkinson's disease using artificial neural networks. However, there is a limitation in the use environment because the data is collected using 6 infrared cameras after attaching 37 reflective markers to the skin of the experimenter.

The thesis Human activity recognition from accelerometer data using convolutional neural network is a convolutional neural network that classifies acceleration 3-axis (x, y, z-axis) data into three gait patterns (walking, running, standing) using a sensor built into a smartphone. A solution 1D neural network is proposed. However, since this method measures the smartphone in a hand or pocket, the movement of the hand during walking or movement due to the sloshing of the pocket is also reflected in the data, making it difficult to accurately measure the gait pattern and input it in the data pre-processing process. There is a limit in that the classification performance fluctuates depending on how the data is defined.

Meanwhile, various wearable devices such as smart watches, sports bands, and smart insoles are being developed as sensor modules have recently been miniaturized and low-power sensor technologies have been developed. Since the use of wearable sensors has fewer environmental constraints for data collection, it is relatively easy to collect data in daily life. It has the advantage of low processing load.

Registered Patent Publication No. 10-1583369

Accordingly, the present invention has been proposed to solve the various problems of the prior art, and an object of the present invention is to collect gait information using a smart insole with various types of sensors, and perform neural network analysis for each type of sensor. , to provide an apparatus and method for classifying a type of gait using the analyzed information.

In order to achieve the above object, an apparatus for classifying walking types by performing neural network analysis for each detection information according to the technical idea of the present invention is a user's gait from an insole including two or more of a pressure sensor, an acceleration sensor, and a gyro sensor. It characterized in that it comprises an information collection unit for receiving the information about, and a classification unit for classifying the type of walking by performing a deep learning model calculation for each type of sensor among the information about the gait received from the information collection unit.

In addition, the classification unit, a feature extraction unit that performs a deep learning model calculation for each type of sensor received from the information collection unit, a fully connected network capable of classifying the type of walking by connecting the information calculated in the feature extraction unit It may be characterized in that it includes a gait classification unit for determining the type of gait by inputting into the .

In addition, the deep learning model of the extraction unit is composed of a plurality of pieces corresponding to the type of sensor, and each deep learning model is pre-learned with one type of sensor information, and when information about walking is received, a feature map for each deep learning model (feature map) may be output.

In addition, the deep learning model may be characterized as a DCNN (Deep Convolutional Neural Network).

In addition, the learning of the deep learning model may be characterized by using a back-propagation learning algorithm.

In addition, the deep learning model may be characterized by using a Rectifier Linear Unit (ReLU) as an activation function to prevent a vanishing gradient phenomenon.

In addition, the deep learning model may be characterized by performing batch normalization after applying a batch activation function to prevent an internal covariance shift phenomenon and increase learning stability.

In addition, the deep learning model may include a plurality of convolutional layers, and each layer may include a filter of a corresponding feature level.

In addition, the feature map output from the deep learning model may be characterized in that it is in the form of a tensor.

In addition, the gait classification unit may be characterized in that one feature vector is constructed by connecting the feature maps output from each deep learning model of the feature extraction unit, and the feature vector is input to a fully connected layer. have.

In addition, the fully connected layer includes a plurality of computational layers and one output layer, and the computational layer prevents over-fitting problems and improves regularization performance before passing a value to the output layer. In order to improve it, it can be characterized by using the dropout method in which nodes in the layer are randomly selected and deleted as much as the dropout ratio during training, and then only the remaining nodes are used for learning.

In addition, the output layer is composed of as many nodes as the number of classes to be classified, and each node calculates the final output value related to the type of gait by applying the weighted sum of the nodes of the operation layer to a softmax function. can be done with

In addition, the information collection unit further comprises a preprocessor for defining a section of a unit step in the information about the walking received by the information collection unit, dividing the data for each unit step, and normalizing the divided data so that the length of the divided data is the same. can be characterized.

In addition, the preprocessor may represent the detection values for the unit steps for each type of sensor in an array, and convert each array into a line of information using a lexicographic ordering operator.

On the other hand, in order to achieve the above object, the method of classifying walking types by performing neural network analysis for each detection information according to the technical idea of the present invention is that the information collection unit includes two or more of a pressure sensor, an acceleration sensor, and a gyro sensor. Receiving information about the user's gait from the insole; dividing, by a preprocessing unit, data for each unit step by defining a section of a unit step in the information about the gait, and normalizing the divided data to have the same length; and classifying the type of gait by performing a deep learning model calculation for each type of sensor among the normalized gait-related information.

In addition, the step of classifying the type of gait by performing a deep learning model calculation for each type of sensor among the information on the normalized gait by the classification unit includes the deep learning model for each type of sensor included in the information on the normalized gait. outputting a feature map by performing an operation; After linking the feature maps into one, input to a fully connected network capable of classifying the type of gait, and determining the type of gait using the value output from the fully connected network. .

In addition, the deep learning model may be characterized as a DCNN (Deep Convolutional Neural Network).

In addition, the learning of the deep learning model may be characterized by using a back-propagation learning algorithm.

In addition, the deep learning model may be characterized by using a Rectifier Linear Unit (ReLU) as an activation function to prevent a vanishing gradient phenomenon.

In addition, the deep learning model may be characterized by performing batch normalization after applying a batch activation function to prevent an internal covariance shift phenomenon and increase learning stability.

In addition, the deep learning model may include a plurality of convolutional layers, and each layer may include a filter of a corresponding feature level.

In addition, the feature map output from the deep learning model may be characterized in that it is in the form of a tensor.

In addition, the fully connected network includes a plurality of computational layers and one output layer, and the computational layer prevents over-fitting problems and improves regularization performance before passing values to the output layer. In order to improve it, it can be characterized by using the dropout method of learning with only the remaining nodes after randomly selecting and deleting nodes in the layer as much as the dropout ratio during training.

In addition, the output layer is composed of as many nodes as the number of classes to be classified, and each node calculates the final output value related to the type of gait by applying the weighted sum of the nodes of the operation layer to a softmax function. can be done with

According to the apparatus and method for classifying walking types by performing neural network analysis for each detection information according to the present invention,

First, the present invention has an effect that the classification performance of the type of walking is significantly higher than that of the prior art with only one step operation.

Second, the present invention can exhibit a high classification rate with a smaller learning amount than a deep learning network that requires a large amount of data by using a feature extraction method based on discriminant analysis.

Third, the present invention has the effect of quickly performing classification calculation of walking types even with significantly less computing power than the conventional deep learning network by using the analysis-based feature extraction method.

Fourth, if functions related to user identification and disease diagnosis are learned in the classification unit for classifying walking types, there is a possibility that it can be usefully used in the security field and the medical field.

1 is an assembly view of the smart insole FootLogger.
2 is a block diagram of an apparatus for classifying walking types by performing neural network analysis for each detection information according to an embodiment of the present invention.
3 is a view showing that the embodiment of the present invention divided the gait cycle into a stance phase and a swing phase.
4 is an exemplary view showing sensor values detected using the FootLogger insole in an array.
5 is a diagram illustrating a process in which the pre-processing unit of this embodiment removes noise.
6 is a diagram illustrating a process of normalizing sensor values in a row after the preprocessor in this embodiment displays the sensor values in an array;
Fig. 7 is a view showing the overall structure and process for classifying a walking type according to this embodiment.
8 is a diagram illustrating a process in which data arranged in a two-dimensional form for each sensor is input to an individual deep learning model (convolutional layer) and is calculated, and a feature map is extracted from each deep learning model.
9 is a flowchart of a method of classifying a gait type by performing a neural network analysis for each detection information according to an embodiment of the present invention.
10 is a graph showing the classification rate of single-modal DCNN for 1 step (k = 1) to 5 steps (k = 5). (a) is a classification rate using 1000 randomly selected training data samples and 1000 test samples; (b) The classification rate when 7-fold cross-validation was performed.
11 is a graph showing the classification rate of multi-modal DCNN for 1 step (k = 1) to 5 steps (k = 5). (a) is a classification rate using 1000 randomly selected training data samples and 1000 test samples; (b) The classification rate when 7-fold cross-validation was performed.
12 is a diagram showing the structure of a fully connected network that determines the type of gait using feature vectors of single-modal DCNN and multi-modal DCNN as inputs.

An apparatus and method for classifying a gait type by performing a neural network analysis for each detection information according to embodiments of the present invention will be described in detail with reference to the accompanying drawings. Since the present invention can have various changes and can have various forms, specific embodiments are illustrated in the drawings and described in detail in the text. However, this is not intended to limit the present invention to the specific disclosed form, it should be understood to include all modifications, equivalents and substitutes included in the spirit and scope of the present invention. In describing each figure, like reference numerals have been used for like elements.

In addition, unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and should not be interpreted in an ideal or excessively formal meaning unless explicitly defined in the present application. does not

An apparatus and method for classifying a gait type by performing a neural network analysis for each detection information according to an embodiment of the present invention proposes a method for classifying a gait type using various types of sensor data mounted on the smart insole 1 . For data collection, a pressure sensor, an acceleration sensor, and a gyro sensor of FootLogger, a commercial smart insole (1), were used. classify This embodiment is largely composed of a preprocessing step of dividing information detected by continuous walking into unit steps and normalizing the data, and a classification step of extracting features using a deep convolutional neural network and determining the type of walking.

The preprocessing step improves the accuracy of step division of data by removing noise generated during the sensing process based on the swing phase characteristics during the gait cycle. In the classification step, a gait feature map is created using a deep convolutional neural network for each type of sensor installed in the smart insole 1, and each feature map is combined to form a fully connected network that can finally determine the gait type. . Here, a 'fully connected network' means a network that necessarily has a branch between any two nodes.

The excellent classification performance of the present invention was confirmed through an experiment in which 7 types of gait were measured for 14 adults in their 20s and 30s.

An apparatus for classifying a gait type by performing a neural network analysis for each detection information according to an embodiment of the present invention will be described in detail with reference to the drawings.

Referring to FIG. 1 , the insole 1 used in this embodiment is a smart insole called FootLogger manufactured by 3L-Labs. FootLogger has a built-in pressure sensor array consisting of 8 pressure sensors, a 3-axis acceleration sensor array, and a 3-axis gyro sensor array. measure In the case of the pressure sensor, values of 0, 1, and 2 are output according to the strength of the pressure. 0 is the state without pressure, that is, the swing phase (the state with the foot off the ground), and 1 and 2 are the stance phase (the state where the foot is on the ground). It means the strength of the pressure in the standing state). In the experiment, the information detected by the insole 1 was transmitted to the Android smartphone using Bluetooth communication, and transmitted to the device for classifying the walking type of this embodiment using the smartphone application.

Referring to FIG. 2 , an apparatus for classifying a gait type by performing a neural network analysis for each detection information according to an embodiment of the present invention is a user's gait from an insole 1 including at least two of a pressure sensor, an acceleration sensor, and a gyro sensor. An information collection unit 10 for receiving information about , and a classification unit 40 for classifying the type of walking by performing a deep learning model calculation for each type of sensor among the information about walking received from the information collection unit 10 . ) is included.

In addition, a preprocessor (normalizing) that defines a section of a unit step in the information about the walk received by the information collection unit 10, divides the data for each unit step, and normalizes the divided data so that the length of the divided data is the same ( 20) is further included.

The apparatus for classifying walking types may be implemented by installing and executing an application program performing a function of each functional unit in a single server or a plurality of servers connected by a network.

Referring to FIG. 3 , the gait cycle refers to an operation from the moment when one foot touches the ground until it comes off the ground and then touches the ground again. The gait cycle consists of a total of 7 stages (heel strike, foot flat, mid stance, heel off, toe off, mid swing, and late swing). The heel strike stage is a stage indicating the start of the gait cycle, and the heel of one foot is in contact with the ground, and the foot flat stage is a stage in which the sole of one foot is in contact with the ground. In the mid stance phase, the leg of one foot is perpendicular to the ground and stopped, and at this time, the leg of the other foot moves forward. The heel-off stage refers to a time when one heel starts to be lifted off the floor, and the toe-off stage is a stage in which the front toe of one foot falls off the floor and the sole of the other foot touches the ground at the same time. The mid swing stage is a stage in which one foot moves forward with one foot off the ground, and the late swing stage refers to the stage just before the start of the next gait cycle.

The preprocessor 20 of this embodiment defines, based on one foot, heel strike, foot flat, mid stance, heel off, and toe off stages in which the foot is in contact with the ground as a stance phase, and the foot is separated from the ground. The mid swing and late swing phases are defined as the swing phase.

The step unit setting unit 22 of the preprocessor 20 divides the unit step section based on the stance step and the swing step of the gait cycle in order to divide the data continuously measured during the walking time into unit step sections.

4 is an example of gait data measured using the FootLogger insole (1). Among the data of the pressure sensor array, it is observed that the values of all pressure sensors in the array are continuously 0 and the stance stage is divided into a stance stage in which the values of some pressure sensors are measured as 1 or 2. You can check the gait cycle by seeing the steps alternately. In this example, one step was defined from the starting point of the swing phase to the ending point of the stance phase based on the left foot.

The information arranging unit 23 of the preprocessor 20 stores the unit step data samples in the form of a two-dimensional matrix for each of the pressure sensor array, the acceleration sensor array, and the gyro sensor array. A column of the matrix is an index of the sensor, and a row means a measured time point. In the case of the pressure sensor array consisting of 8 sensors per foot, there are a total of 16 rows, and in the case of the acceleration sensor array and the gyro sensor array, there are 6 rows.

Since the swing phase of the gait cycle is when the foot is off the ground, it is ideal for all eight pressure sensors to have a value of 0. However, when data was measured using the actual FootLogger insole (1), non-zero values were sometimes measured in some sensors even in the swing stage due to various factors such as potential difference between sensors included in the insole (1) and heat generation. For example, in an actual experiment, the value of the 3rd pressure sensor in the swing phase was sometimes measured as 1. Since this embodiment divides the unit step based on the swing stage, such noise may cause a false detection in which one swing stage appears to have occurred twice, thereby deteriorating the gait type classification performance of this embodiment.

5 shows a process in which the noise removing unit 24 of the preprocessing unit 20 removes noise generated in the swing stage. First, the point at which the sum of all values of the 8 pressure sensors of the left foot insole (1) becomes 0 is defined as the starting point of the swing phase (SP Flag=True in FIG. 5), and the starting point of the next swing phase is 8 It is defined as the point in time when a non-zero value is measured by two or more sensors among the pressure sensors (SP Flag=False before the start of the swing phase). If the sum of the sensor array is 1 instead of 0 between the starting point of the swing phase and the starting point of the stance phase, the corresponding pressure sensor value is determined as noise and the corresponding value is corrected to 0.

Even the same person may have a different walking speed while measuring data or each time data is being measured. Variations in walking speed according to the measurement time point regardless of the type of gait hinder the extraction of gait characteristics for classifying gait types.

6 shows original data and normalized data for a unit step. The normalization unit 26 of the preprocessor 20 resizes the unit steps from all swing stages to the stance stage based on the shortest unit step time t in order to extract features that are less sensitive to the measurement situation. Normalize so that the length for all steps is the same. The experiments in this example set t to 63. The normalization unit 26 converts the measured values for the normalized unit steps of the pressure sensor array, the acceleration sensor array, and the gyro sensor array into an array of 63×16, 63×6, and 63×6, respectively, and then performs a pre-order operator ( Using the lexicographic ordering operator, it is stored as a vector (x) of 1008×1, 378×1, and 378×1.

The gait information detected using the footlogger insole (1) is time series data in which the sensor value is detected at 0.01 second intervals. Since walking is a continuous motion, the correlation between sensor measurement values at each time point of time series data is relatively large. Therefore, the classification unit 40 of this embodiment designs a deep learning model network that classifies the types of walking based on various types of sensor measurement values using DCNN (Deep Convolutional Neural Network) that can utilize the correlation of data. did.

In general, a convolution-based artificial neural network consists of a feature extractor and a fully connected layer. The feature extractor consists of three steps: a filter layer, a non-linearity activation function layer, and a feature pooling layer.

The classification unit 40 of this embodiment connects the information calculated by the feature extraction unit 42 and the feature extraction unit 42 for performing deep learning model calculation for each sensor information received from the information collection unit 10, It includes a gait classification unit 44 that determines the type of gait by inputting it into a fully connected network capable of classifying the gait type.

Fig. 7 shows the overall structure for classifying the type of walking in this embodiment. In addition, FIG. 8 shows a process in which an individual deep learning model is configured for each type of sensor and a feature map is extracted from each deep learning model. The feature extraction unit 42 first constructs an individual deep learning model network for each sensor information, and each deep learning model independently learns to extract a feature map for each sensor array. That is, the deep learning model of the extraction unit is composed of a plurality of pieces corresponding to the type of sensor, and each deep learning model is pre-learned with one type of sensor information. map) is printed.

The DCNN receives information in the form of a normalized two-dimensional array from the preprocessor 20 as an input, and performs a convolution operation with various filters in a convolution layer. In this embodiment, the data detected for each sensor array was normalized to a t×W size through a preprocessing process and used as an input for DCNN. W is the number of sensors in the sensor array. This example set t to 63. In addition, in this embodiment, corresponding to the footlogger insole 1, W is set to 16, 3, and 3 for the pressure sensor array, the acceleration sensor array, and the gyro sensor array, respectively.

가 된다.In the experiment of this embodiment, in order to determine how many steps are required to extract a gait feature for distinguishing a gait type, a classification experiment was performed on a data sample used as an input of DCNN by increasing the number of steps from 1 step. If k steps are defined as one gait sample, the input data of DCNN is

becomes

), 스트라이드(s)의 세 가지 하이퍼파라미터(hyper-parameter)가 필요하다.The deep learning model of the feature extraction unit 42 of this embodiment includes three convolutional layers corresponding to three types of sensors. Each convolutional layer includes filters of a corresponding feature level. The number of filters to be used (f) and the filter size (

) and stride(s), we need three hyper-parameters.

로 설정하였다. 필터링 스트라이드(stride)는 DCNN의 입력 데이터에 포함된 걸음 수(k)에 따라 각각 다르게 설정하였다(s=1은 k=1,2; s=2는 k=3,4,5).The first convolution layer used a total of 32 types of filters, and the size of the filters was

was set to The filtering stride was set differently according to the number of steps (k) included in the DCNN input data (s=1 is k=1,2; s=2 is k=3,4,5).

For the second and third convolutional layers, 64 and 128 filters were used, respectively. The size of the filter was set differently depending on the size of the output signal in the previous layer. The second convolution layer was set to 1 (filter width) × 20 (filter height) × 64 (number of filters), and the third convolution layer was set to 1 × 20 × 128.

8 shows how each convolutional layer (deep learning model) performs an operation on input data in a two-dimensional form. Red, yellow, and green on the input data mean some of the 32 different kernels in the first layer. When a DCNN-based convolution operation is performed between the kernel and input data, one scalar value is output, and when the convolution operation of different kernels for one input data is completed, a Tensor-type feature map as shown at the bottom This is output.

DCNN was trained using a back-propagation learning algorithm. In the learning process, a vanishing gradient phenomenon may occur. To prevent this, a Rectifier Linear Unit (ReLU) was used as an activation function. Gradient loss is a phenomenon in which when the number of layers of the neural network increases when backpropagating an error, the differential value of the input has a value close to 0, so that the gradient cannot be transmitted.

In addition, in the deep learning model of this embodiment, batch normalization was performed after applying the batch activation function to prevent the internal covariance shift phenomenon and increase learning stability. The internal covariance shift is a phenomenon in which the distribution of input data continues to vary due to the nonlinear activation function used in each layer of the neural network. The batch size was set to 32.

DCNN independently learned for each sensor array outputs a feature map for each sensor type.

The gait classification unit 44 concatenates each feature map in series to form one feature vector, and inputs the feature vector to a fully connected layer. The fully connected layer includes two computational layers except for the output layer. Before passing the value to the output layer, the computation layer randomly selects and deletes nodes in the layer by the dropout ratio during training to prevent over-fitting problems and improve regularization performance. The dropout method is used to learn with only the remaining nodes after doing so. This example was tested while changing the dropout ratio from 0.5 to 0.7.

The output layer consists of as many nodes as the number of classes to be classified. Each node calculates a final output value capable of determining a gait type by applying the sum of weights of nodes of the calculation layer to a softmax function.

Next, a method of classifying walking types by performing neural network analysis for each detection information according to an embodiment of the present invention will be described.

Referring to FIG. 10 , in the method of classifying walking types by performing neural network analysis for each detection information according to an embodiment of the present invention, the information collection unit is obtained from an insole 1 including two or more of a pressure sensor, an acceleration sensor, and a gyro sensor. and receiving information about the user's gait (S120).

In addition, the preprocessor 20 defines a section of a unit step in the walking information, divides the data for each unit step, and normalizes the divided data to have the same length (S140). do.

In addition, the classification unit 40 includes a step (S160) of classifying the type of gait by performing a deep learning model calculation for each type of sensor among the normalized gait information.

In particular, in step S160, performing deep learning model calculation for each type of sensor included in the normalized gait information and outputting a feature map (S162) and linking the feature map into one and determining the type of gait by inputting it into a fully connected network capable of classifying the type, and using the value output from the fully connected network (S166).

At this time, the deep learning model of this embodiment is characterized in that DCNN (Deep Convolutional Neural Network).

In addition, the learning of the deep learning model is characterized by using a back-propagation learning algorithm.

The deep learning model of this example uses a Rectifier Linear Unit (ReLU) as an activation function to prevent a vanishing gradient phenomenon.

In addition, the deep learning model performs batch normalization after applying the batch activation function to prevent the internal covariance shift phenomenon and increase learning stability.

In addition, the deep learning model includes a plurality of convolutional layers, and each layer includes a filter of a corresponding feature level.

The feature map output from the deep learning model is in the form of a tensor.

A fully connected network includes a plurality of computational layers and one output layer. Before passing values to the output layer, the computational layer randomizes the nodes in the layer by a dropout ratio during training to prevent over-fitting problems and improve regularization performance. The dropout method is used to learn with only the remaining nodes after selecting and deleting them.

The output layer consists of as many nodes as the number of classes to be classified, and each node calculates the final output value for the gait type by applying the weighted sum of the nodes of the operation layer to the softmax function.

Experiment.

For data collection, a total of 7 types of gait data were detected as walking, hill climbing, hill descending, stair climbing, stair descending, running, and warning for 14 adults in their 20s and 30s. Data for walking, running, and alerting were measured for 3 minutes, and data from the starting point to the end point were measured regardless of time for hill climbing and descending, and for stair climbing and descending. Each person's gait measurement value was divided, normalized, and stored for each unit step through the preprocessing process of the preprocessor 20 . Information on the data used in the experiment is summarized in Table 1.

walking style number of steps Measurement time and reference walking 2,295 3 minutes rise 1,577 starting point -> arrival point go down 1,586 starting point -> arrival point rise 747 starting point -> arrival point go down 971 starting point -> arrival point running 3,642 3 minutes trot 2,714 3 minutes

The data required for learning and testing were randomly selected and configured by 1000 each from the total data, and the above operation was repeated 20 times to increase statistical reliability to calculate the average classification rate (Fig. 9 (a) and Fig. 10 (a)) )).

In addition, additional K-fold cross-validation experiments were performed (FIGS. 9 (b) and 10 (b)). In the K-fold cross-validation, the total sample was randomly divided into K equal-sized subsets. One subset of the K subsets was used as the test set to validate the model while the remaining K-1 subset was used as the training set.

Then, the cross-validation process was repeated K times. Each subset was used exactly once as a test set. Since K was usually set to the number of classes, a 7-fold cross-validation was performed in the experiment. In addition, to increase the statistical reliability, the above procedure was repeated 10 times and the average classification ratio was calculated.

In order to determine the number of steps required to obtain information that can classify gait types, the classification speed was investigated using a gait sample consisting of steps from one (k = 1) to five (k = 5) steps. The training data samples were normalized to have zero mean and unit variance, and the test data samples were normalized using the mean and variance of the training data samples. Table 2 shows the total number of gait data samples according to the value of k and the number of training data samples and test samples in the 7-fold cross-validation experiment for each k.

k total number of steps number of training samples number of test samples One 13,531 11,598 1933 2 6,742 5,778 964 3 4,476 3,836 640 4 3,347 2,868 479 5 2,671 2,289 382

For classification experiments, Matlab 2018a was used in a Windows environment, and Keras 2.1.5 was used for DCNN-related libraries. In the learning process, categorical cross entropy was used as a loss function to calculate the difference between the output value and the correct label value. An adaptive momentum optimizer was used as a method for optimizing the weights of the network, and the learning rate of the adaptive momentum optimizer was set to 0.0001.

Referring to FIG. 12 , the classification performance is a mono-modal DCNN using a feature map of one type of sensor array data, a bi-modal DCNN using a feature map of two or more types of sensor arrays as a multi-modal DCNN, and three types. The classification rate for tri-modal DCNN using all of the sensor arrays was evaluated, and a discriminant analysis method using NLDA (DA-NLDA) and a comparative experiment were performed.

First, in order to check the classification performance for each type of sensor, the results of classifying the gait type using the independently learned mono-modal DCNN for each of the pressure sensor array, the acceleration sensor array, and the gyro sensor array are shown in Table 3. To determine how many steps are required to extract gait characteristics for classification of gait types, from a gait sample composed of gait measurements for one step (k = 1) to a gait sample including measurements of five steps (l = The recognition rate up to 5) was measured.

k #One #2 #3 #4 #5 Mono-Modal DCNN pressure 84.32 84.90 85.52 88.28 88.87 acceleration 88.17 87.39 90.55 90.99 90.38 gyro 88.16 90.36 91.74 91.84 93.03 DA-NLDA pressure 26.10 77.32 81.68 83.35 84.41

The example of the present invention showed a higher classification performance than DA-NLDA for all cases at a minimum of 4.46% (pressure sensor walking sample with 5 steps) and up to 58.22% (pressure sensor walking sample with one step) higher than DA-NLDA. Considering that the performance of the gyro sensor is somewhat higher than that of other sensors among the feature maps of the three types of sensor arrays, it is judged that the gait pattern created by the direction of movement of the foot during walking is useful for distinguishing the type of gait.

In particular, in the case of a gait sample composed of one step, DA-NLDA had only 26.10% to 43.32% of classification performance, so it was not possible to properly extract features useful for gait type classification. However, the embodiment of the present invention showed a high classification rate of 84.82% to 88.18% depending on the type of sensor only with a walking sample including one step.

10 and 11 are results of using 1000 training samples and 1000 test samples for all k. Referring to the results, it can be seen that the classification rate increases as the number of steps included in the walking sample increases, but when the number of steps is 4 to 5, the increase in the classification rate slows down (pressure sensor, acceleration sensor) or decreases (gyro sensor). could This means that as the number of steps constituting the sample increases, information useful for extracting high-quality features for gait classification increases, but it can be seen that the effect of obtaining additional information on the increase in the number of steps disappears from the gait sample containing 4 or more steps.

The highest classification rate appeared when k=1, because the training data when k=1 is the largest. This means that even if the data is measured for a short period of time, better features can be extracted if the amount of data samples is sufficient.

k #One #2 #3 #4 #5 Bi-Modal ours pressure + acceleration 89.96 90.86 90.82 91.57 93.09 pressure + gyro 90.24 91.19 90.81 92.68 92.39 Acceleration + Gyro 90.03 90.84 91.56 90.88 91.68 Tri-Modal ours Pressure + Acceleration + Gyro 90.22 92.68 92.66 92.98 93.53

Table 4 shows the experimental results of multi-modal DCNN using feature maps of several types of sensors. Multi-modal (bi-modal and tri-modal) DCNN experiments combine feature maps of two or more types of sensors to create a new feature map and classify gait types. Comparing with the results of mono-modal DCNN in Table 3, it can be seen that the overall classification performance of multi-modal DCNN is better, and the classification performance is higher when using three types of sensors than when using two types of sensors. performance was shown.

On the other hand, in the case of multi-modal DCNN, as in the mono-modal DCNN experiment, the higher the number of steps included in the walking sample, the higher the classification rate. In addition, in both bi-modal DCNN and tri-modal DCNN, it can be seen that classification performance is saturated from gait samples containing two or more steps. do.

Although preferred embodiments of the present invention have been described above, various changes, modifications and equivalents may be used in the present invention. It is clear that the present invention can be equally applied by appropriately modifying the above embodiments. Therefore, the above description does not limit the scope of the present invention, which is defined by the limits of the following claims.

1: Smart insole
10: information collection unit
20: preprocessor
22: step unit setting unit
23: information arrangement unit
24: noise removal unit
26: normalization unit
40: classification unit
42: feature extraction unit
44: walking classification unit

Claims (24)

an information collection unit for receiving information about a user's gait from a smart insole in which two or more of a pressure sensor, an acceleration sensor, and a gyro sensor are built-in;
It characterized in that it comprises a classification unit for classifying the type of walking by performing a deep learning model calculation for each type of sensor among the information about the walking received from the information collection unit,
The information collection unit further comprises a preprocessor for defining a section of a unit step from the information about the walking received by the information collection unit, dividing the data for each unit step, and normalizing the divided data so that the lengths of the divided data are all the same. and
The preprocessor is
a step unit setting unit that divides the information on the walking continuously measured during the walking time in the smart insole based on the stance step and the swing step of the walking cycle into unit step sections;
an information arrangement unit for storing data samples for the unit steps in the pressure sensor, the acceleration sensor, and the gyro sensor in the form of a two-dimensional matrix, respectively;
a noise removing unit that removes noise generated in the swing phase of the walking cycle; and
It characterized in that it comprises a; normalization unit for normalizing the data for the unit step based on the time of the shortest unit step,
The classification unit,
a feature extraction unit for performing a deep learning model operation for each type of sensor received from the information collection unit;
It characterized in that it includes a gait classification unit that determines the type of gait by connecting the information calculated in the feature extraction unit and inputting it into a fully connected network capable of classifying the type of gait,
The deep learning model of the extraction unit is composed of a plurality of pieces corresponding to the type of sensor, and each deep learning model is pre-learned with one type of sensor information, and when information about walking is received, a feature map for each deep learning model (feature map) It is characterized by outputting a map),
The gait classification unit connects the feature maps output from each deep learning model of the feature extraction unit to form one feature vector, and inputs the feature vector to a fully connected layer,
The fully connected layer includes a plurality of computation layers and one output layer,
Before passing the value to the output layer, the computation layer reduces the nodes in the layer by a dropout ratio during training to prevent over-fitting problems and improve regularization performance. It is characterized by using a dropout method that learns with only the remaining nodes after randomly selecting and deleting them.
The output layer is composed of as many nodes as the number of classes to be classified, and each node calculates the final output value related to the gait type by applying the weighted sum of the nodes of the operation layer to a softmax function. A device for classifying gait types by performing neural network analysis for each detection information. delete delete According to claim 1,
The deep learning model is a DCNN (Deep Convolutional Neural Network) apparatus for classifying a walking type by performing a neural network analysis for each detection information, characterized in that. 5. The method of claim 4,
A device for classifying walking types by performing neural network analysis for each detection information, characterized in that the learning of the deep learning model uses a back-propagation learning algorithm. 6. The method of claim 5,
The deep learning model classifies a gait type by performing a neural network analysis for each detection information, characterized in that it uses a Rectifier Linear Unit (ReLU) as an activation function to prevent a vanishing gradient phenomenon. 7. The method of claim 6,
The deep learning model is a device for classifying gait types by performing neural network analysis for each detection information, characterized in that the batch normalization is performed after applying the batch activation function to prevent the internal covariance shift phenomenon and increase learning stability . According to claim 1,
The deep learning model includes a plurality of convolutional layers, and each layer includes a filter of a corresponding feature level. An apparatus for classifying a gait type by performing a neural network analysis for each detection information. According to claim 1,
A device for classifying walking types by performing neural network analysis for each detection information, characterized in that the feature map output from the deep learning model is in the form of a tensor. delete delete delete delete According to claim 1,
The pre-processing unit represents a detection value for a unit step for each type of sensor in a two-dimensional array, and uses a lexicographic ordering operator to convert each two-dimensional array into a line of information. Neural network analysis by detection information A device that classifies gait types by performing Receiving, by an information collection unit, information about a user's gait from a smart insole in which two or more of a pressure sensor, an acceleration sensor, and a gyro sensor are embedded;
A preprocessing step comprising: a preprocessing unit defining a section of a unit step in the walking information, dividing the data for each unit step, and normalizing the divided data to have the same length;
The pre-processing step is
dividing the information on the gait continuously measured during the gait time in the smart insole into unit step sections based on the stance stage and the swing stage of the gait cycle in the step unit setting unit;
storing data samples for the unit steps in the pressure sensor, the acceleration sensor, and the gyro sensor in the form of a two-dimensional matrix in the information arrangement unit;
removing noise generated in the swing phase of the gait cycle by a noise removing unit; and
The normalizing step of normalizing the data for the unit step in the normalization unit based on the time of the shortest unit step; characterized in that it comprises a,
It characterized in that it comprises the step of classifying the type of walking by performing a deep learning model calculation for each type of sensor among the information on the normalized walking by the classification unit,
The step of classifying the type of walking by performing a deep learning model calculation for each type of sensor among the information on the normalized walking by the classification unit includes:
outputting a feature map by performing a deep learning model calculation for each type of sensor included in the normalized gait information;
It characterized in that it comprises the step of inputting into a fully connected network capable of classifying the type of gait after linking the feature map into one, and determining the type of gait using the value output from the fully connected network,
The fully connected network includes a plurality of computation layers and one output layer,
Before passing the value to the output layer, the computation layer reduces the nodes in the layer by a dropout ratio during training to prevent over-fitting problems and improve regularization performance. It is characterized by using a dropout method that learns with only the remaining nodes after randomly selecting and deleting them.
The output layer is composed of as many nodes as the number of classes to be classified, and each node calculates the final output value related to the gait type by applying the weighted sum of the nodes of the operation layer to a softmax function. A method of classifying gait types by performing neural network analysis for each detection information. delete 16. The method of claim 15,
The deep learning model is a method of classifying a walking type by performing a neural network analysis for each detection information, characterized in that DCNN (Deep Convolutional Neural Network). 18. The method of claim 17,
The learning of the deep learning model is a method of classifying walking types by performing neural network analysis for each detection information, characterized in that using a back-propagation learning algorithm. 18. The method of claim 17,
The deep learning model classifies a gait type by performing a neural network analysis for each detection information, characterized in that it uses a Rectifier Linear Unit (ReLU) as an activation function to prevent a vanishing gradient phenomenon. 20. The method of claim 19,
The deep learning model is a method of classifying gait types by performing neural network analysis for each detection information, characterized in that batch normalization is performed after applying a batch activation function to prevent the internal covariance shift phenomenon and increase learning stability . 16. The method of claim 15,
The deep learning model includes a plurality of convolutional layers, and each layer includes a filter of a corresponding feature level. A method of classifying a gait type by performing a neural network analysis for each detection information. 16. The method of claim 15,
A method of classifying a walking type by performing a neural network analysis for each detection information, characterized in that the feature map output from the deep learning model is in the form of a tensor. delete delete

Images