KR20240109140A - System for detecting and classifying transitions between various walking environments based on EMG signals and method thereof - Google Patents

Abstract

The present invention relates to a system and method for detecting walking environment transitions using electromyography signals. The walking environment transition detection system includes: a neural network model learning device that trains a neural network model using training data consisting of electromyography signals to classify walking environment transitions; and a walking environment change detection device that detects and classifies the user's walking environment change through the learned neural network model using the user's electromyography signals. Includes. The learning data is composed of a combination of electromyographic signals of the extensors and flexors of the knee joint, the extensors and flexors of the ankle joint, and the extensors and flexors of the midfoot joint, and each electromyographic signal is generated in a walking environment consisting of a single terrain. It is characterized by being labeled as one of the walking tasks, and walking tasks in a walking environment transition consisting of successive different terrains.

Description

System for detecting and classifying transitions between various walking environments based on EMG signals and method thereof}

The present invention relates to a walking environment transition detection system and method using electromyography signals, and more specifically, to artificial intelligence using learning data consisting of a combination of electromyographic signals of extensor and flexor muscles for the knee joint, ankle joint, and metatarsal joint. The present invention relates to a walking environment transition detection system and method using electromyography signals that can accurately detect and classify walking environment transitions by deep learning a neural network model.

Recently, research on robotic walking assistance exoskeletons or prosthetics that enables natural movement of robots through communication between users and robots has been attracting attention. Human-robot interaction is a technology that allows humans and robots to recognize intentions and engage in cognitive interaction through various communication channels.

Here, intention recognition is an important issue to achieve synchronization between robot movements and actual humans.

Meanwhile, wearable walking assistance devices help users with reduced physical abilities walk safely. Pedestrians encounter various walking environments such as flats, stairs, and hills in their daily lives, and they continue to walk on the current terrain or cross terrain that is different from the existing terrain. Wearable walking aids have precise control strategies depending on the type of terrain. Therefore, the wearable walking assistance device worn by the pedestrian must secure information about the pedestrian's intention and future situation and perform an accurate control strategy accordingly.

For this reason, in order to appropriately and safely control the walking assistance device according to the walking environment, it is necessary to accurately recognize the intent of the pedestrian wearing the walking assistance device and the walking environment.

Based on bioelectrical signals such as electromyography (EMG) signals, electroencephalography (EEG) signals, and electrooculogram (EOG), research has been conducted on how to recognize user movement intentions in robotic devices. And, intent recognition based on electromyography signals has been used for communication between devices and human movements.

In this way, electromyography signals containing neural information enable the pedestrian's motion intention to be predicted in advance just before the person moves. However, mechanical sensors can measure signals as force or torque through contact with the external environment after a person moves. Therefore, in the control system, electromyography signals provide more useful information for estimating the pedestrian's walking speed and slope than mechanical sensor signals, thereby improving synchronization between pedestrians and machines such as walking assistance devices.

Therefore, surface electromyography (sEMG) signals are electrical signals containing neural information related to human movement, and by analyzing these surface electromyography signals, they can be used as signals to recognize human movement intentions, classify movements, and control assistive devices. there is.

However, because surface EMG signals have non-linear and complex patterns, it is very difficult to analyze and classify surface EMG signals.

Meanwhile, neural networks have the ability to learn and analyze complex systems, and have recently been widely applied in fields such as pattern recognition and adaptive control. According to a previous study, a pattern recognition method using surface electromyography signals was performed and a machine learning algorithm was applied to classify nine hand movements. In addition, the electromyography signals of the lower extremities measured during walking were applied to machine learning to classify walking stages.

Korean Patent Publication No. 10-2022-0152725 Korean Patent Publication No. 10-2352537 Korean Patent Publication No. 10-2302719

In order to solve the above-mentioned problems, the present invention deep-learns a neural network model, and provides a walking method based on electromyography signals that detects and classifies the user's various walking environments and transitions between various walking environments using only the user's electromyography signals. The purpose is to provide an environmental transition detection system and method.

The walking environment transition detection system according to the first feature of the present invention for achieving the above-mentioned technical problem is a neural network that models the walking environment transition to classify the walking environment transition by learning a neural network model using learning data consisting of electromyography signals. model learning device; and a walking environment change detection device that detects and classifies the user's walking environment change through the learned neural network model using the user's electromyography signals.

In the walking environment transition detection system according to the first feature described above, the learning data consists of a combination of electromyographic signals of the extensors and flexors of the knee joint, the extensors and flexors of the ankle joint, and the extensors and flexors of the metatarsal joint. under,

The electromyography signals constituting the learning data are preferably labeled as one of walking tasks in a walking environment with a single terrain and walking tasks in a walking environment transition with successive different terrains.

In the walking environment transition detection system according to the first feature described above, the walking environment composed of the single terrain includes flat ground, climbing stairs, descending stairs, uphill, and downhill, and the walking environment transition composed of the continuous different terrains. It is preferable that it consists of a combination of two of the following: flat ground, climbing stairs, going down stairs, uphill, or downhill.

In the walking environment transition detection system according to the first feature described above, the neural network model learning device includes a preprocessing module that preprocesses and provides learning data; And a learning module that trains a neural network model using preprocessed learning data,

The preprocessing module includes a band-pass filter for rectifying electromyography signal data; A low-pass filter for filtering the rectified EMG signal data; a normalization module that normalizes the filtered EMG signal data to the peak EMG amplitude of each pedestrian walking on level ground; It is desirable to have a.

The walking environment transition detection method according to the second feature of the present invention includes the steps of (a) using learning data consisting of electromyography signals to train a neural network model to classify the walking environment transition; and (b) detecting and classifying a change in the user's walking environment using the learned neural network model using the user's electromyography signals.

In the walking environment transition detection method according to the second feature described above, the learning data consists of a combination of electromyographic signals of the extensors and flexors of the knee joint, the extensors and flexors of the ankle joint, and the extensors and flexors of the metatarsal joint. under,

The electromyography signals constituting the learning data are preferably labeled as one of walking tasks in a walking environment with a single terrain and walking tasks in a walking environment transition with successive different terrains.

In the walking environment transition detection method according to the second feature described above, the walking environment composed of the single terrain includes flat ground, climbing stairs, descending stairs, uphill, and downhill, and the walking environment transition composed of the continuous different terrains. It is preferable that it consists of a combination of two of the following: flat ground, climbing stairs, going down stairs, uphill, or downhill.

In the walking environment transition detection method according to the second feature described above, step (b) includes: (b1) preprocessing learning data; and (b2) learning a neural network model using the preprocessed training data,

In the step (b1), the EMG signal data is rectified using a band-pass filter, the rectified EMG signal data is filtered using a low-pass filter, and the filtered EMG signal data is used for each pedestrian's level walking. It is desirable to normalize to the peak EMG amplitude.

The walking environment transition detection system based on electromyography signals according to the present invention uses electromyography signals consisting of a combination of the extensors and flexors of the knee joint, the extensors and flexors of the ankle joint, and the extensors and flexors of the metatarsal joint. By learning a neural network model, it is possible to obtain classification accuracy that is almost similar to that learned using the muscles of all joints. Therefore, the walking environment change detection system according to the present invention can accurately detect the walking environment change using a minimum of electromyography signal data.

Figure 1 is a configuration diagram showing a walking environment transition detection system based on electromyography signals according to a preferred embodiment of the present invention.
Figure 2 is a schematic diagram showing the objects of EMG signals used to detect a walking environment change in the walking environment change detection system based on EMG signals according to a preferred embodiment of the present invention.
Figure 3 is a schematic diagram illustrating various walking environments and various walking environment transition states to be detected and classified in the walking environment transition detection system based on electromyography signals according to a preferred embodiment of the present invention.
Figure 4 is a block diagram illustrating a neural network model in the walking environment transition detection system based on electromyography signals according to a preferred embodiment of the present invention.
Figure 5 shows the errors for identifying 13 walking tasks by using the entire profile of EMG signals of all muscles during the stance phase as input in the walking environment transition detection system based on electromyography signals according to a preferred embodiment of the present invention. This is a diagram showing a confusion matrix.
Figure 6 is a graph showing the classification accuracy when using the activity of the flexor and extensor muscle groups for each joint in the walking environment transition detection system based on electromyography signals according to a preferred embodiment of the present invention.
Figure 7 shows a walking environment transition detection system based on electromyography signals according to a preferred embodiment of the present invention, using the entire profile of muscle activation of the extensor muscles of the knee joint, the extensor muscles of the ankle joint, and the flexor muscles of the metatarsal joint during the stance phase. This is a diagram showing the error matrix for identifying 13 walking tasks.
Figure 8 shows classification accuracies in a continuous walking environment, a stairs-to-flat transition environment, and a hill-to-flat transition environment according to each muscle combination in the walking environment transition detection system based on electromyography signals according to a preferred embodiment of the present invention. This is the graph shown.

Hereinafter, a walking environment transition detection system and method based on electromyography signals according to a preferred embodiment of the present invention will be described in detail with reference to the attached drawings.

Figure 1 is a configuration diagram showing a walking environment transition detection system based on electromyography signals according to a preferred embodiment of the present invention. Referring to FIG. 1, the walking environment change detection system 1 according to the present invention includes a signal collection device 10, a database 20 including learning data, a neural network learning device 30, and a walking environment change detection device 40. ) is provided. The walking environment transition detection system according to the present invention can be implemented by a computing system capable of learning a neural network model.

The database 20 stores and manages learning data acquired by a signal collection device, and the learning data includes pedestrians acquired in a walking environment state consisting of a single terrain and a walking environment transition state consisting of continuous different terrains. Electromyography signals include electromyography signals for a combination of extensors and flexors of the knee joint, extensors and flexors of the ankle joint, and extensors and flexors of the metatarsal joint. The walking environment consisting of the single terrain includes flat land, climbing stairs, going down stairs, uphill, and downhill, and the walking environment transition consisting of the continuous different terrains includes two of flat ground, climbing stairs, going down stairs, uphill, and downhill. A combination is preferable. The extensors of the knee joint include the rectus femoris, vastus medialis, and vastus lateralis, the extensors of the ankle joint include the medial gastrocnemius, lateral gastrocnemius, and gastrocnemius, and the flexors of the metatarsal joint include the flexor digitorum longus.

To distinguish between continuous walking situations in a walking environment consisting of a single terrain, the extensor muscles of the ankle joint are an important parameter. However, the important parameters that distinguish between continuous walking in a walking environment consisting of a single terrain and switching between different walking environments in succession are the extensor muscles of the knee joint and the flexor muscles of the metatarsal joint. Therefore, although judgment using data from all muscles shows the best results, it is important to improve ease of control by reducing input signals in the design and control of auxiliary devices such as prosthetics. Therefore, the walking environment transition detection system according to the present invention uses a combination of electromyographic signals for the extensors of the knee joint, the extensors of the ankle joint, and the flexors of the metatarsal joint in order to simultaneously implement fast and accurate control of the assistive device. do.

Figure 2 is a schematic diagram showing the objects of EMG signals used to detect a walking environment change in the walking environment change detection system based on EMG signals according to a preferred embodiment of the present invention. In Figure 2, (a) is the knee joint, (b) is the ankle joint, and (c) is the metatarsal joint.

Figure 3 is a schematic diagram illustrating various walking environments and various walking environment transition states to be detected and classified in the walking environment transition detection system based on electromyography signals according to a preferred embodiment of the present invention. Referring to Figure 3, the 13 walking tasks to be detected by the system according to the present invention are (a) walking on level ground (FG), (b) climbing stairs (US), (c) descending stairs (DS), ( d) uphill walking (UH), (e) downhill walking (DH), (f) transition from level walking to stair climbing (FG-US), (g) transition from stair climbing to level walking (US-FG), (h) transition from walking down stairs to walking on level ground (DS-FG), (i) transition from walking on level ground to walking down stairs (FG-DS), (j) transition from walking on level ground to walking uphill (FG-UH). , (k) transition from uphill walking to level walking (UH-FG), (l) transition from downhill walking to level walking (DH-FG), (m) transition from level walking to downhill walking (FG-DH). )am.

The signal collection device 10 collects EMG signals measured by EMG sensors attached to the pedestrians' bodies and stores them in the database. The electromyography sensors include medial gastrocnemius (MG), lateral gastrocnemius (LG), gastrocnemius (Sol), rectus femoris (RF), vastus medialis (VM), vastus lateralis (VL), semimembranosus (ST), and biceps femoris. It is attached to (BF), tibialis anterior (TA), flexor hallucis longus (FHL), and extensor digitorum longus (EDL), and measures muscle activity data during walking. It is preferable to collect the muscle activity data during the stance phase when walking in each walking environment and in the transition state between each walking environment.

The neural network learning device 30 includes a preprocessing module 300 that preprocesses and provides learning data, and a learning module 310 that trains a neural network model using the preprocessed learning data.

The preprocessing module 300 filters and rectifies the EMG signal data using a band-pass filter of 20 to 500 Hz, filters the rectified EMG signal data using a low-pass filter with a cutoff frequency of 10 Hz, and filters the EMG signal data using a low-pass filter with a cutoff frequency of 10 Hz. Signal data are normalized to each pedestrian's peak EMG amplitude while walking on level ground. Furthermore, it is desirable that all EMG signal data be interpolated to match the size of the input data to train and evaluate the neural network model.

Each preprocessed EMG signal data is separated into each individual muscle, and the EMG signal data separated into individual muscles is labeled as one of the walking tasks in the walking environment and walking environment transition state, and is used as input data for the neural network model. Some of the learning data is used as input data for training the neural network model, and the rest of the learning data is used as input data for evaluating the neural network model.

The learning module learns a neural network model by training and evaluating the neural network model using labeled EMG signal data.

The neural network model for identifying a walking task according to the present invention includes an input layer, at least three or more hidden layers, and an output layer, wherein the output layer is used for walking in a plurality of walking environments and in a switching state of a plurality of walking environments. It has multiple output nodes including walking.

Figure 4 is a block diagram illustrating a neural network model in the walking environment transition detection system based on electromyography signals according to a preferred embodiment of the present invention. Referring to (a) of FIG. 4, the neural network model according to the present invention may be composed of an input layer, three hidden layers, and an output layer. Figure 4(b) shows an exemplary neuron structure. In order to prevent the gradient loss problem, a linear unit rectified in the neural network is used as an activation function, and a dropout technique is used to prevent the overfitting problem. It is characterized by using .

The walking environment transition detection system using the neural network model according to the present invention can determine the current walking environment and walking environment transition with high accuracy, and muscle activation data measured from the extensor muscles (RF, VL, VM) of the knee joint Achieves a 95.4% success rate when used.

Figure 5 shows the errors for identifying 13 walking tasks by using the entire profile of EMG signals of all muscles during the stance phase as input in the walking environment transition detection system based on electromyography signals according to a preferred embodiment of the present invention. This is a diagram showing a confusion matrix. Referring to Figure 5, the muscle sets according to the order of accuracy for identifying each current walking environment and walking environment transition are the extensors of the ankle joint (LG, MG, Sol), and the extensors of the knee joint (RF, VL, VM). ), knee joint flexors (ST, BF), MTP flexors (FHL), ankle joint flexors (TA), and MTP extensors (EDL).

Figure 6 is a graph showing the classification accuracy when using the activity of the combination of flexors and extensors for each joint in the walking environment transition detection system based on electromyography signals according to a preferred embodiment of the present invention. In Figure 6, the flexors of the knee joint represent the semitendinosus and biceps femoris, the extensors of the knee joint represent the rectus femoris, vastus medialis, and vastus lateralis, the flexors of the ankle joint represent the tibialis anterior muscle, and the extensors of the ankle joint. represents the soleus, medial gastrocnemius, and lateral gastrocnemius, and the flexor and extensor muscles of the MTP joint represent the flexor pollicis longus and extensor digitorum longus, respectively.

Referring to Figure 6, when using the activity of the combination of flexors and extensors for each joint, when learning and classifying a neural network model based on the activation of LG, MG, and Sol, which are the combination of extensor muscles of the ankle joint, the classification accuracy is A classification performance of 78.1% was achieved.

Figure 7 shows that in the walking environment transition detection system based on electromyography signals according to a preferred embodiment of the present invention, 13 walking tasks are identified using the entire profile of knee extensor, ankle extensor, and MTP flexor muscle activation during the stance phase. This is a diagram showing the error matrix for: Referring to Figure 7, when classifying the current walking environment and walking environment transition according to the combination of the extensor group and flexor group for each joint, the extensors of the knee joint (RF, VL, VM) and the extensors of the ankle joint (LG, The classification accuracy for the combination of MG, Sol), and MTP joint flexors (FHL) was 90.9%.

Figure 8 shows classification accuracies in a continuous walking environment, a stair-level transition environment, and a hill-flat transition environment according to each muscle combination in the walking environment transition detection system based on electromyography signals according to a preferred embodiment of the present invention. This is the graph shown.

In Figure 8, (a) is a combination of the extensors of the knee joint, (b) is a combination of the extensors of the knee joint, extensor muscles of the ankle joint, and flexors of the MTP joint, and (c) is a combination of all muscles. It shows classification accuracies in continuous walking tasks, transition stair-walking tasks, and transition hill-walking tasks.

Referring to Figure 8, as a result of comparing the classification accuracy in each environment, it can be seen that the classification accuracy in the hill-flat transition environment is the lowest. Additionally, referring to Figure 8, it can be seen that when using the muscle combination in (b), classification accuracy is almost similar to when using a combination of all muscles.

Although the present invention has been described above with a focus on preferred embodiments, this is only an example and does not limit the present invention, and those skilled in the art will understand that it does not deviate from the essential characteristics of the present invention. It will be apparent that various modifications and applications not exemplified above are possible within the scope. In addition, these variations and differences in application should be construed as being included in the scope of the present invention as defined in the appended claims.

1: Walking environment transition detection system
10: signal collection device
20: database
30: Neural network learning device
300: Preprocessing module
310: Learning module
40: Walking environment transition detection device

Claims (10)

A neural network model learning device that trains a neural network model using learning data consisting of electromyography signals to classify walking environment transitions; and
a walking environment change detection device that detects and classifies a user's walking environment change using the learned neural network model using the user's electromyography signals;
A walking environment transition detection system comprising: The method of claim 1, wherein the learning data is:
A walking environment transition detection system characterized by a combination of electromyographic signals from the extensors and flexors of the knee joint, the extensors and flexors of the ankle joint, and the extensors and flexors of the midfoot joint. The method of claim 2, wherein the electromyography signals constituting the learning data are:
A walking environment transition detection system, characterized in that it is labeled as one of walking tasks in a walking environment consisting of a single terrain, and walking tasks in a walking environment transition consisting of successive different terrains. The method of claim 3, wherein the walking environment consisting of a single terrain includes flat ground, going up stairs, going down stairs, uphill, and downhill,
A walking environment transition detection system, characterized in that the walking environment transition consisting of the continuous different terrains consists of a combination of two of flat ground, climbing stairs, descending stairs, uphill, and downhill. The method of claim 1, wherein the neural network model learning device,
A preprocessing module that preprocesses and provides learning data; and
A learning module that trains a neural network model using preprocessed learning data;
The preprocessing module is,
A band-pass filter for rectifying electromyography signal data;
A low-pass filter for filtering the rectified EMG signal data;
a normalization module that normalizes the filtered EMG signal data to the peak EMG amplitude of each pedestrian walking on level ground;
A walking environment transition detection system comprising: (a) using learning data consisting of electromyography signals, learning a neural network model to classify walking environment transitions; and
(b) detecting and classifying the user's walking environment transition through the learned neural network model using the user's electromyography signals;
A walking environment transition detection method comprising: The method of claim 6, wherein the learning data is:
A walking environment transition detection method comprising a combination of electromyographic signals from the extensors and flexors of the knee joint, the extensors and flexors of the ankle joint, and the extensors and flexors of the midfoot joint. The method of claim 7, wherein the electromyography signals constituting the learning data are:
A walking environment transition detection method, characterized in that it is labeled as one of walking tasks in a walking environment composed of a single terrain, and walking tasks in a walking environment transition composed of successive different terrains. The method of claim 8, wherein the walking environment consisting of a single terrain includes flat ground, going up stairs, going down stairs, uphill, and downhill,
A walking environment transition detection method, characterized in that the walking environment transition consisting of the continuous different terrains consists of a combination of two of flat ground, climbing stairs, descending stairs, uphill, and downhill. The method of claim 6, wherein step (a) is,
(a1) preprocessing the learning data; and
(a2) learning a neural network model using preprocessed learning data;
Equipped with
In step (a1), the EMG signal data is rectified using a band-pass filter, the rectified EMG signal data is filtered using a low-pass filter, and the filtered EMG signal data is used for each pedestrian's level walking. A walking environment transition detection method characterized by normalizing to the peak EMG amplitude.