CN116340849B - A non-contact cross-domain human activity recognition method based on metric learning - Google Patents

Abstract

A non-contact cross-domain human activity recognition method based on metric learning is characterized in that data enhancement is carried out on collected activity data by adopting a self-encoder, and then recognition of new activity categories which are not in a training set is completed by adopting the metric learning. The method comprises the following specific steps: acquiring corresponding wireless signal data when personnel are active in an indoor environment, and extracting CSI original data from the wireless signal data; performing data preprocessing, data interpolation, unifying data length and data denoising on original CSI data; performing data enhancement on the data with known activity types by using a self-encoder, and expanding a data set; and (3) using the expanded data set training feature extraction network to input the to-be-identified activity data and the support set into the feature extraction network to obtain corresponding features, and using a metric learning method to compare the to-be-identified data features with the support set data features one by one so as to judge the activity type. The method can realize higher recognition precision for untrained activity types, and enhances generalization and robustness.

Description

A non-contact cross-domain human activity recognition method based on metric learning

technical field

The invention relates to the field of activity recognition, in particular to a non-contact cross-domain human activity recognition method based on metric learning.

Background technique

Wi-Fi-based Human Activity Recognition (HAR) aims to identify human activities using Wi-Fi signals. Since it does not require users to wear special equipment and additional deployment costs, Wi-Fi-based human body recognition has received more and more attention in building intelligent applications for human-computer interaction, intelligent surveillance, and intrusion detection. However, because collecting Channel State Information (CSI) data requires a lot of manpower and material resources, the existing public Wi-Fi perception datasets are very scarce. Therefore, Wi-Fi sensing does not have enough labeled data to train a machine learning model with good performance. At the same time, the wireless signal is affected by the physical environment and human behavior. Different types of human body movements (position, posture or movement) during transmission may cause similar signal fluctuations, which undoubtedly makes the process of activity recognition more complicated. Especially in practical applications, the system may need to recognize some activity types that have never been seen in the training phase, and it is a great challenge to recognize unknown activity types.

Traditional Wi-Fi-based recognition usually adopts a supervised method, which can only recognize pre-defined activities, and requires a large amount of data to learn sufficient prior knowledge, and their complicated training process aggravates the cost of model training, making the overall Expenses have increased substantially. These activity recognition methods first collect a large number of labeled samples and then train a traditional machine learning model to identify the type of activity. However, when it is necessary to identify new activity types that did not appear in the training process, since the activity categories of the new data are different from the activity categories of the data in the training set, it is necessary to identify new activity data across categories. At this time, the recognition performance of the model of the above method will decline severely. To address this issue, some recent work collects some new types of activity data that did not appear in the training set to retrain the model. However, the process of retraining undoubtedly increases the overhead of model training.

In the field of computer vision, metric learning is widely used to recognize unseen activities and scenes. The metric learning problem is usually formulated as an optimization problem, optimizing some objective function that measures the similarity of data. Map the data from the original vector space to the hidden space to obtain activity-related features. In response to the above problems, due to the small data sets of Wi-Fi CSI, there may be situations in which unknown types of activities need to be identified in actual scenarios.

Contents of the invention

The purpose of the present invention is to provide a non-contact cross-domain human activity recognition method based on metric learning, which uses an autoencoder to expand the activity data set to enrich the data set to improve the recognition accuracy, and then compares the data through the method based on metric learning The similarity between them is used to realize the function of identifying new activity categories that do not appear in the training set.

A non-contact cross-domain human activity recognition method based on metric learning, comprising the following steps:

Step 1, collect the wireless signal data corresponding to people's activities in the indoor environment, and extract the original signal of CSI from it;

Step 2, performing data preprocessing on the original CSI data collected in step 1, data interpolation with a uniform length, and data denoising to remove noise signals caused by hardware devices;

Step 3, use the autoencoder to perform data enhancement on the data processed in step 2, and generate data related to corresponding activities to expand the data set;

Step 4, use the method based on metric learning to establish an activity recognition model, train the data generated in step 3 and the original data, learn the similarity between the data, train the activity-related feature extraction model, and compare the data to be recognized with the support The characteristics of the collected data are used to identify unknown types of activities.

The beneficial effect that the present invention reaches:

(1) Using the method of metric learning to compare the data features of the unknown type of activities to be identified with the data features of the support set one by one, so as to realize the identification of unknown types of activities.

(2) It solves the problem of unknown type activity recognition in wireless sensing, and can achieve high recognition accuracy for untrained activity types without additional training costs, which enhances the robustness of the activity recognition network.

(3) Inspired by the core idea of metric learning, the network is trained by the criterion of making similar samples closer and dissimilar samples farther apart, and the recognition of unknown types of activities can be realized without complicated training strategies.

Description of drawings

Fig. 1 is a schematic flowchart of a method for identifying activities of an unknown type based on channel state information in an embodiment of the present invention.

Fig. 2 is a schematic diagram of the network structure of the unknown activity recognition network in the embodiment of the present invention.

Fig. 3 is a schematic diagram of cross-domain activity recognition effects in a laboratory scenario in an embodiment of the present invention.

Fig. 4 is a schematic diagram of cross-domain activity recognition effects in a meeting room scene in an embodiment of the present invention.

Fig. 5 is a schematic diagram of the cross-domain activity recognition effect in the bedroom scene in the embodiment of the present invention.

Fig. 6 is a schematic diagram of the cross-domain activity recognition effect in the corridor scene in the embodiment of the present invention.

Fig. 7 is a schematic diagram of the influence of the data enhancement method in the embodiment of the present invention on the accuracy of activity recognition.

Detailed ways

The technical solution of the present invention will be further described in detail below in conjunction with the accompanying drawings.

As shown in Figure 1, the present invention provides a non-contact cross-domain human activity recognition method based on metric learning, the process of which is as follows:

Step 1: Using a computer equipped with an Intel 5300 network card and three antennas as the receiver of the experiment, and a computer equipped with an Intel 5300 network card and one antenna as the transmitter of the experiment, deploy the device in four different daily life scenarios ( laboratories, conference rooms, offices, bedrooms). The linear distance between the transmitter and receiver is 3 meters, and the height is 1 meter from the ground. The transceiver sends and receives data packets during volunteer activities and collects data related to personnel activities. Use Linux802.11 CSI tools to extract CSI raw data for human activity recognition from the collected activity data.

Step 2: CSI (Channel State Information) describes the characteristics of the signal propagation space. Different activities will cause different multipath propagation of the signal. Using this characteristic, Wi-Fi can be used for human activity recognition.

The wireless signal transmitted by the sending end is affected by the physical environment or human factors (position, posture, action) during the transmission process, forming multiple paths such as direct radiation, reflection, and scattering, resulting in multipath effects. When there is human activity, the signal at the receiving end changes. At this time, the signal received by the receiving end reflects the multi-path change caused by personnel activities. CSI contains amplitude and phase information on each subcarrier, characterizes the characteristics of signal spatial propagation, and can reflect signal changes. The signal received by the receiver can be described as:

in is the channel matrix, representing the CSI information; the received and transmitted signal vectors are respectively and ; is additive white Gaussian noise.

Since an RF signal propagates through multiple paths from transmitter to receiver in an indoor environment, CSI is the sum of all path signals. So the channel matrix can be expressed as:

here is the number of paths, is complex attenuation, is the first The propagation length of the path, is the wavelength. From this model, it can be obtained that when people are active in the room, the signal is reflected by the human body, which changes the length of the signal path, thereby affecting the channel state matrix.

Step 2-1: Data interpolation. Due to the influence of device hardware and channels, packet loss may occur during data transmission. Use cubic spline interpolation to complete the CSI data according to the timestamp, so that the length of the CSI data is uniform. Interpolation completion Matlab directly provides interpolation functions, which is a commonly used data processing method.

Step 2-2: Data denoising, for human daily activities, the amplitude of CSI fluctuates between 30 ~ 60 Hz. Under the premise of ensuring that the signal is not distorted, the noise caused by the hardware device is removed to the greatest extent. The present invention uses a Butterworth low-pass filter to denoise the CSI data in step 2-1, removes high-frequency noise in the environment, and utilizes The Hample filter removes outliers in the CSI data to obtain the preprocessed data.

Step 3: To generate more corresponding activity data from the collected data, a multi-autoencoder module is designed, where each submodule is dedicated to one activity type. The overall framework of the data enhancement module is shown in (a) in Figure 2. For the first activity type, using the Autoencoders to generate relevant activity data. Each of these autoencoders is an encoder-decoder structure. First, the data is compressed by an encoder, feature information related to its activity is extracted, and the input is encoded as a normal distribution . Hypothetically generate data The probability of ; The encoder is input as The probability of outputting latent vector z under the condition of . The extracted features and Gaussian random noise are then fed into the decoder to reconstruct the data, where the decoder is described as .

Finally, a loss function is introduced during the training phase. The goal is to train the encoder to output the probability of latent vector z approximates the probability of generating the latent vector z ,in conform to normal distribution of . Define the encoder loss as the Kullback–Leibler divergence, and Respectively represent the category The expectation and variance of the normal distribution of .

The output of the encoder is mixed with noise into the decoder to reconstruct the input ,in is the real value, and the minimum reconstruction loss is

Then, the overall loss function of the data augmentation module is defined as:

Through the data enhancement module, a large amount of synthetic data similar to but different from the training data can be generated to expand the training set.

Step 4: For unknown activity types, they cannot be directly mapped to known labels. Inspired by the idea of metric learning, by learning the similarity between two samples, the simple classification problem can be transformed into a comparison problem, that is, comparing the similarity between samples. Thus the present invention proposes an activity recognition module.

Step 4-1: Divide the dataset.

Based on step 3, the enhanced and expanded data set is obtained, and this data set is used as the training set. Afterwards, with Label the training set in the form of , where the positive samples with the benchmark sample have the same class labels, while the negative samples has a different activity category label. Therefore, the training dataset can be expressed as ,in corresponding to the first samples.

For activity data of unknown type, a set of data (each sample corresponding to a category) is randomly selected from the activity data set of unknown type to generate a support set. Describe the support set as ,in for a random sample the corresponding label, Corresponds to the number of activities of unknown type.

Step 4-2: Train the feature extractor to extract activity-related features.

The overall framework of the feature extraction module is shown in (b) in Figure 2. Specifically, the activity recognition method consists of three modules that use the same feed-forward network and share the same parameters. The activity recognition method has three inputs and corresponding two intermediate output values, which represent the L2 distance of the two inputs; in order to output a comparison operator from the model, the Softmax function is applied to the output to create a ratio degree; the feedforward network mainly It consists of six network layers, CNN→LSTM→CNN→LSTM→CNN→Linear; the CNN layer learns the deep feature signal of CSI, which is composed of BatchNorm→CNN→Maxpool; the LSTM layer learns the correlation of CSI signals in the time domain, specifically It is BatchNorm→LSTM→Dropout; the linear layer Linear is composed of Linear→Relu→Linear, and maps the extracted data to the feature space.

The purpose of training is to make the data distance between the same category as small as possible and the data distance between different categories as large as possible. So the loss is defined as:

in,

In the formula, the function E() refers to the feature extractor; the goal is to make the loss of the feature extraction network approach to 0, and define , express and The Euclidean distance between express and Euclidean distance between.

Three feedforward networks are updated simultaneously using the backpropagation algorithm. Finally, the training set is fed into a feature extractor, and a feature extractor is obtained through training to extract activity-related features in CSI data.

Step 4-3: Activity Recognition.

As shown in Figure 2, the input support set And the target sample data to the feature extraction network to obtain the corresponding activity features. The activity recognition process is shown in (c) in Figure 2. Measure the similarity between the eigenvalues of the target sample and all the eigenvalues of the samples in the support set, and select the label corresponding to the support set sample with the largest similarity as the label of the target sample .

in Refers to the label corresponding to the sample in the support set, /> Refers to the support set, i refers to the i-th data in the support set; support set samples and target samples/> Cosine similarity between /> for

Through the above steps, the identification of unknown types of activities is completed.

To evaluate the robustness of the proposed method in different environments, the proposed method is implemented in 4 indoor scenarios (laboratory, conference room, bedroom, and living room) with various complex wireless environments. Each scene uses two laptops (Think-pad X200) as transceivers, both devices are equipped with Intel 5300 cards, and Linux802.11n CSI Tool is installed for collecting CSI data. The number of antennas at the transmitting end is Nt=1, and the number of antennas at the receiving end is Nr=3. By collecting human activity signals, CSI data of 1 × 3 × 30 = 90 subcarriers can be obtained, where each transceiver antenna pair has 30 subcarriers. In this method, the signal frequency is 5.8 GHz, the bandwidth is 20 MHz, and the sampling frequency is 200 Hz.

In order to evaluate the adaptability of this method to different people, in each experimental scenario, 7 volunteers with different heights and weights were recruited to complete 8 activities (standing up, standing, sitting down, sitting down, bending over, marching, wave and turn). The data for each scene is split into a training set (contains four activity types), a test set (contains four unseen activity types), and a support set (contains only one sample of each activity).

Use the accuracy rate to evaluate the performance of the system.

The recognition effect of the present invention is as follows:

1. Overall recognition accuracy.

The proposed method is verified in four scenarios. According to the table below, it can be seen that compared with some existing networks, the recognition accuracy of this method has been significantly improved. Specifically, the activity recognition method of the present invention has significantly improved the accuracy of activity recognition compared with CNN and MatNet; in addition, the accuracy rate of the method proposed in the present invention in four life scenarios compared with the transfer learning method reached 9.31%, 9.31%, 5.25%, 10.22% and 21.33% improvements. This is mainly due to the method proposed in the present invention to mine the relationship between data, and identify by comparing the similarity between the data and the support set data, so as to distinguish samples more effectively.

Table 1

model Laboratory meeting room Bedroom Parlor CNN 26.15 23.48 23.79 21.74 MatNet 55.35 52.87 53.10 52.61 transfer 63.19 68.85 70.28 56.67 WIUS 72.50 74.10 80.50 78.00

2. The recognition accuracy of new categories of activities that did not appear in the training set.

By drawing the confusion matrix of activity classification, the effectiveness of the activity recognition method of the present invention in different categories of activities is described in detail. As shown in Figure 3-6, the accuracy rates of new category activity recognition reached 72.5%, 74.1%, 80.5% and 78% respectively. The experimental results show that the correct recognition probability of the unseen "stand up" action is higher than other actions. The reason is that when the person stands up, they first experience a rapid acceleration and then the velocity drops to zero. Furthermore, standing up is a continuous process of approaching the sending and receiving pair, which is well differentiated from other activities. Both "waving" and "turning around" have a process of moving away from the transceiver link to the longest distance and then approaching the transceiver pair, so they are easily misclassified into the other party's category.

3. The impact of data enhancement methods on the accuracy of activity recognition.

To verify the effectiveness of the data augmentation method proposed in this method, ablation analysis is performed with and without data augmentation for cross-domain activity recognition. As shown in Figure 7, after enhancing the data set, the recognition accuracy of unknown activities of this method has been greatly improved. In the four life scenarios, the average recognition accuracy of unknown activities of this method has increased by 10.95%. Especially in bedrooms and corridors, the expansion of the data set enables this method to have enough data to train the model, which improves the feature extraction ability of the model.

The above descriptions are only preferred embodiments of the present invention, and the scope of protection of the present invention is not limited to the above embodiments, but all equivalent modifications or changes made by those of ordinary skill in the art according to the disclosure of the present invention should be included within the scope of protection described in the claims.

Claims (4)

1. A non-contact cross-domain human activity recognition method based on metric learning, characterized in that: the method comprises the steps of: Step 1, collect the wireless signal data corresponding to people's activities in the indoor environment, and extract the original signal of CSI from it; Step 2, performing data preprocessing on the original CSI data collected in step 1, data interpolation with a uniform length, and data denoising to remove noise signals caused by hardware devices; Step 3, use the autoencoder to perform data enhancement on the data processed in step 2, and generate data related to corresponding activities to expand the data set; Step 4, use the method based on metric learning to establish an activity recognition model, train the data generated in step 3 and the original data, learn the similarity between the data, train the activity-related feature extraction model, and compare the data to be recognized with the support Collect the characteristics of the data to identify unknown types of activities; In step 4, the activity recognition module includes a feature extraction network and an activity classification module; Step 4-1: Divide the dataset; Based on step 3, the enhanced and expanded data set is obtained, and this data set is used as the training set; the training data set is expressed as train={(x ₁ , x ₁₊ , x _1- ), (x ₂ , x ₂₊ , x _2- ),...(x _i , x _i+ , x _i- )}, where the positive sample x ₊ has the same class label as the baseline sample x, and the negative sample x ₋ has a different active class label; From an active dataset of unknown type, randomly select a set of data to generate a support set; describe the support set as where /> is the data sample /> The corresponding label, k is the number of activities of unknown type; Step 4-2: Train the feature extractor to extract activity-related features; The feature extraction module consists of three modules, which use the same feed-forward network and share the same parameters; the feature extraction module has three inputs and corresponding two intermediate output values, which represent the L2 distance of the two inputs; in order to extract from the model A comparison operator is output, and the Softmax function is applied to the output to create a ratio; the feedforward network is mainly composed of six network layers, CNN→LSTM→CNN→LSTM→CNN→Linear; the CNN layer learns the deep feature signal of CSI, It consists of BatchNorm→CNN→Maxpool; the LSTM layer learns the correlation of CSI signals in the time domain, specifically BatchNorm→LSTM→Dropout; the linear layer Linear is composed of Linear→Relu→Linear, which maps the extracted data to the feature space ; The purpose of training is to make the data distance between the same category as small as possible, and the data distance between different categories as large as possible, so the loss is defined as: in, In the formula, the function E() refers to the feature extractor; the goal is to make the loss of the feature extraction network close to 0, define margin=1, d _- means the Euclidean distance between x _- and x, d ₊ means x ₊ and Euclidean distance between x; Step 4-3: activity recognition; Input the support set S and the target sample data to the feature extraction network to obtain the corresponding activity features; measure the similarity between the feature value of the target sample and all sample feature values in the support set, and select the label corresponding to the support set sample with the largest similarity as The label y _t of the target sample; in, refers to the label corresponding to the sample in the support set, S refers to the support set, and i refers to the i-th data in the support set; the cosine similarity d _k between the support set sample and the target sample x _t is: 2. A non-contact cross-domain human activity recognition method based on metric learning according to claim 1, characterized in that: in step 1, use Linux 802.11 CSI tools to extract the original signal of CSI therefrom. 3. A kind of non-contact cross-domain human activity recognition method based on metric learning according to claim 1, characterized in that: in step 2, the data preprocessing method is as follows: Step 2-1: Data interpolation, for packet loss that occurs during data transmission, use cubic spline interpolation to complete the CSI data according to the timestamp; Step 2-2: Data denoising, use the Butterworth low-pass filter to denoise the CSI data in step 2-1, remove high-frequency noise in the environment, and use the Hampel filter to remove outliers in the CSI data, Get the preprocessed data. 4. A kind of non-contact cross-domain human activity recognition method based on metric learning according to claim 1, characterized in that: in step 3, the specific method of data enhancement is as follows: Use an autoencoder to generate relevant activity data, where each autoencoder is an encoder-decoder structure; first, the data is compressed by the encoder, feature information related to its activity is extracted, and the input is encoded as a positive state distribution N(0, σ); assuming that the probability of generating data x is P(x); the encoder outputs the probability of latent vector z under the condition that the input is x is P(z|x); then the extracted features and Gaussian random noise is fed into the decoder to reconstruct the data; Finally, a loss function is introduced during the training phase; the goal is to train the encoder to output a latent vector z with probability P(z|x) that approximates the probability P(z) of generating latent vector z, where z conforms to the positive The encoder loss is defined as the Kullback–Leibler divergence, and μ _i and σ _i represent the expectation and variance of the normal distribution of category i, respectively; L _encoder = D _KL (N(μ _i , σ _i ), N(0, σ)); The output of the encoder is mixed with noise n into the decoder to reconstruct the input x, where is the real value, the minimum reconstruction loss is: Then, the overall loss function of the data augmentation module is defined as: L _aug = min(L _encoder , L _decoder ).