CN111160443B - A method for activity and user recognition based on deep multi-task learning - Google Patents

A method for activity and user recognition based on deep multi-task learning Download PDF

Info

Publication number
CN111160443B
CN111160443B CN201911355355.1A CN201911355355A CN111160443B CN 111160443 B CN111160443 B CN 111160443B CN 201911355355 A CN201911355355 A CN 201911355355A CN 111160443 B CN111160443 B CN 111160443B
Authority
CN
China
Prior art keywords
activity
user
network
parameters
recognition
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Expired - Fee Related
Application number
CN201911355355.1A
Other languages
Chinese (zh)
Other versions
CN111160443A (en
Inventor
陈岭
张毅
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Zhejiang University
Original Assignee
Zhejiang University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Zhejiang University filed Critical Zhejiang University
Priority to CN201911355355.1A priority Critical patent/CN111160443B/en
Publication of CN111160443A publication Critical patent/CN111160443A/en
Application granted granted Critical
Publication of CN111160443B publication Critical patent/CN111160443B/en
Expired - Fee Related legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G06F18/2415—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/044—Recurrent networks, e.g. Hopfield networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Artificial Intelligence (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Software Systems (AREA)
  • Mathematical Physics (AREA)
  • Computing Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Probability & Statistics with Applications (AREA)
  • Evolutionary Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Image Analysis (AREA)

Abstract

本发明公开了一种基于深度多任务学习的活动和用户识别方法，包括：(1)利用可穿戴传感器采集各用户活动时的传感器数据，并进行预处理；(2)联合构建用于预测活动和用户的活动识别网络和用户识别网络，其中，活动识别网络和用户识别网络共享部分隐参数，并引入相互注意力机制，利用对方学习到的知识为特征各部分赋予权重；(3)构建联合损失对活动识别网络和用户识别网络进行协同优化，得到参数确定的活动识别模型和用户识别模型；(4)将预处理后的传感器数据输入至活动识别模型和用户识别模型中，利用活动识别模型获得活动识别结果，利用用户识别模型获得用户识别结果。该方法提升了活动和用户的识别能力。

Figure 201911355355

The invention discloses an activity and user identification method based on deep multi-task learning, including: (1) using wearable sensors to collect sensor data of each user's activities, and performing preprocessing; (2) joint construction for predicting activities and the user's activity recognition network and user recognition network, wherein the activity recognition network and the user recognition network share some hidden parameters, and introduce a mutual attention mechanism, and use the knowledge learned by the other party to give weights to each part of the feature; (3) construct a joint Co-optimize the activity recognition network and user recognition network to obtain the activity recognition model and user recognition model with determined parameters; (4) input the preprocessed sensor data into the activity recognition model and user recognition model, and use the activity recognition model The activity recognition result is obtained, and the user recognition result is obtained by using the user recognition model. This approach improves the recognition of activities and users.

Figure 201911355355

Description

Activity and user identification method based on deep multitasking learning
Technical Field
The invention relates to the field of activity recognition and user recognition, in particular to an activity and user recognition method based on deep multitask learning.
Background
The method for deducing the activity and the user information based on the activity of the wearable sensor and the user identification by utilizing the data acquired by the sensor worn by the user is two key tasks in the fields of pervasive and mobile computing, and has important significance for realizing the computation with human centers. The wearable sensor-based activity and user identification can provide support for health support, skill assessment, biometric identification, and other applications.
Most wearable sensor-based activity recognition methods mix labeling samples of all training users, build an activity classifier using supervised learning methods, and use it directly for new users, ignoring differences in different user behavior patterns. Related studies have shown that different users have different patterns of behavior, which means that there is a distribution difference between the sensor data acquired from different users, and therefore, when an activity recognition model that performs well on a training user is directly applied to a new user, a significant performance degradation tends to occur. How to guarantee the recognition performance of new users becomes an important challenge for activity recognition based on wearable sensors. To cope with this challenge, several methods have been proposed at home and abroad, which can be roughly divided into two categories: the first type of method builds user-independent features for modeling, so that the generalization capability of the model on new users is guaranteed. Such methods have difficulty in fully utilizing the user information in the training data, and impair their activity characterization capabilities while enhancing feature user independence. The second class of methods builds a custom model for each user for its behavior pattern. Such methods often require data acquisition and model adaptation for each new user, which is limited in use at a high cost.
Most of the user identification methods based on the wearable sensors only support the identification of users in the walking activity scene, and although the methods can achieve good identification effect, the limited applicable scene limits the use of the methods in daily life. Extending the activity scenarios supported by wearable sensor-based user identification from walking to other daily activities is valuable, but also challenging, because there are significant differences in sensor data from activity scenario to activity scenario.
Existing approaches typically model activity recognition or user recognition tasks alone, ignoring the correlation between the two tasks. The activity related information acquired by the activity recognition task is helpful for the user to recognize task perception and adapt to different activity scenes; the user-related information learned by the user identification task enables the activity identification model to take into account the current user's behavior pattern when identifying human activity.
Disclosure of Invention
The invention aims to solve the technical problem of improving the generalization capability of an identification model by utilizing the correlation between two tasks of activity identification and user identification.
In order to solve the above problems, the present invention provides an activity and user identification method based on deep multitasking learning, comprising the following steps:
(1) Collecting sensor data of each user during activities by using a wearable sensor, and preprocessing;
(2) Jointly constructing an activity recognition network and a user recognition network for predicting activities and users, wherein the activity recognition network and the user recognition network share part hidden parameters and introduce a mutual attention mechanism, and weighting each part of the characteristics by utilizing knowledge learned by the other party;
(3) Constructing joint loss to perform collaborative optimization on the activity recognition network and the user recognition network to obtain an activity recognition model and a user recognition model with determined parameters;
(4) And inputting the preprocessed sensor data into the activity recognition model and the user recognition model, obtaining an activity recognition result by using the activity recognition model, and obtaining a user recognition result by using the user recognition model.
Compared with the prior method, the method has the advantages that:
1) And jointly constructing an activity recognition model and a user recognition model, and utilizing hidden parameters to carry out cross-task knowledge sharing so as to realize interaction promotion by utilizing the commonality and the difference between the two tasks.
2) A mutual attention mechanism is introduced between the activity recognition model and the user recognition model, so that the two models can mutually utilize knowledge learned by each other to give weight to each part of the feature, thereby adapting to the change of the user and the activity scene.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a general flow diagram of an activity and user identification method based on deep multitasking learning provided by an embodiment;
FIG. 2 is a general architecture diagram of an activity and user identification method based on deep multitasking learning provided by an embodiment;
FIG. 3 is a schematic diagram of a convolutional neural network architecture provided by an embodiment;
fig. 4 is a schematic diagram of a bidirectional long and short time memory network according to an embodiment.
Detailed Description
The present invention will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present invention more apparent. It should be understood that the detailed description is presented by way of example only and is not intended to limit the scope of the invention.
FIG. 1 is a general flow diagram of an activity and user identification method based on deep multitasking learning provided by an embodiment; fig. 2 is an overall architecture diagram of an activity and user recognition method based on deep multitasking learning provided in an embodiment, in fig. 2, conv1D and Pool1D represent a one-dimensional convolutional network layer and a one-dimensional max-pooling layer, respectively, U represents a bidirectional long-short-time memory network element, attNet represents an attention network, which outputs a weight vector w to distinguish the importance of a feature r,
Figure BDA0002335756710000041
representing Hadamard sum, ">
Figure BDA0002335756710000042
Representing the hadamard product.
Referring to fig. 1 and 2, the activity and user recognition method based on deep multitasking learning provided in the embodiment includes a data set construction stage, a hidden parameter initialization stage, and a model training stage.
Data set construction phase
The data set construction stage mainly collects sensor data, preprocesses the collected sensor data, and constructs a training data set, and the specific process is as follows:
step 1-1, acquiring sensor data of each user when different activities are performed by using a wearable sensor;
in this step, wearable sensor data is recorded as the user performs different activities. Common wearable sensors include accelerometers, gyroscopes, magnetometers, and the like. In order to ensure the recognition accuracy, the number, the type and the placement orientation of the sensors are required to be consistent when the training data set is constructed and when the training data set is actually used.
And step 1-2, carrying out outlier elimination and normalization processing on the acquired data, and dividing the processed data by utilizing a sliding window to obtain a data sample.
In this step, the processing of the acquired data mainly includes:
a) Abnormal value detection is carried out on the original sensor data, and invalid values (such as values exceeding a normal range and missing values) in the abnormal value detection are removed or mean filling is carried out.
b) Carrying out dispersion standardization processing on sensor data according to channels, so that the processed data are standardized to be within an [ -1,1] interval, and the conversion formula is as follows:
Figure BDA0002335756710000051
wherein X is the original value, X min To be the minimum value of the channel where the value is located, X max For the maximum value of the channel in which the value is located, x' is the value after normalization.
c) The processed data was divided using a fixed length sliding window, the sliding window length was set manually according to experience, and the overlap was set to 50%.
And step 1-3, taking each data sample, the corresponding active label and the user label as a training sample, and constructing a training data set. One training sample may be denoted (x, a, u), where x represents the data sample, a represents the active tag, and u represents the user tag.
Step 1-4, the training data set is batched according to a fixed size, and the total number of batches is N.
In this step, the batch size is manually set according to experience, and the total number of batches N is calculated as follows:
Figure BDA0002335756710000052
wherein M is total To train the total number of samples in the dataset, M batch Is of a batch size.
Hidden parameter initialization stage
The specific process is as follows:
step 2-1, sequentially selecting a batch of training samples from the training dataset, and repeating steps 2-2 to 2-5 for each training sample (x, a, u) in the batch.
Step 2-2, respectively processing the data sample x by using two independent convolutional neural networks in the active recognition network and the user recognition network to obtain an intermediate layer characteristic representation r of the sample a And r u 。
In this step, the two convolutional neural networks have the same network architecture and are respectively composed of three convolutional network layers and one max-pooling layer, as shown in fig. 3, wherein the convolutional operation and the max-pooling operation are performed along the time dimension. In fig. 3, conv1D and Pool1D represent a one-dimensional convolutional network layer and a one-dimensional max-pooling layer, respectively, the numbers before @ represent the size of the convolutional kernel or pooling region, the numbers after @ represent the number of generated feature maps, and ReLU represents the activation function.
Step 2-3, representing the intermediate layer characteristics as r a And r u The vector sequence with the length of l is expanded and is respectively input into two independent bidirectional long-short-time memory networks in the activity recognition network and the user recognition network to extract the time dependency relationship to obtain the characteristic representation vector sequence
Figure BDA0002335756710000061
And->
Figure BDA0002335756710000062
In this step, the intermediate layer features represent r a And r u The time dimension of (c) remains unchanged and the other dimensions are expanded. The two bidirectional long-short-time memory networks have the same network architecture, and each bidirectional long-short-time memory network layer comprises a bidirectional long-short-time memory network layer which is shown in a time expansion schematic diagram in fig. 4. In FIG. 4, L f And L b Respectively representing a forward long-short time memory network unit and a backward long-short time memory network unit, wherein the two long-short time memory network layers are opposite in directionThe short-time memory network layer is connected to the same output, one long-time memory network layer processes the input sequence in the forward direction, the other one processes in the backward direction, and the output can acquire the time dependence relationship between the forward direction and the backward direction at the same time.
Step 2-4, using two independent attention networks in the activity recognition network and the user recognition network to represent the feature vector sequence e a Mapping to weight vectors
Figure BDA0002335756710000063
Representing the vector sequence e for the feature u Each part is given weight; representing the feature by a vector sequence e u Mapping to weight vector->
Figure BDA0002335756710000064
Representing the vector sequence e for the feature a Each portion is given a weight.
In this step, the two attention networks have the same network architecture, and each includes a fully connected network layer, and the output of the fully connected network layer is normalized by a softmax function to obtain a weight vector, and the process is as shown in formulas (3) - (8):
ω a ＝AttNet a (e u ) (3)
Figure BDA0002335756710000065
Figure BDA0002335756710000066
ω u ＝AttNet u (e a ) (6)
Figure BDA0002335756710000067
Figure BDA0002335756710000071
wherein the method comprises the steps of，AttNet u (. Cndot.) and AttNet a (. Cndot.) represents the attention network, ω u And omega a Representing non-normalized weight vector, alpha a And alpha u Is a weighted sum of the feature representations.
Step 2-5, utilizing an activity classification output layer formed from fully-connected network layer and softmax function in the activity recognition network, according to the weighted sum alpha of characteristic representation a Obtaining an activity prediction result a'; using a user classification output layer in the user identification network, which consists of a fully connected network layer and a softmax function, a weighted sum alpha of feature representations is used u Obtaining a user prediction result u'.
In this step, the activity and user classification layer outputs probability distributions of different activity labels and different user labels, and the calculation formula is as follows:
Figure BDA0002335756710000072
Figure BDA0002335756710000073
wherein alpha is a And alpha u Representing the input of the last step, θ a And theta u Parameters representing the fully connected network layer, |a| represents the number of active labels, |u| represents the number of user labels, a ′ i Representing the probability of predicting an active tag as i, u ′ i Representing the probability of predicting the user tag as i.
And 2-6, calculating activity classification loss according to the activity prediction and the activity label for all training samples of the batch, calculating user classification loss according to the user prediction and the user label, and updating parameters of each part of the network according to the loss.
In this step, the loss is calculated using a cross entropy loss function, the calculation formula is as follows:
Figure BDA0002335756710000074
Figure BDA0002335756710000075
Figure BDA0002335756710000076
wherein L is a And L u All representing loss of a single sample, M batch Indicating the batch size, loss indicates the joint Loss of a batch of samples. Adam algorithm is used for network parameter update.
Step 2-7, if the specified iteration times are reached, entering step 2-8, otherwise returning to step 2-1.
And 2-8, stacking convolution kernels of the updated two convolution neural networks corresponding to the convolution network layers to form a high-order tensor, and decomposing by a Tucker to obtain an initial hidden parameter.
In this step, the convolution kernel of each convolution network layer may be represented as a d 1 ×d 2 ×d 3 The convolution kernels of the corresponding convolution network layers of the two convolution neural networks are stacked to form d by the dimension third-order tensor 1 ×d 2 ×d 3 ×d 4 (d 4 =2) a four-dimensional tensor T, which is subjected to a Tucker decomposition to obtain the initial hidden parameters, as shown in equation (14):
T＝S· (1,2) R (1) · (1,2) R (2) · (1,2) R (3) · (1,2) R (4) (14)
wherein S is a v 1 ×v 2 ×v 3 ×v 4 Dimension fourth-order tensor, R (i) Is d i ×v i The dimension matrix, S and R are initial hidden parameters, R (4) Hidden parameters, S, R, specific to the identification network of the activity and the identification network of the user (1) ，R (2) And R is (3) Hidden parameters representing the sharing of an active identification network and a user identification network, subscript · (i,j) Representing coordinate axes for performing tensor dot product operations。
Model training stage
Step 3-1, sequentially selecting a batch of training samples from the training dataset, and repeating steps 3-2 to 3-3 for each training sample (x, a, u) in the batch.
Step 3-2, respectively processing the data sample x by using two convolutional neural networks sharing part of hidden parameters in the active recognition network and the user recognition network to obtain an intermediate layer characteristic representation r of the sample a And r u . The hidden parameters are stored and updated instead of the convolution kernels, specifically, in forward propagation, the hidden parameters are firstly used for generating the convolution kernels, and then the generated convolution kernels are used for carrying out convolution operation; in back propagation, hidden parameters are updated.
In this step, a corresponding convolution kernel is generated from the hidden parameter by inverse operation of the Tucker decomposition, and a convolution operation is performed using the generated convolution kernel. Wherein, part of hidden parameters are shared by the activity recognition network and the user recognition network, and when one network is trained, the updating of the shared hidden parameters also affects the convolution kernel generated by the other network, thereby realizing the cross-task sharing of knowledge.
Step 3-3, obtaining an activity prediction result a 'and a user prediction result u' according to the steps 2-3-2-5; namely, specifically comprises:
step 3-3-1, r a And r u Expanded into a vector sequence with the length of l, and respectively input into two independent bidirectional long-short-time memory networks in the activity recognition network and the user recognition network to obtain a characteristic representation vector sequence
Figure BDA0002335756710000091
And
Figure BDA0002335756710000092
the implementation details of the step are consistent with those of the step 2-3 in the hidden parameter initialization stage.
Step 3-3-2, using two independent attention networks of the activity recognition network and the user recognition network, e a Mapping to weight vectors
Figure BDA0002335756710000093
E is u Each part is given weight; will e u Mapping to weight vector->
Figure BDA0002335756710000094
E is a Each portion is given a weight.
The implementation details of the step are consistent with those of the step 2-4 in the hidden parameter initialization stage.
Step 3-3-3, obtaining an activity prediction result a' by using an activity classification output layer which is formed by a fully connected network layer and a softmax function in an activity recognition network; and obtaining a user prediction result u' by using a user classification output layer in the user identification network, wherein the user classification output layer consists of a fully connected network layer and a softmax function.
The implementation details of the step are consistent with those of the step 2-5 in the hidden parameter initialization stage.
And 3-4, updating parameters of each part of the network according to the steps 2-6.
In this step, for all training samples of the batch, activity classification losses are calculated according to the activity predictions and the activity labels, user classification losses are calculated according to the user predictions and the user labels, and parameters of each part of the network are updated according to the losses.
The implementation details of the step are consistent with those of the step 2-6 in the hidden parameter initialization stage.
And 3-5, if the specified training iteration times are reached, finishing training, obtaining an active recognition model and a user recognition model with determined parameters, and otherwise, returning to the step 3-1.
After the network parameters of the activity recognition model and the user recognition model are determined through the data set construction stage, the hidden parameter initialization stage and the model training stage, the activity recognition model and the user recognition model can be utilized for carrying out activity and user recognition, and the specific process is as follows:
for the acquired sensor data, performing outlier elimination and normalization processing on the data according to the steps 1-2 in the data set construction stage, and dividing the processed data to obtain a data sample;
processing the data sample by using an activity recognition model determined by the network parameters to obtain an activity recognition result;
and processing the data sample by using the user identification model determined by the network parameters to obtain a user identification result.
The invention jointly builds the activity recognition and user recognition models, utilizes hidden parameters to share information between the two models, further utilizes the commonality and the difference between the two tasks to realize mutual promotion, introduces a mutual attention mechanism, enables the two models to mutually utilize the knowledge learned by the other party to give weight to each part of the feature, thereby adapting to the change of the user and the activity scene, and has wide application prospect in the fields of health support, skill evaluation, biological recognition and the like.
The foregoing detailed description of the preferred embodiments and advantages of the invention will be appreciated that the foregoing description is merely illustrative of the presently preferred embodiments of the invention, and that no changes, additions, substitutions and equivalents of those embodiments are intended to be included within the scope of the invention.

Claims (5)

1.一种基于深度多任务学习的活动和用户识别方法，包括以下步骤：1. A method for activity and user identification based on deep multi-task learning, comprising the following steps: (1)利用可穿戴传感器采集各用户活动时的传感器数据，并进行预处理得到数据样本x，将数据样本x和对应的活动标签a和用户标签u作为训练样本，并构建训练样本集；(1) Using wearable sensors to collect sensor data from each user during activity, and preprocessing to obtain data samples x, using data samples x and corresponding activity labels a and user labels u as training samples, and constructing a training sample set; (2)从训练样本集中选择部分训练样本，来联合构建用于预测活动和用户的活动识别网络和用户识别网络，其中，活动识别网络和用户识别网络共享部分隐参数，并引入相互注意力机制，利用对方学习到的知识为特征各部分赋予权重，包括：(2) Select some training samples from the training sample set to jointly construct an activity recognition network and a user recognition network for predicting activities and users. The activity recognition network and the user recognition network share some hidden parameters and introduce a mutual attention mechanism to use the knowledge learned by each other to assign weights to each part of the feature, including: (2-1)从训练样本集中选择部分训练样本；(2-1) Select some training samples from the training sample set; (2-2)利用活动识别网络和用户识别网络中两个独立的卷积神经网络分别处理数据样本x，得到样本的中间层特征表示ra和ru；(2-2) Use two independent convolutional neural networks in the activity recognition network and the user recognition network to process the data sample x respectively, and obtain the intermediate layer feature representations ra and ru of the sample; (2-3)将中间层特征表示ra和ru展开为长度为l的向量序列，并分别输入到活动识别网络和用户识别网络中两个独立的双向长短时记忆网络，以提取时间依赖关系得到特征表示向量序列
Figure FDA0004127316950000011
和
Figure FDA0004127316950000012
其中，中间层特征表示ra和ru的时间维保持不变，其它维展开，两个双向长短时记忆网络具有相同的网络架构，均包含一个双向长短时记忆网络层，双向长短时记忆网络层将方向相反的两个长短时记忆网络层连接到同一输出，一个长短时记忆网络层前向处理输入序列，另一个后向处理，输出可以同时获取前向和后向的时间依赖关系；(2-3) Expand the intermediate layer feature representations ra and ru into a vector sequence of length l and input them into two independent bidirectional long short-term memory networks in the activity recognition network and the user recognition network respectively to extract the temporal dependency and obtain the feature representation vector sequence
Figure FDA0004127316950000011
and
Figure FDA0004127316950000012
Among them, the time dimension of the intermediate layer feature representations r a and r u remains unchanged, and the other dimensions are expanded. The two bidirectional long short-term memory networks have the same network architecture, both of which contain a bidirectional long short-term memory network layer. The bidirectional long short-term memory network layer connects two long short-term memory network layers in opposite directions to the same output. One long short-term memory network layer processes the input sequence forward, and the other processes it backward. The output can simultaneously obtain the forward and backward time dependencies. (2-4)利用活动识别网络和用户识别网络中两个独立的注意力网络，将特征表示向量序列ea映射到权重向量
Figure FDA0004127316950000013
为特征表示向量序列eu各部分赋予权重；将特征表示向量序列eu映射到权重向量
Figure FDA0004127316950000014
为特征表示向量序列ea各部分赋予权重；(2-4) Using two independent attention networks in the activity recognition network and the user recognition network, the feature representation vector sequence e a is mapped to the weight vector
Figure FDA0004127316950000013
Assign weights to each part of the feature representation vector sequence e u ; map the feature representation vector sequence e u to the weight vector
Figure FDA0004127316950000014
Assign weights to each part of the feature representation vector sequence e a ; (2-5)利用活动识别网络中的由全连接网络层和softmax函数构成的活动分类输出层，根据特征表示的加权和αa得到活动预测结果a′；利用用户识别网络中的由全连接网络层和softmax函数构成的用户分类输出层，根据特征表示的加权和αu得到用户预测结果u′；(2-5) Using the activity classification output layer composed of a fully connected network layer and a softmax function in the activity recognition network, the activity prediction result a′ is obtained according to the weighted sum α a of the feature representation; using the user classification output layer composed of a fully connected network layer and a softmax function in the user recognition network, the user prediction result u′ is obtained according to the weighted sum α u of the feature representation; (2-6)根据活动预测和活动标签计算活动分类损失，根据用户预测和用户标签计算用户分类损失，并依据损失更新网络各部分的参数；(2-6) Calculate the activity classification loss based on the activity prediction and activity label, calculate the user classification loss based on the user prediction and user label, and update the parameters of each part of the network based on the loss; (2-7)将被更新过的两个卷积神经网络对应卷积网络层的卷积核堆叠，构成高阶张量，并通过Tucker分解得到初始的隐参数，包括：(2-7) The convolution kernels of the corresponding convolutional network layers of the two updated convolutional neural networks are stacked to form a high-order tensor, and the initial latent parameters are obtained through Tucker decomposition, including: 每个卷积网络层的卷积核可以表示为一个d1×d2×d3维三阶张量，将两个卷积神经网络对应卷积网络层的卷积核堆叠构成一个d1×d2×d3×d4维四阶张量T，其中，d4＝2，对张量T进行Tucker分解从而得到初始的隐参数，如公式(1)所示：The convolution kernel of each convolutional network layer can be expressed as a d 1 × d 2 × d 3- dimensional third-order tensor. The convolution kernels of the corresponding convolutional network layers of two convolutional neural networks are stacked to form a d 1 × d 2 × d 3 × d 4 -dimensional fourth-order tensor T, where d 4 = 2. Tucker decomposition is performed on the tensor T to obtain the initial hidden parameters, as shown in formula (1): T＝S·(1，2)R(1)·(1，2)R(2)·(1，2)R(3)·(1，2)R(4) (1)T＝S · (1, 2) R (1) · (1, 2 ) R ( 2) · (1, 2) R (3) · (1, 2) R (4) (1) 其中，S是一个v1×v2×v3×v4维四阶张量，R(i)是一个di×vi维矩阵，S和R均为初始的隐参数，R(4)表示活动识别网络和用户识别网络特有的隐参数，S，R(1)，R(2)，和R(3)表示活动识别网络和用户识别网络共享的隐参数，下标·(i，j)表示执行张量点积操作的坐标轴；Where S is a v 1 ×v 2 ×v 3 ×v 4 -dimensional fourth-order tensor, R (i) is a d i × vi dimensional matrix, S and R are both initial latent parameters, R (4) represents the latent parameters unique to the activity recognition network and the user recognition network, S, R (1) , R (2) , and R (3) represent the latent parameters shared by the activity recognition network and the user recognition network, and the subscript · (i, j) represents the coordinate axis on which the tensor dot product operation is performed; (3)从训练样本集中选择部分训练样本，来构建联合损失对活动识别网络和用户识别网络进行协同优化，得到参数确定的活动识别模型和用户识别模型；(3) Selecting some training samples from the training sample set to construct a joint loss to collaboratively optimize the activity recognition network and the user recognition network, and obtaining an activity recognition model and a user recognition model with determined parameters; (4)将预处理后的传感器数据输入至活动识别模型和用户识别模型中，利用活动识别模型获得活动识别结果，利用用户识别模型获得用户识别结果。(4) The preprocessed sensor data is input into the activity recognition model and the user recognition model, and the activity recognition result is obtained by using the activity recognition model, and the user recognition result is obtained by using the user recognition model. 2.如权利要求1所述的基于深度多任务学习的活动和用户识别方法，其特征在于，步骤(2-4)中，两个注意力网络具有相同的网络架构，均包含一个全连接网络层，其输出经过softmax函数归一化得到权重向量，过程如公式(2)～(7)所示：2. The activity and user identification method based on deep multi-task learning as claimed in claim 1 is characterized in that in step (2-4), the two attention networks have the same network architecture, both of which include a fully connected network layer, and their outputs are normalized by a softmax function to obtain a weight vector, and the process is shown in formulas (2) to (7): ωa＝AttNeta(eu) (2)ω a =AttNet a (e u ) (2)
Figure FDA0004127316950000031
Figure FDA0004127316950000031
Figure FDA0004127316950000032
Figure FDA0004127316950000032
 ωu＝AttNetu(ea) (5)ω u =AttNet u (e a ) (5)
Figure FDA0004127316950000033
Figure FDA0004127316950000033
Figure FDA0004127316950000034
Figure FDA0004127316950000034
 其中，AttNetu(·)和AttNeta(·)表示注意力网络，ωu和ωa表示未归一化的权重向量，αa和αu为特征表示的加权和。Among them, AttNet u (·) and AttNet a (·) represent attention networks, ω u and ω a represent unnormalized weight vectors, and α a and α u are the weighted sums of feature representations. 3.如权利要求1所述的基于深度多任务学习的活动和用户识别方法，其特征在于，步骤(2-5)中，活动和用户分类层输出不同活动标签和不同用户标签的概率分布，计算公式如下：3. The activity and user identification method based on deep multi-task learning as claimed in claim 1, characterized in that in step (2-5), the activity and user classification layer outputs the probability distribution of different activity labels and different user labels, and the calculation formula is as follows:
Figure FDA0004127316950000035
Figure FDA0004127316950000035
Figure FDA0004127316950000036
Figure FDA0004127316950000036
 其中，αa和αu表示上一步的输入，θa和θu表示全连接网络层的参数，|A|表示活动标签数量，|U|表示用户标签数量，a′i表示预测活动标签为i的概率，u′i表示预测用户标签为i的概率。Among them, αa and αu represent the input of the previous step, θa and θu represent the parameters of the fully connected network layer, |A| represents the number of activity tags, |U| represents the number of user tags, a′i represents the probability of predicting the activity tag as i, and u′i represents the probability of predicting the user tag as i. 4.如权利要求1所述的基于深度多任务学习的活动和用户识别方法，其特征在于，步骤(3)具体包括：4. The method for activity and user identification based on deep multi-task learning according to claim 1, wherein step (3) specifically comprises: (3-1)从训练样本集中选择部分训练样本；(3-1) Select some training samples from the training sample set; (3-2)利用共享部分隐参数的两个卷积神经网络分别处理数据样本x，得到该样本的中间层特征表示ra和ru，隐参数代替卷积核被保存和更新，具体地，前向传播中，首先由隐参数生成卷积核，再由生成的卷积核进行卷积操作；反向传播中，隐参数被更新；(3-2) Two convolutional neural networks that share some hidden parameters are used to process the data sample x respectively, and the intermediate layer feature representations ra and ru of the sample are obtained. The hidden parameters are saved and updated instead of the convolution kernel. Specifically, in the forward propagation, the convolution kernel is first generated by the hidden parameters, and then the convolution operation is performed by the generated convolution kernel; in the backward propagation, the hidden parameters are updated; (3-3)按照步骤(2-3)～(2-5)的步骤获得活动预测结果a′和用户预测结果u′；(3-3) Obtaining the activity prediction result a′ and the user prediction result u′ according to steps (2-3) to (2-5); (3-4)按照步骤(2-6)更新网络各部分的参数以确定网络参数，获得参数确定的活动识别模型和用户识别模型。(3-4) According to step (2-6), update the parameters of each part of the network to determine the network parameters, and obtain the activity recognition model and user recognition model determined by the parameters. 5.如权利要求1所述的基于深度多任务学习的活动和用户识别方法，其特征在于，步骤(2-6)中，使用交叉熵损失函数计算损失，计算公式如下：5. The method for activity and user identification based on deep multi-task learning according to claim 1, characterized in that in step (2-6), the loss is calculated using a cross entropy loss function, and the calculation formula is as follows:
Figure FDA0004127316950000041
Figure FDA0004127316950000041
Figure FDA0004127316950000042
Figure FDA0004127316950000042
Figure FDA0004127316950000043
Figure FDA0004127316950000043
 其中，La和Lu均表示单个样本的损失，Mbatch表示批大小，Loss表示一批样本的联合损失，Adam算法被用于网络参数更新，|A|表示活动标签数量，|U|表示用户标签数量，a′i表示预测活动标签为i的概率，u′i表示预测用户标签为i的概率。Among them, La and Lu both represent the loss of a single sample, Mbatch represents the batch size, Loss represents the joint loss of a batch of samples, the Adam algorithm is used to update the network parameters, |A| represents the number of activity labels, |U| represents the number of user labels, a′i represents the probability of predicting the activity label as i, and u′i represents the probability of predicting the user label as i.
CN201911355355.1A 2019-12-25 2019-12-25 A method for activity and user recognition based on deep multi-task learning Expired - Fee Related CN111160443B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201911355355.1A CN111160443B (en) 2019-12-25 2019-12-25 A method for activity and user recognition based on deep multi-task learning

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201911355355.1A CN111160443B (en) 2019-12-25 2019-12-25 A method for activity and user recognition based on deep multi-task learning

Publications (2)

Publication Number Publication Date
CN111160443A CN111160443A (en) 2020-05-15
CN111160443B true CN111160443B (en) 2023-05-23

Family

ID=70558071

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201911355355.1A Expired - Fee Related CN111160443B (en) 2019-12-25 2019-12-25 A method for activity and user recognition based on deep multi-task learning

Country Status (1)

Country Link
CN (1) CN111160443B (en)

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111709243B (en) * 2020-06-19 2023-07-07 南京优慧信安科技有限公司 Knowledge extraction method and device based on deep learning
CN113472484B (en) * 2021-06-29 2022-08-05 哈尔滨工业大学 Internet of things equipment user feature code identification method based on cross entropy iterative learning
CN114462557B (en) * 2022-04-13 2022-07-01 北京大学 Physiological state identification and analysis method based on multi-source information fusion
CN116304625A (en) * 2023-01-06 2023-06-23 湖南大学 A method for electric stealing user identification and electric stealing time period detection
CN121996934B (en) * 2026-04-09 2026-07-24 浙江大学 Cross-user wearable activity recognition method based on group-specific concept-aware representation learning

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105404877A (en) * 2015-12-08 2016-03-16 商汤集团有限公司 Face attribute prediction method and device based on deep learning and multi-task learning
CN108846384A (en) * 2018-07-09 2018-11-20 北京邮电大学 Merge the multitask coordinated recognition methods and system of video-aware
CN108875836A (en) * 2018-06-27 2018-11-23 浙江大学 A kind of simple-complicated activity collaboration recognition methods based on depth multi-task learning
AU2018101751A4 (en) * 2018-11-17 2018-12-20 Feng Chen A distributed attack detection and information security sharing algorithm over multitask networks
CN109857843A (en) * 2018-12-25 2019-06-07 科大讯飞股份有限公司 Document-based interactive method and system

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105404877A (en) * 2015-12-08 2016-03-16 商汤集团有限公司 Face attribute prediction method and device based on deep learning and multi-task learning
CN108875836A (en) * 2018-06-27 2018-11-23 浙江大学 A kind of simple-complicated activity collaboration recognition methods based on depth multi-task learning
CN108846384A (en) * 2018-07-09 2018-11-20 北京邮电大学 Merge the multitask coordinated recognition methods and system of video-aware
AU2018101751A4 (en) * 2018-11-17 2018-12-20 Feng Chen A distributed attack detection and information security sharing algorithm over multitask networks
CN109857843A (en) * 2018-12-25 2019-06-07 科大讯飞股份有限公司 Document-based interactive method and system

Also Published As

Publication number Publication date
CN111160443A (en) 2020-05-15

Similar Documents

Publication Publication Date Title
Sansano et al. A study of deep neural networks for human activity recognition
Chen et al. Smartphone sensor-based human activity recognition using feature fusion and maximum full a posteriori
CN111695415B (en) Image recognition method and related equipment
Shanthamallu et al. A brief survey of machine learning methods and their sensor and IoT applications
CN110309861B (en) A Multimodal Human Activity Recognition Method Based on Generative Adversarial Networks
Ahishali et al. Advance warning methodologies for covid-19 using chest x-ray images
US8935195B2 (en) Method of identification and devices thereof
CN111160443A (en) An activity and user recognition method based on deep multi-task learning
CN111539941A (en) Parkinson's disease leg flexibility task evaluation method and system, storage medium and terminal
CN117591967A (en) A multi-modal emotion recognition method and system based on confidence fusion
CN111160462A (en) Unsupervised personalized human activity recognition method based on multi-sensor data alignment
Khatun et al. Human activity recognition using smartphone sensor based on selective classifiers
WO2019208793A1 (en) Movement state recognition model learning device, movement state recognition device, method, and program
CN112560712A (en) Behavior identification method, device and medium based on time-enhanced graph convolutional network
CN118986286A (en) Sleep stage method and device based on graph structure, electronic equipment and storage medium
Mahapatra et al. MRMR-SSA: a hybrid approach for optimal feature selection
Nia et al. The power of ANN-random forest algorithm in human activities recognition using IMU data
Wei et al. SemiHAR: Improving semisupervised human activity recognition via multitask learning
Parvathaneni et al. Human activity recognition with CNN in FastAI
CN115942243B (en) Environment self-adaptive positioning method, device and readable medium based on improved TCA algorithm
CN113112005A (en) Domain self-adaption method based on attention mechanism
Das et al. Utilizing deep convolutional neural networks and non-negative matrix factorization for multi-modal image fusion
CN120747650A (en) Multi-mode brain network classification method based on feature decoupling and dynamic graph construction
CN112183299A (en) Pedestrian attribute prediction method and device, electronic equipment and storage medium
CN118736410A (en) A change detection method for multi-temporal hyperspectral images based on cross-layer feature fusion

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant
CF01 Termination of patent right due to non-payment of annual fee
CF01 Termination of patent right due to non-payment of annual fee

Granted publication date: 20230523