CN115526249A - Time series early classification method, terminal device and storage medium - Google Patents

Abstract

The invention discloses a time sequence early classification method, terminal equipment and a storage medium, wherein a training set is constructed by utilizing time sequence data of human body actions; training the neural network by using a training set; inputting the training set into a trained neural network to obtain the classification probability of all data of the training set at all times, and calculating a probability exit threshold value by using the classification probability; inputting the observable data at the moment t into a trained neural network to obtain the classification probability Pt at the moment t, taking the maximum value of Pt, stopping inputting the observable data if the maximum value is greater than the probability exit threshold, and taking the classification result at the moment t as the final classification result of the observable data at the moment t. The invention can be self-adaptive to new data which is continuously added, and more distinctive features are extracted, so that the accuracy of early classification of time data is improved; the method can be self-adaptive to sample content and difficulty, and extracts more specific class characteristics so as to improve the accuracy of early classification.

Description

Time series early classification method, terminal equipment and storage medium

technical field

The invention relates to the technical field of time series data classification, in particular to a time series early classification method, a terminal device and a storage medium.

Background technique

In recent years, with the development of smart wearable devices, time-series data of human actions can be obtained everywhere for personal health monitoring, smart home control, etc., time-series classification tasks have attracted extensive attention. However, for some specific time-sensitive applications such as falling of the elderly, it is desirable to classify the time series as soon as possible. In addition, early classification of human activities helps to minimize the response time of the system, thereby improving user experience. Therefore, it is of great research significance to classify time series data as quickly and accurately as possible.

In the early classification, data was input continuously over time, and the length of the data was constantly changing, and its characteristics at different moments were quite different, so it was difficult for the classifier to classify time series of arbitrary length.

Traditional early time series classification methods can be divided into prefix-based methods, shapets-based methods, and posterior probability-based methods. However, these methods usually take a lot of time to train multiple classifiers for time series data of different lengths, and require a lot of expert experience to design manual features or set exit thresholds. Compared with traditional methods, in this era of big data, methods based on deep learning can automatically extract more effective features.

The current deep learning-based early time series classification methods can be divided into one-stage methods and two-stage methods. The two-stage method usually uses the training set to train the classification model in the first stage, and then formulates certain exit rules or sets a fixed exit threshold in the second stage, and obtains the classification results in advance when the classification probability meets the exit conditions. The one-stage method usually establishes a classification subnetwork and an exit subnetwork at the same time, and trains them. The classification subnetwork obtains classification results, and the exit subnetwork is used to indicate whether to exit at this moment.

Since the input data for early classification is constantly changing, deep learning-based methods usually use recurrent neural networks to adapt to data of ever-increasing length. Due to the forgetting defect of its recursive structure, the recurrent neural network cannot classify long time series well, and its local feature extraction ability is poor. Some methods combine convolutional neural networks to extract local features. Unfortunately, the parameters of conventional convolution kernels are fixed, and they have fixed and the same feature matching templates for any time and any sample, and do not fully consider intra-class differences and inter-class differences. difference.

Contents of the invention

The technical problem to be solved by the present invention is to provide a time series early classification method, terminal equipment and storage medium in view of the deficiencies in the prior art, fully consider intra-class differences and inter-class differences, and improve the accuracy of human action time data classification.

In order to solve the above technical problems, the technical solution adopted in the present invention is: a time series early classification method, comprising the following steps:

S1. Construct a training set using time series data of human actions;

S2. Using the training set to train the neural network;

S3. Input the training set into the trained neural network, obtain the classification probabilities of all data in the training set at all moments, and use the classification probabilities to calculate the probability exit threshold;

S4. Input the observable data at time t into the trained neural network, obtain the classification probability Pt at time t, and take the maximum value of Pt. If the maximum value is greater than the probability exit threshold, stop inputting observable data. The classification result at time t is taken as the final classification result; otherwise, the value of t is increased by 1, and step S4 is repeated until the exit condition is satisfied.

After obtaining the classification probability, the present invention uses the classification probability to calculate the probability exit threshold, and further determines whether to exit the training according to the magnitude relationship between the probability exit threshold and the classification probability. The threshold of the present invention is not a fixed threshold, but calculated according to the classification probability, so it can adapt to the changing characteristics of the input data of the early classification. In the present invention, if the exit condition is not met, then as time goes on, the data will continue to be input into the classifier (trained neural network) for classification until the exit condition is met, fully considering the intra-class difference and the inter-class difference, greatly Improved accuracy for early time series classification.

In the present invention, the neural network includes:

A plurality of cascaded first convolution modules are used to perform feature extraction on input data to obtain first features;

A plurality of cascaded second convolution modules input the first features for extracting high-level features of the input data;

The average pooling layer, the input is the high-level feature, and the output is the fusion feature corresponding to the sequence of different time lengths;

A linear layer, the input is the fusion feature, and the output is the prediction score of different categories;

The exponential normalization layer is used to normalize the prediction score and output classification probability.

In the present invention, the input data is input to the convolution block (the first convolution module) to extract the underlying features, and then the underlying features are input to the dynamic convolution block (the second convolution module) to extract time-adaptive high-level features, Then the high-level features are input into the average pooling layer to obtain the fusion features corresponding to sequences of different time lengths. These fusion features are passed through the linear layer to obtain the prediction scores of different categories, and finally the scores are passed through the exponential normalization layer to obtain the output probability.

In the present invention, the first convolution module is a bottom-level convolution module for extracting bottom-level features (first features or primary features), and the second convolution module is a dynamic convolution module for extracting high-level features.

In the present invention, cascading refers to sequential connection, for example, the output of the first first convolution module is connected to the input of the second first convolution module, and the output of the second first convolution module is connected to the third first convolution module. The input connection of a convolutional module, and so on.

In the present invention, the first convolution module includes a sequentially connected first dilated causal convolution layer, a first normalization layer, a second dilated causal convolution layer, a second normalization layer and a first linear activation unit ; The input feature of the first convolution module is connected with the output feature residual.

In the present invention, the second convolution module includes a dynamic convolution layer, a third normalization layer, and a second linear activation unit; wherein, the dynamic convolution layer includes a convolution kernel generation module, and the convolution kernel The generating module is used to generate a convolution kernel corresponding to each time by using the input data; and perform feature extraction on the input data by using the convolution kernel.

The convolution parameters of the conventional convolution block are fixed and have nothing to do with the content and length of the sample, which is not conducive to the extraction of the features of the early classification flow data. Therefore, after the convolution block extracts the initial low-level features, the present invention designs The dynamic convolution module enables the features extracted by the dynamic convolution module to adapt to the data content and length changes, further improving the classification accuracy.

In the present invention, the convolution kernel generation module includes a first convolution layer, a modified linear unit, a batch normalization layer and a second convolution layer connected in sequence.

For the same sample, the conventional convolution kernel uses the same convolution kernel for the whole time period, and the features are not distinguishable. As the amount of data increases, the information gain is limited. In comparison, the time-adaptive convolution module designed by the present invention Generating convolution kernels specific to increasing data over time enables extraction of more discriminative features.

The implementation process of using the classification probability to calculate the probability exit threshold includes:

Sorting the classification probabilities and removing duplicates in the classification probabilities;

Take the median of the adjacent classification probabilities to obtain a series of threshold candidate values;

Select the threshold candidate value with the smallest cost as the probability exit threshold.

Directly specifying the threshold of the data set requires a lot of expert experience, and the threshold is not suitable for all data sets, and the generalization performance is poor. And the present invention adopts the cost formula method, can automatically obtain the threshold value that is applicable to different data sets, and can obtain the threshold value (adjusting the value of α according to actual demand according to the demand adjustment formula, for example: when the classification accuracy rate has higher When there is a higher demand, the value of α needs to be increased, and when there is a higher demand for early exit, the value of α needs to be decreased). The method of calculating the threshold using the cost formula also has the advantage of being interpretable.

Use the following formula to calculate the cost of the threshold candidate value: Cost _β = α*(1-Acc _β )+(1-α) Earliness _β ; where Acc _β is the maximum value of the classification probability, and exit the calculation when it is greater than the threshold candidate value β Earliness _β is the earlyness calculated when the maximum value of the classification probability is greater than the threshold candidate value β, and α is the weight coefficient.

In the present invention, the value of α is 0.8 through a large number of experimental studies and analysis.

As an inventive concept, the present invention also provides a terminal device, including a memory, a processor, and a computer program stored on the memory; the processor executes the computer program to implement the steps of the above method of the present invention.

A computer-readable storage medium, on which computer programs/instructions are stored; when the computer programs/instructions are executed by a processor, the steps of the above method of the present invention are implemented.

Compared with prior art, the beneficial effect that the present invention has is:

1. For the data whose length is constantly increasing, the present invention can adapt to the continuously increasing new data and extract more distinguishing features to improve the accuracy of early classification of time data;

2. For different types of data, the present invention can adapt to the sample content and difficulty, and extract more class-specific features to improve the accuracy of early classification.

Description of drawings

Fig. 1 is the method flow chart of embodiment 1 of the present invention;

Fig. 2 is a neural network structural diagram of Embodiment 1 of the present invention;

FIG. 3 is a structural diagram of a dynamic convolution module in Embodiment 1 of the present invention.

detailed description

In order to make the purpose, technical solutions and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments It is a part of embodiments of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

In this document, the terms "first", "second" and other similar words are not intended to imply any order, quantity and importance, but are only used to distinguish different elements. In this document, the terms "a", "an" and other similar words are not intended to mean that there is only one of said things, but that the description is for only one of said things, which may have one or more indivual. In this document, the terms "comprising", "comprising" and other similar words are intended to indicate logical interrelationships, and cannot be regarded as denoting spatial structural relationships. For example, "A includes B" is intended to mean that B logically belongs to A, but not that B is spatially inside A. Additionally, the meanings of the terms "comprising", "comprising" and other similar words are to be regarded as open rather than closed. For example, "A includes B" means that B belongs to A, but B does not necessarily constitute the whole of A, and A may also include C, D, E and other elements.

Example 1

训练训练模型的所有参数θ，其中N为训练集中训练集的样本个数，T为完整的时间序列数据的长度。然后，将训练集输入训练好的DTCN，得到训练集所有数据的所有时刻的分类概率，根据本发明制定的退出规则，计算出该数据集下的概率退出阈值。具体的通过退出规则计算阈值的过程如下：首先将这些分类概率进行排序，并去掉其中的重复项得到{P₁,P₂,…,P_T}，然后将相邻的分类概率取中值，如β₁＝(P₁+P₂)/2，，得到一系列的阈值候选值{β₁，β₂，…，β_T}。对每个候选阈值β，本发明定义了成本公式：As shown in Figure 1, in this embodiment, in the training phase, firstly, the network (DTCN, the specific structure of the network is in the next subsection) designed by the present invention is trained, specifically, all time series data of the training set are input into this In the network designed by the invention, the loss function is used

Train all parameters θ of the training model, where N is the number of samples in the training set in the training set, and T is the length of the complete time series data. Then, the training set is input into the trained DTCN to obtain the classification probabilities of all data in the training set at all moments, and the probability exit threshold under the data set is calculated according to the exit rule formulated by the present invention. The specific process of calculating the threshold value through the exit rule is as follows: first sort these classification probabilities, and remove the duplicate items among them to obtain {P ₁ ,P ₂ ,…,P _T }, and then take the median of the adjacent classification probabilities, For example, β ₁ =(P ₁ +P ₂ )/2, to obtain a series of threshold candidate values {β ₁ , β ₂ , . . . , β _T }. For each candidate threshold β, the present invention defines a cost formula:

Cost _β =α*(1-Acc _β )+(1-α)*Earliness _β .

Among them, Acc refers to the accuracy rate that the prediction probability P of the training set exits the calculation when it is greater than the threshold, and Earlyness refers to the earlyness of the prediction probability P of the training set that exits the calculation when it is greater than the threshold. After calculating the cost of all candidate thresholds, select the threshold with the smallest cost as the final exit threshold of the data. In this embodiment, the value of α is selected to be 0.8.

In the test phase, taking sample A as an example, the observable data at time t (observable data refers to data of length t that can be obtained through equipment at time t, and data of length t+1 can be obtained at time t+1. The data. The observable data at different times is slowly increasing over time) and directly input into the trained network model to obtain the classification probability Pt at this time. Take the maximum value of Pt, if it is greater than the exit threshold calculated in the training phase, it is considered that the classification result at this time is reliable and exit, and the classification result at this moment is taken as the final classification result of the sample A. Otherwise, the data continues to be fed into the classifier for classification over time until the exit criteria are met.

The architecture of the DTCN proposed in this embodiment is shown in FIG. 2 , which is mainly composed of a convolution block and a dynamic convolution block. Specifically, the input data is input to the convolution block to extract the underlying features, and then the underlying features are input to the dynamic convolution block to extract time-adaptive high-level features, and then the high-level features are input to the average pooling layer to obtain sequences corresponding to different time lengths. These fusion features are passed through the linear layer to obtain the prediction scores of different categories, and finally the scores are passed through the exponential normalization layer to obtain the output probability.

The underlying convolutional block (the first convolutional module) is a convolutional module for extracting the underlying primary features (the first feature), which includes dilated causal convolution, normalization layer, and rectified linear unit (ReLU) . The reason for introducing causal convolution is to avoid information leakage, so that the characteristics at this moment have nothing to do with the characteristics after this moment; the expansion convolution effectively expands the receptive field; the normalization layer avoids overfitting during the training process; ReLU Enhanced nonlinearity of extracted features. The convolution parameters of the conventional convolution block are fixed and have nothing to do with the content and length of the sample, which is not conducive to the extraction of the features of the early classification flow data. Therefore, after the convolution block extracts the initial low-level features, the present invention designs The dynamic convolution module enables the features extracted by the dynamic convolution module to adapt to data content and length changes.

The design of the dynamic convolution module in this embodiment is specifically shown in FIG. 3 . Pass the input features through the convolution kernel generation module to generate each time-specific convolution kernel. During training, the convolution kernel generation module learns how to generate convolution kernels that are adaptive to the content of the data. The kernel generation module consists of a convolution of size 1, a batch normalization layer, and a ReLU. The convolution kernel generation module produces a time-adaptive convolution kernel of size K and channel sharing. The input features are convoluted using the newly generated time-adaptive convolution kernel to obtain the output features. For samples of different categories, through the design of the dynamic convolution generation module, a convolution kernel with adaptive content is generated, which can extract class-specific features; for the same sample, the conventional convolution kernel uses the same convolution kernel for the whole time Kernel, the features are not discriminative, and the information gain is limited as the amount of data increases. In comparison, the designed time-adaptive convolution module generates a convolution kernel specific to the increased data over time, which can extract more distinguishing features.

The time-adaptive features output by the dynamic convolution module are fused by the average pooling layer for each time period to obtain the fusion features of different time periods. Finally, the fusion features at different times are obtained through the linear layer and the normalized index layer. Classification probability at each moment.

Experiments were carried out on two commonly used human action recognition data sets, and ideal results were obtained (the index HM used in the following table is calculated by early and accurate, and is a comprehensive index. The embodiment of the present invention The method is DETSCN).

Table 1 Comparison of classification results

In Table 1 above, ECLN, ETMD, and EARLIEST methods refer to:

ECLN: Ruβwurm M, Tavenard R, Lefèvre S, et al. Early classification for agricultural monitoring from satellite time series[J].arXiv preprint arXiv:1908.10283,2019.

ETMD: Sharma A, Singh S K, Udmale S S, et al. Early Transportation Mode Detection Using Smartphone Sensing Data[J].IEEE Sensors Journal,2021,21(14):15651-15659.

EARLIEST: Hartvigsen T, Sen C, Kong X, et al.Adaptive-halting policynetwork for early classification[C]//Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery&Data Mining.2019:101-1.

Example 2

Embodiment 2 of the present invention provides a terminal device corresponding to Embodiment 1 above. The terminal device may be a processing device for a client, such as a mobile phone, a notebook computer, a tablet computer, a desktop computer, etc., to execute the method of the above embodiment.

The terminal device in this embodiment includes a memory, a processor, and a computer program stored in the memory; the processor executes the computer program in the memory to implement the steps of the method in Embodiment 1 above.

In some implementations, the memory may be a high-speed random access memory (RAM: Random Access Memory), and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

In other implementations, the processor may be various types of general-purpose processors such as a central processing unit (CPU) and a digital signal processor (DSP), which are not limited herein.

Example 3

Embodiment 3 of the present invention provides a computer-readable storage medium corresponding to Embodiment 1 above, on which computer programs/instructions are stored. When the computer program/instruction is executed by the processor, the steps of the method in Embodiment 1 above are realized.

A computer readable storage medium may be a tangible device that holds and stores instructions for use by an instruction execution device. A computer readable storage medium may be, for example, but is not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any combination of the above.

Those skilled in the art should understand that the embodiments of the present application may be provided as methods, systems, or computer program products. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein. The solutions in the embodiments of the present application can be realized by using various computer languages, for example, the object-oriented programming language Java and the literal translation scripting language JavaScript.

The present application is described with reference to flowcharts and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present application. It should be understood that each procedure and/or block in the flowchart and/or block diagram, and a combination of procedures and/or blocks in the flowchart and/or block diagram can be realized by computer program instructions. These computer program instructions may be provided to a general purpose computer, special purpose computer, embedded processor, or processor of other programmable data processing equipment to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing equipment produce a An apparatus for realizing the functions specified in one or more procedures of the flowchart and/or one or more blocks of the block diagram.

These computer program instructions can also be loaded onto a computer or other programmable data processing device, causing a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby The instructions provide steps for implementing the functions specified in the flow chart or blocks of the flowchart and/or the block or blocks of the block diagrams.

While preferred embodiments of the present application have been described, additional changes and modifications to these embodiments can be made by those skilled in the art once the basic inventive concept is appreciated. Therefore, the appended claims are intended to be construed to cover the preferred embodiment and all changes and modifications which fall within the scope of the application.

Obviously, those skilled in the art can make various changes and modifications to the application without departing from the spirit and scope of the application. In this way, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalent technologies, the present application is also intended to include these modifications and variations.

Claims (10)

1. A time series early classification method, is characterized in that, comprises the following steps: S1. Construct a training set using time series data of human actions; S2. Using the training set to train the neural network; S3. Input the training set into the trained neural network, obtain the classification probabilities of all data in the training set at all moments, and use the classification probabilities to calculate the probability exit threshold; S4. Input the observable data at time t into the trained neural network, obtain the classification probability Pt at time t, and take the maximum value of Pt. If the maximum value is greater than the probability exit threshold, stop inputting observable data. The classification result at time t is taken as the final classification result of the observable data at time t; otherwise, the value of t is increased by 1, and step S4 is repeated. 2. the time series early classification method according to claim 1, is characterized in that, described neural network comprises: A plurality of cascaded first convolution modules are used to perform feature extraction on input data to obtain first features; A plurality of cascaded second convolution modules, the input of which is the first feature, is used to extract high-level features of the input data; The average pooling layer, the input is the high-level feature, and the output is the fusion feature corresponding to the sequence of different time lengths; A linear layer, the input is the fusion feature, and the output is the prediction score of different categories; The exponential normalization layer is used to normalize the prediction score and output classification probability. 3. The time series early classification method according to claim 2, wherein the first convolution module comprises a sequentially connected first hole causal convolution layer, a first normalization layer, and a second hole causal convolution A product layer, a second normalization layer, and a first linear activation unit; the input features of the first convolution module are connected to the output feature residuals. 4. The time series early classification method according to claim 2, wherein the second convolution module includes a dynamic convolution layer, a third normalization layer, and a second linear activation unit; wherein the dynamic The convolution layer includes a convolution kernel generating module, which is used to generate a convolution kernel corresponding to each time by using input data; and perform feature extraction on the input data by using the convolution kernel. 5. The time series early classification method according to claim 4, wherein the convolution kernel generation module comprises a sequentially connected first convolutional layer, a modified linear unit, a batch normalization layer, and a second convolutional layer layer. 6. The time series early classification method according to any one of claims 1 to 4, wherein the implementation process of calculating the probability exit threshold using the classification probability includes: Sorting the classification probabilities and removing duplicates in the classification probabilities; Take the median of the adjacent classification probabilities to obtain a series of threshold candidate values; Select the threshold candidate value with the smallest cost as the probability exit threshold. 7. time series early classification method according to claim 5, is characterized in that, utilizes the cost of following formula to calculate threshold value candidate value: Cost _β =α*(1-Acc _β )+(1-α)*Earliness _β ; Among them, Acc _β is the accuracy rate that the maximum value of the classification probability exits the calculation when it is greater than the threshold candidate value β; Earliness _β is the earlyness that the maximum value of the classification probability exits the calculation when it is greater than the threshold candidate value β, and α is the weight coefficient . 8. The time series early classification method according to claim 6, wherein the value of α is 0.8. 9. A terminal device, comprising a memory, a processor, and a computer program stored on the memory; characterized in that, the processor executes the computer program to implement the steps of the method according to any one of claims 1-8. 10. A computer-readable storage medium, on which computer programs/instructions are stored; characterized in that, when the computer program/instructions are executed by a processor, the steps of the method according to any one of claims 1-8 are implemented.

Images