WO2018053912A1

WO2018053912A1 - Method for real-time action recognition, and related bracelet and computing device

Info

Publication number: WO2018053912A1
Application number: PCT/CN2016/104841
Authority: WO
Inventors: 孙贤军; 程楠; 陈默; 唐佳茵
Original assignee: 上海葡萄纬度科技有限公司
Priority date: 2016-09-23
Filing date: 2016-11-07
Publication date: 2018-03-29
Also published as: CN107870681A

Abstract

A method for timely recognizing actions, comprising: acquiring movement data (ST1); collecting movement samples, and randomly classifying the movement samples into samples for training and samples for testing (ST2); performing marking, feature extraction, feature learning, and model training by using the samples for training, to obtain an action recognition model (ST3); and testing the action recognition model by using the samples for testing (ST4). Also disclosed are a bracelet and a computing device for real-time action recognition using the method.

Description

Method for recognizing actions in real time and corresponding wristbands and computing devices

Technical field

The present invention relates to the field of wearable device technology, and more particularly to a method for recognizing an action in real time and a corresponding wristband and computing device.

Background technique

The bracelet is a popular wearable device that allows users to record real-time data on sports by wearing a bracelet. In the case that parents are now more concerned about the health of children, the bracelet has gradually appeared in children's wearable devices. However, the current use of wristbands is relatively simple, and users often lose interest after using them for a period of time. It is not easy to play a role in supervising sports, nor can it help children to keep fit. At the same time, it is difficult to make bracelets on the market at present. Accurately identify human movements in real time, such as jumping, squatting, swimming, playing basketball, playing badminton, etc.

In response to the above problems, the bracelet and some electronic products are combined to guide children's sports with interest as a guide. Using the pattern recognition algorithm to identify and classify some specified actions (jump, squat, left jump, right jump, etc.), and provide the classification results to subsequent software (such as software on computing devices such as mobile phones, tablets, etc.) For advanced applications (such as interactive entertainment, physical education).

Summary of the invention

In order to solve the above problems, the present invention provides a method for recognizing an action in real time and a wristband using the same.

The method utilizes signal processing and pattern recognition algorithms to identify and classify certain specified actions (jump, squat, left jump, right jump, etc.).

According to a first main aspect of the present invention, a method for recognizing an action in real time is provided, comprising:

Perform motion data collection; collect motion samples and randomly divide them into training required samples and test required samples; use the training required samples to perform sample labeling and feature extraction Taking, feature learning and model training, obtaining a motion recognition model; and testing the motion recognition model using the samples required for the test.

According to a second main aspect of the present invention, a wristband for realizing a real-time recognition action is provided, comprising:

a data measuring device for measuring motion related data; and a wireless communication module for transmitting the measured related data of the motion to a computing device for use with the wristband.

According to a third main aspect of the present invention, a computing device for implementing a real-time recognition action is provided, comprising:

Data receiving means for receiving data related to motion obtained by the data measuring device of the computing device from a wristband used with the computing device; and a data reconstruction and synchronization module for acquiring by the data receiving device The data is reconstructed and synchronized as necessary.

According to the technical solution of the present invention, real-time and relatively accurate and reliable recognition of actions can be realized by a simple method.

DRAWINGS

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the embodiments or the prior art description will be briefly described below. Obviously, the drawings in the following description merely describe some embodiments of the invention. These drawings are not intended to be limiting of the invention, but are exemplary.

1 is a schematic view showing a wristband according to an embodiment of the present invention;

Figure 2 is a functional block diagram showing the functional modules of the wristband of Figure 1;

3 is a schematic diagram showing a wristband and a computing device in accordance with an embodiment of the present invention;

4 is a flow chart showing the basic steps of a real-time motion recognition method according to the present invention;

5(a) and 5(b) illustrate exemplary raw gyro data and acceleration data, respectively;

Figure 6 shows an example of gyro data synchronized with the accelerometer time after reconstruction.

detailed description

The technical solutions in the embodiments of the present invention are clearly and completely described in the following with reference to the accompanying drawings in the embodiments of the present invention. It is obvious that the described embodiments are only a part of the embodiments of the present invention, but not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative efforts are within the scope of the present invention.

FIG. 1 is a schematic view showing a wristband according to an embodiment of the present invention. As shown in FIG. 1 , the wristband is generally annular, and the instrument panel is embedded in the position of the ring. The outer surface of the instrument panel is coated with a color EVA material coating, and the chain plate is provided on both sides of the instrument panel, and the ring structure is formed by the chain belt. The chain belt is an EVA chain belt of the same color and material as the coating layer, and is integrally formed with the coating layer, and the chain connection at both sides is adjusted by the pin buckle to adjust the tightness when the bracelet is worn.

2 is a functional block diagram showing functional modules of the wristband of FIG. 1. As shown in FIG. 2, the wristband includes: a data measuring device including an accelerometer and a gyro instrument for detecting a triaxial acceleration and a triaxial angular velocity of motion, respectively; a Bluetooth module for transmitting the data measuring device via Bluetooth A computing device that works with a bracelet, such as a cell phone, tablet, computer, and so on.

3 is a schematic diagram showing a wristband and computing device in accordance with an embodiment of the present invention. As shown in FIG. 3, the functional modules included in the wristband and their functions are the same as the corresponding portions shown in FIG. 2, and the data receiving device and data are included in the computing device (eg, mobile phone, tablet, computer, etc.). Rebuild and sync module. The data receiving device is configured to receive data measured by the data measuring device from the wristband via Bluetooth, and the data reconstruction and synchronization module is configured to perform reconstruction and synchronization, which will be further described below, on the data acquired by the data receiving device.

4 is a flow chart showing the basic steps of a real-time motion recognition method in accordance with the present invention. As shown in FIG. 4, the real-time motion recognition method according to the present invention mainly includes: 1. In step ST1, motion data acquisition is performed by a sensor of the wristband; 2. In step ST2, motion samples are collected and divided. To train the required samples and test the required samples in two parts; 3. In step ST3, use the samples required for training to perform feature learning and model training of the samples to obtain a motion recognition model; 4. In step ST4, use the test required The sample tests the motion recognition model. These steps will be further explained below.

The first step is to collect motion data.

This step is divided into three steps. The basic task is to read the raw measurement data of the sensor from the wristband device containing the data measuring device, and then transfer the data to the computing device (mobile phone, tablet or PC) via Bluetooth, and then Perform the necessary sensor signal reconstruction and synchronization. Specific steps are as follows:

(a) Motion data collection

Accelerometers and gyroscopes are provided in the wristband device to provide three-axis acceleration measurements and angular velocity measurements. These measurements are then packaged in a suitable data organization format that conforms to the Bluetooth communication protocol (eg, BLE 4.0 Bluetooth Transmission Protocol).

(b) Transfer data via Bluetooth

The data packet of the above measured value is transmitted by the Bluetooth transmission protocol of BLE4.0, and several sets of such data packets are transmitted each time. This Bluetooth transmission protocol is supported by most current devices and is a low-power Bluetooth transmission protocol that can be used to extend battery life and reduce battery capacity. The data receiving end sends the resolvable command to the wristband. When the wristband receives the data, the data measurement of the local sensor is started, and the data that the hand ring updates continuously is sent to the data receiving end through the Bluetooth transmission protocol, and the specific steps are as follows:

(1) Client initialization configuration:

A client installed on a computing device such as a mobile phone, tablet, computer, etc. initializes and turns on Bluetooth, sends a control command to the Bluetooth layer of the wristband, turns on Bluetooth monitoring, and is ready to receive sensor data of the wristband.

(2) Transmitting organization data: Two sensors on the wristband every 10ms will report local real-time data to the bracelet control system layer.

The bracelet control system layer adds the currently acquired sensor data to the serial number and the timestamp generated by the data, and calls the Bluetooth transport layer to send to the client.

(3) After receiving the data, the client performs signal reconstruction through linear interpolation calculation according to the received data packet sequence number and timestamp, and processes the data into smooth data to reduce data sequence discontinuity and uneven time stamp. The effect of signal distortion. Step (c) below specifies how to perform this signal reconstruction.

(c) Sensor signal reconstruction and synchronization

Because the sampling time and sampling rate of the accelerometer and gyroscope data are difficult to synchronize, the gyroscope data with a low sampling rate is reconstructed by data interpolation to synchronize with the moment of the accelerometer. Take the most commonly used linear interpolation method as an example. The specific steps are as follows:

Assume that the data of the gyroscope at time t1 is G(t1), and the data of the gyroscope at time t2 is G(t2), and the data value G(t) of the time t between the time t1 and the time t2 of the gyroscope is as follows. :

Figures 5(a) and 5(b) show example raw gyroscope data and acceleration data, respectively. Figure 6 shows an example of gyro data synchronized with the accelerometer time after reconstruction.

Specifically, how to obtain the gyroscope at the time t=[03:51:49.384] as an example to illustrate the process of reconstructing and synchronizing data using the original gyroscope:

First step. Find the data of the gyro data time t1=[03:51:49.376] and t2=[03:51:49.387] which are close to the acceleration time t=[03:51:49.384].

In the second step, the gyroscope data at the time t=[03:51:49.384] is calculated by the linear interpolation method, and the G(t=[03:51:49.384]) time data value is obtained by substituting the above formula.

In the third step, the above two steps are repeated to obtain the reconstructed gyroscope data synchronized with the accelerometer time shown in FIG. 6.

The second step is to collect the motion samples and randomly divide them into training samples and test samples. The two parts

Let the child wear a bracelet and guide the child to perform specified actions (such as jumping, squatting, jumping to the left, jumping to the right, etc.) to collect samples of the child's designated actions. And the collected samples are randomly divided into two parts, one part is the sample set required for training, and the other part is the sample set required for testing.

The third step is to use the samples required for training to perform sample labeling, feature extraction, and feature learning. Training with the model to get the motion recognition model

For the sample set required for training in the second step, perform feature learning and model training of the sample. The specific steps are as follows:

(a) Mark the sample categories in the sample set required for training, marking the data as jumping, squatting, jumping to the left, jumping to the right, and so on.

(b) Feature extraction

Using each of the above-mentioned classified standard data, one or more of the following 18 features (the model coefficient has 3 features and the wavelet peak has 2 features, a total of 18) are extracted. Specifically, these features are respectively :

If the sample is an m × n-dimensional matrix (a _ij ) _{i = 1, ..., m, j = 1, ..., n} , m represents the length of the intercepted signal, that is, the number of signals taken at several moments, n = 6 denotes 6 channels of the signal, that is, 3 channels of the gyroscope x, y, z and 3 channels of the accelerometer x, y, z; B = (b _ij ) _{i = 1, ..., 18, j=1,...,6} is an 18×6-dimensional matrix representing the feature matrix corresponding to the sample.

1. Absolute value mean:

2. Absolute value mean ratio:

3. Variance: The second-order central moment of each column of the sample, which can be calculated by:

4. Kurtosis: The fourth-order central moment of each column of the sample, which can be calculated by:

μ stands for mean and σ stands for root mean square

5. Skewness, which is the third-order central moment of each column of the sample, which can be calculated by:

6. Root mean square, which can be calculated by:

7. Average absolute deviation, which can be calculated by:

8. Zero-crossing rate: It indicates the number of times the data changes from a positive number to a negative number, or from a negative number to a positive number.

9. Energy: The sum of the squares of the coefficients of the Fourier transform, which can be calculated by the following formula.

10. Correlation coefficient

The correlation coefficient of the X and Y signal channels can be calculated by the following formula.

Where cov(X, Y) represents the covariance of X and Y, and δ _X and δ _Y are the standard deviations of X and Y.

11. Model coefficients (3)

First perform AR (autoregressive) modeling on the data of each channel, ie

Where a _i (n) is the data of the i-th channel, u(n) is the white noise sequence with the variance σ ² , p is the order of the AR model, and λ is the coefficient of the AR model, which can be obtained by the Burg algorithm. We take p=4 and select λ2, λ3, λ4 as features to extract.

12. Quartile difference: The quartile is used to describe the degree of dispersion of data. The calculation method is to calculate the difference between the third quartile and the first quartile after arranging the data from large to small.

13. Wavelet energy

The data for each channel is decomposed based on the wavelet transform of the multi-resolution analysis to obtain:

Where d _jk is the detail coefficient, a _jk is the approximation coefficient, φ _jk (n) is the wavelet function, defined as φ _jk (n)=2 ^j/2 φ(2 ^j nk),

Is a scaling function, defined as

j is a variable representing the scale of expansion, k is a variable representing time, and J is the number of decomposition layers.

The wavelet energy (WE) is equal to the sum of the squares of the wavelet detail coefficients after decomposition. We select the db5 wavelet as the mother wavelet and extract the high-frequency detail coefficient components of the 4th and 5th layers.

14. Fractal dimension

The formula for calculating the fractal dimension is as follows, where ε is the length of one side of the small cube, N(ε) is the number obtained by covering the measured body with this small cube, and the dimension formula means covering by using a small cube with a side length of ε The measured shape determines the dimension of the shape.

15. Small peaks (2, peak and peak number)

The db4 wavelet is used to decompose the acceleration amplitude and the gyro amplitude separately for 7 layers, and the peak of the fourth layer approximation coefficient is detected.

Number of wavelet peaks: The number of peaks of the fourth layer approximation coefficient.

Wavelet peak mean: The average of the peaks of the fourth layer approximation coefficient.

(c) Training data

Randomly select a certain number of combinations of the above features, using some pattern classification algorithms (including but not limited to such as decision trees, Bayesian networks, artificial neural networks, K-nearest neighbors, support vector machines, Boosting, etc.), after calculation, from the above The model parameters of the selected pattern classification algorithm are obtained in the feature, and then the motion recognition model is obtained.

If the support vector machine algorithm is used, the parameter values of ω and b in ω ^T x+b=0 are obtained; if the K-nearest neighbor algorithm is used, the appropriate distance measure between the feature vectors is obtained; for example, using an artificial neural network algorithm, Then get the offset value of the weight of the network node. Those skilled in the art are familiar with how to use the above algorithm to perform data training to obtain a related motion recognition model, and thus will not be described herein.

The fourth step is to test the motion recognition model with the samples required for the test.

Using the test sample set obtained in the second step, repeat the step (b) in the third step to extract the feature, and put the feature into the motion recognition model to calculate the matching, and obtain the classification result, that is, identify the specific type of the action (jumping) Still squatting, left or right jump, etc.).

The above is a general description of the basic steps of the real-time motion recognition method. In the following, the real-time identification of some specific actions by the present method will be further described in conjunction with the specific embodiments.

Embodiment 1, a method for recognizing a jumping action in real time

The subject s person, each person doing jumping, squatting, jumping to the left, and jumping to the right for each d times, each action gets sd action data, starting from the starting point of each action data, taking f before and after Successive time data, such that an action data segment of length f is extracted, and the gyroscope and the accelerometer have a total of 6 signal channels, and the data segment of the action has 6 f measurement values; for each of the data segments Channels, each extracting 18 features, 6 channels can acquire 108 features, so that each type of action obtains a total of sd groups, each group of 108 features of the sample; randomly select 1/10 of the feature samples as test samples, the rest Mostly as training samples; select any pattern classification algorithm (including and not limited to such as decision tree, Bayesian network, artificial neural network, K-nearest neighbor, support vector machine, Boosting, etc.). In order to simplify the description, the K-proximity method with training time complexity of 0 is used to train the training samples to obtain the motion recognition model.

Finally, the test sample is taken as a test to test the classification accuracy rate. Specifically, for four different types of actions (upward jump, left jump, right jump, squat), each action type acquires sd/10 data, calculates their respective characteristics, and puts these features into the already trained In a good model, the matching is calculated, and the classification result of each action is obtained. For example, whether the jumping action is recognized as a jump, whether the squatting action is recognized as a squat, and what ratio is correctly classified. The test results show that the recognition accuracy of these actions can reach 90% to 95%.

The set of products skillfully applies the pattern recognition algorithm, extracts the characteristics of the action information, classifies the actions, and has a fast calculation speed and accurate classification, which can unify the hardware and software technology well. This set of products has the following characteristics:

1. The hardware design is simple.

Pure software processing algorithms, in addition to the sensors needed to acquire data, do not require any additional hardware design coordination.

2. The detection algorithm is simple.

It utilizes sophisticated pattern recognition algorithms including, but not limited to, decision trees, Bayesian networks, artificial neural networks, K-nearest neighbors, support vector machines, Boosting, and the like.

3. Fast calculation speed

Each sorting test takes about 150ms.

4. Detection performance is stable

The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. It should be understood that the features disclosed in the above embodiments may be used singly or in combination, unless otherwise specified. Various modifications to these embodiments are obvious to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention as disclosed herein is not limited to the specific embodiments disclosed, but is intended to cover modifications within the spirit and scope of the invention as defined by the appended claims.

Claims

A method for recognizing an action in real time, comprising:

Perform motion data collection;

Collect motion samples and randomly divide them into training samples and test samples.

Using the sample required for the training to perform sample tagging, feature extraction, feature learning, and model training of the sample to obtain a motion recognition model;

The motion recognition model is tested using the samples required for the test.
The method according to claim 1, wherein performing the motion data acquisition further comprises adding the serial number to the collected motion data and the time stamp when the data is generated.
The method of claim 2 wherein said motion data comprises motion triaxial acceleration data and triaxial angular velocity data.
The method of claim 3 wherein performing motion data acquisition further comprises reconstructing said triaxial angular velocity data having a lower sampling rate using numerical interpolation to synchronize with said triaxial acceleration data.
The method of claim 4 wherein said data interpolation is a linear interpolation.
The method according to claim 1, wherein the sample identification, the feature extraction, the feature learning and the model training of the sample are performed by using the sample required for training, and the motion recognition model is obtained, and the action type corresponding to the sample required for the training is marked. .
The method according to claim 6, wherein the sample identification, the feature extraction, the feature learning and the model training of the sample are performed by using the sample required for training, and the motion recognition model is further obtained, and the obtained training required sample is extracted from the following samples. One or more of the characteristics:

Absolute value mean, absolute value mean ratio, variance, kurtosis, skewness, root mean square, mean absolute deviation, zero crossing rate, energy, correlation coefficient, model coefficient, interquartile range, wavelet energy, fractal dimension, small crest.
The method according to claim 7, wherein the sample identification, feature extraction, feature learning and model training of the sample are performed by using the sample required for training, and the motion recognition model is further obtained by using a pattern classification algorithm for calculation, based on the extracted The model parameters are obtained by one or more of the features to obtain a motion recognition model.
The method according to claim 8, wherein the pattern classification algorithm is selected from the group consisting of a decision tree, a Bayesian network, an artificial neural network, a K-nearest neighbor, a support vector machine, a Boosting, and the like.
The method according to claim 8, wherein the testing the motion recognition model by using the sample required for testing comprises repeating the feature extraction using the sample required for the test, and then identifying the motion of the motion recognition model The effect is evaluated.
The method of claim 1 wherein the action comprises one or more of jumping, squatting, jumping to the left, and jumping to the right.
A bracelet for real-time recognition actions, including:

a data measuring device for measuring motion related data;

And a wireless communication module, configured to send the measured related data of the motion to a computing device used in conjunction with the wristband.
The wristband according to claim 12, wherein said data measuring means comprises an accelerometer and a gyroscope, and said motion related data comprises moving triaxial acceleration data and triaxial angular velocity data.
The wristband according to claim 12, wherein the wireless communication module is a Bluetooth communication module.
The wristband according to claim 12, wherein said wristband further comprises means for adding a sequence number to said motion related data and a time stamp when said data is generated.
A computing device for implementing real-time recognition actions, comprising:

Data receiving means for receiving data relating to motion obtained by the data measuring device of the computing device from a wristband used in conjunction with the computing device;

A data reconstruction and synchronization module for performing necessary reconstruction and synchronization of data acquired by the data receiving device.
The computing device of claim 16 wherein said data receiving device receives said motion related data from said wristband via a Bluetooth communication module.
The computing device of claim 16 wherein the motion related data comprises motion triaxial acceleration data and triaxial angular velocity data.
The computing device of claim 18, wherein said computing device further comprises data reconstruction means for reconstructing said triaxial angular velocity data having a lower sampling rate using numerical interpolation to cause said The three-axis acceleration data is synchronized.
The computing device of claim 18 wherein said data interpolation is a linear interpolation.
The computing device of claim 16 wherein said computing device further comprises feature extraction means for extracting one or more of the following features for the obtained motion samples required for training:

Absolute value mean, absolute value mean ratio, variance, kurtosis, skewness, root mean square, mean absolute deviation, zero crossing rate, energy, correlation coefficient, model coefficient, interquartile range, wavelet energy, fractal dimension, small crest.
The computing device of claim 21, wherein said computing device further comprises motion recognition model determining means for performing a calculation using a pattern classification algorithm, obtaining model parameters based on the extracted one or more of said features To get the motion recognition model.
The computing device of claim 22, wherein the pattern classification algorithm is selected from the group consisting of a decision tree, a Bayesian network, an artificial neural network, a K-nearest neighbor, a support vector machine, a Boosting, and the like.
The computing device of claim 16, wherein the action comprises one or more of jumping, squatting, jumping to the left, and jumping to the right.
The computing device of claim 16, wherein the computing device comprises one or more of a cell phone, a tablet, and a computer.