CN113952731B - Motion sensing game action recognition method and system based on multi-stage joint training - Google Patents
Motion sensing game action recognition method and system based on multi-stage joint training Download PDFInfo
- Publication number
- CN113952731B CN113952731B CN202111571304.XA CN202111571304A CN113952731B CN 113952731 B CN113952731 B CN 113952731B CN 202111571304 A CN202111571304 A CN 202111571304A CN 113952731 B CN113952731 B CN 113952731B
- Authority
- CN
- China
- Prior art keywords
- game
- action
- meta
- motion
- training
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- A—HUMAN NECESSITIES
- A63—SPORTS; GAMES; AMUSEMENTS
- A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- A63F13/00—Video games, i.e. games using an electronically generated display having two or more dimensions
- A63F13/60—Generating or modifying game content before or while executing the game program, e.g. authoring tools specially adapted for game development or game-integrated level editor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
- G06N20/20—Ensemble learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- A—HUMAN NECESSITIES
- A63—SPORTS; GAMES; AMUSEMENTS
- A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- A63F2300/00—Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game
- A63F2300/60—Methods for processing data by generating or executing the game program
- A63F2300/6009—Methods for processing data by generating or executing the game program for importing or creating game content, e.g. authoring tools during game development, adapting content to different platforms, use of a scripting language to create content
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Artificial Intelligence (AREA)
- General Engineering & Computer Science (AREA)
- Evolutionary Computation (AREA)
- General Physics & Mathematics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Software Systems (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Mathematical Physics (AREA)
- Computing Systems (AREA)
- Biophysics (AREA)
- Molecular Biology (AREA)
- General Health & Medical Sciences (AREA)
- Computational Linguistics (AREA)
- Biomedical Technology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Health & Medical Sciences (AREA)
- Evolutionary Biology (AREA)
- Multimedia (AREA)
- Medical Informatics (AREA)
- Image Analysis (AREA)
Abstract
The invention relates to a motion recognition method of a multi-stage joint training motion sensing game, which realizes motion recognition of the motion sensing game through joint training in a game host development stage and a game software development stage and comprises the following steps: constructing and training a CNN network capable of recognizing meta-actions; constructing and training a CNN-seq2seq-Adaboost model capable of identifying complex actions; collecting the change of radio frequency signals caused by game actions in a game stage to obtain game action data; interrupting and labeling game action data according to the meta-action; extracting characteristics of the game action data subjected to interruption and label marking, and generating action identification data based on the extracted characteristics; and inputting the generated motion recognition data into a trained CNN-seq2seq-Adaboost model, and outputting a game motion recognition result by the CNN-seq2seq-Adaboost model.
Description
Technical Field
The invention relates to the technical field of wireless perception, in particular to a motion recognition method and system for a multi-stage joint training motion sensing game.
Background
With the deep mind of the concepts of virtual reality, metas and the like, people are no longer satisfied with playing games on electronic screens through a handle or a keyboard. The motion sensing game is a brand-new game form, and can lead players to take part in game contents, thus raising the trend of the game field. At present, motion recognition is carried out on motion sensing games by adopting devices such as a camera, an infrared sensor and a gravity sensor, the cost of the devices is high, and the motion sensing games are difficult to enter common families all the time.
Recently, a motion recognition technology based on radio frequency signals is proposed, and the motion recognition technology is expected to be applied to motion recognition of motion sensing games. The existing research method mainly includes capturing interference of player actions on radio frequency signals, extracting corresponding features from the interference, and using the features to train in a deep learning network to obtain a deep learning model for action recognition, so that the prediction of action accuracy is completed, and the purpose of motion sensing game action recognition is achieved.
However, the conventional method has the following three problems in specific development.
First, the existing method requires a game software developer to autonomously collect a large amount of standard data as a training set to complete the training of the motion recognition model. Compared with traditional game development, the method is extra workload, the development difficulty of game software developers on the games is greatly improved, new actions and action combinations exist in each new motion sensing game, and a new training set needs to be reconstructed to train a new machine learning model. Because a large number of small-amplitude actions exist in the motion sensing game, the requirements of the action recognition model on the quality and the scale of the training set are high, otherwise, the over-fitting or under-fitting condition is easy to occur. The huge training set construction cost makes most game software developers unwilling to accept new somatosensory game schemes based on radio frequency signals.
Second, the deep learning-based motion recognition algorithm is difficult to apply to a new recognition environment. Most of the existing methods collect and mark the motion characteristics of a specific person in a specific environment, and learn the collected data set by using a deep learning algorithm, such as CNN, RNN, and the like, and the obtained model is used for identifying the motion of the person. However, since the radio frequency signal is very sensitive to the surrounding environment and strongly influenced by the multipath effect, the change of the environment and the change of the person both cause the great reduction of the identification accuracy. This makes the method implementable only in the experimental environment. Even if the game software developer finishes training, the training result is greatly influenced by the body type and the playing environment of the player in the playing process of the individual player, and the recognition accuracy is low.
Third, the existing classification algorithm-based method does not take into account the effect of the action timing. Most of the existing action recognition methods based on radio frequency signals are directed to simple actions or a few actions with limited quantity, and adopt an action recognition algorithm taking classification as a core. The algorithm can meet the small-scale action identification requirements, such as step number identification and old people falling identification, but cannot play a good role in a motion sensing game scene. The motion in the motion sensing game is usually composed of a plurality of element motions, such as lifting the left leg, putting down the right hand and the like; the combination of these meta-actions fulfills the action requirements of the motion sensing game. Although existing classified core motion recognition algorithms can exert certain effects in the case of single motion or simple motion combination, they fail in the case of complex motion combined by multiple motions. The reason is that the algorithms with classification as core can only compare actions and cannot understand the meaning of the actions. For example, when the existing algorithm recognizes that the left foot of a person is lifted and the right foot is also lifted, the action matching is considered to be failed, and the action matching cannot be recognized. The existing algorithm cannot comprehensively judge the matching degree of the complete action according to the consistency of the action. This will result in a significant decrease in the accuracy of existing methods in motion sensing games.
Disclosure of Invention
The invention provides a motion sensing game motion recognition method based on multi-stage joint training, which adopts a multi-stage joint training scheme to divide data acquisition training related to motion recognition into 2 stages, and the 2 stages are distributed to two groups of game machine hardware equipment developers and game software developers, so that the development difficulty of motion sensing games is reduced.
A second object of the present invention is to provide a motion recognition system for a multi-stage joint training motion sensing game.
In order to realize the first invention, the technical scheme is as follows:
a motion recognition method of a multi-stage joint training motion sensing game realizes motion recognition of the motion sensing game through joint training in a game host development stage and a game software development stage, and comprises the following steps:
first, game host development stage
Selecting and determining a series of element actions supported by a somatosensory game developed on a game running platform, and collecting radio frequency signal changes caused by the element actions in a standard environment to obtain element action data;
extracting the characteristics of the meta-motion data of the series of meta-motions, and generating a meta-motion training data set based on the extracted characteristics;
constructing a CNN network, and training the constructed CNN network by using the meta-action training data set;
second, game software development stage
Acquiring corresponding complex actions in a standard environment to cause the change of radio frequency signals according to the requirements of the developed motion sensing game, obtaining complex action data, and interrupting and labeling the complex action data according to the meta-actions;
extracting features of the complex motion data subjected to interruption and label marking, and generating a complex motion training data set based on the extracted features;
receiving parameters of a CNN network trained in a game host development stage, constructing a CNN-seq2seq-Adaboost model with the CNN parameters consistent with the CNN network parameters trained in the game host development stage, and training the constructed CNN-seq2seq-Adaboost model by using the complex motion training data set; in the training process, parameters of an input layer and a convolutional layer in the CNN-seq2seq-Adaboost model are kept unchanged;
third, game stage
Collecting the change of radio frequency signals caused by game actions in a game stage to obtain game action data; interrupting and labeling game action data according to the meta-action;
extracting characteristics of the game action data subjected to interruption and label marking, and generating action identification data based on the extracted characteristics;
and inputting the generated motion recognition data into a trained CNN-seq2seq-Adaboost model, and outputting a game motion recognition result by the CNN-seq2seq-Adaboost model.
Preferably, after the metadata motion data, the complex motion data, and the game motion data are collected, the metadata motion data, the complex motion data, and the game motion data are filtered and denoised, respectively.
Preferably, in the development stage of the game host, the feature extraction of the meta-action data of the series of meta-actions includes: and extracting the signal intensity characteristic and the arrival angle characteristic in the meta-action data of each meta-action, constructing a characteristic matrix according to the signal intensity characteristic and the arrival angle characteristic of each meta-action, and marking a corresponding meta-action category label for the characteristic matrix.
Preferably, in the game software development phase, the interrupting and tagging the complex action data according to the meta-action includes:
dividing the complex action data into a plurality of meta-action data segments according to meta-actions, and respectively marking corresponding meta-action data segment labels, wherein the meta-action data segment labels are as follows: location, sub-action, wherein location represents the position of the meta-action data segment in the complex action data, sub-action represents the meta-action category tag of the meta-action data segment, and action represents the category to which the complex action belongs;
in the game stage, the interrupting and labeling of the game action data according to the meta-action comprises: dividing game action data into a plurality of meta-action data segments according to meta-actions, and respectively marking corresponding meta-action data segment labels, wherein the meta-action data segment labels are as follows: [l,s,a]WhereinlIndicating the position of the piece of meta motion data in the game motion data,sa meta-action category tag indicating the piece of meta-action data,aindicating the category to which the game action belongs.
Preferably, the feature extraction of the complex action data for performing interruption and labeling in the game software development stage includes: extracting signal intensity characteristics and arrival angle characteristics in each meta-action data segment, constructing a two-dimensional characteristic matrix according to the signal intensity characteristics and arrival angle characteristics of each meta-action data segment, and constructing elements in a complex action training data set by the two-dimensional characteristic matrices of all the meta-action data segments of the complex action data; the two-dimensional characteristic matrix of each meta-motion data segment is associated with the tag of the meta-motion data segment;
the feature extraction of game action data for performing interruption and label marking in the game stage comprises the following steps: and extracting the signal intensity characteristic and the arrival angle characteristic in each section of meta-action data, constructing a two-dimensional characteristic matrix according to the signal intensity characteristic and the arrival angle characteristic of each section of meta-action data, and constructing action identification data by the two-dimensional characteristic matrix of all the meta-action data sections of the game action data.
Preferably, the training of the constructed CNN-seq2seq-Adaboost model by using the complex motion training data set includes:
1) training the CNN part in the CNN-seq2seq-Adaboost model:
taking a two-dimensional characteristic matrix of each meta-motion data segment in the complex motion training data set as the input of a CNN part, and outputting the prediction category and the probability of the meta-motion data segment by the CNN part;
updating parameters of other layers except the input layer and the convolution layer in the CNN part by utilizing gradient back propagation;
2) training the seq2seq part in the CNN-seq2seq-Adaboost model:
the meta-action class label sequence [ sub-action1, sub-action2, … ] in the complex action training data set]As input, a one-dimensional matrix, and a complex sequence of motion labels [ action1，action2，action3，…]Training as a training target input seq2seq part;
the seq2seq part outputs a new meta-action type label sequence and a complex action label thereof;
updating parameters of the seq2seq part using gradient back propagation;
3) training the Adaboost part in the CNN-seq2seq-Adaboost model:
the meta-motion class label sequence output by the seq2seq part is used as a new label sequence and a complex motion label [ action i ]Together as a training set, the Adaboost part is trained and the recognition accuracy of the complex motion is output.
Preferably, the strong classifier used in the Adaboost part is obtained by weighted averaging of a plurality of weak classifiers, and the weak classifier is CART.
Preferably, the CNN network and the CNN part in the CNN-seq2seq-Adaboost model have the same structure, and each of the CNN network and the CNN part comprises a 1-layer input layer, a 3-layer convolutional layer, a 1-layer average pooling layer, a 2-layer fully-connected layer, and a 1-layer output layer, which are sequentially connected.
Preferably, in the game stage, before the acquisition of the change of the radio frequency signal caused by the game action in the game stage, the operation of data reconstruction is carried out:
collecting radio frequency signal changes caused by making a meta-action in the same game environment to obtain meta-action data in the game environment;
extracting the characteristics of the metadata under the game environment;
comparing the extracted characteristics of the meta-motion data under the game environment with the extracted characteristics of the meta-motion data of the same type of meta-motion in the meta-motion training data set, and determining a ratio;
and adjusting the subsequently obtained action identification data by using the determined ratio.
In order to realize the second invention, the adopted technical scheme is as follows:
a motion sensing game motion recognition system of multi-stage joint training applies the motion sensing game motion recognition method of the multi-stage joint training, and comprises a first-stage training module, a second-stage training module and a game host;
the first-stage training module is used for executing the training step of the game host development stage;
the second stage training module is used for executing the training step of the game software development stage;
the game host is used for executing the steps of the game stage.
Compared with the prior art, the invention has the beneficial effects that:
1) the invention adopts a multi-party combined training scheme, and the structure of the training set is dispersed to two groups of game machine hardware equipment developers and game software developers, thereby reducing the development difficulty of the motion sensing game.
2) The game software developer trains complex actions and retains parameters at the high level of the model, the player performs simple test aiming at the use environment and the body type of the player and reconstructs action recognition data, three-party combined training is performed, and work is divided layer by layer, so that a more accurate action recognition model in the playing environment of the player is realized, and the game experience of the player is improved.
3) The method uses the thought of machine translation, treats each element action as a word, extracts a word vector of each element action by using seq2seq, gives a new meaning to the complex action by learning the time sequence information of the element action in the complex action, maps the new meaning into a new label sequence, classifies the new label sequence by adopting Adaboost, and greatly improves the accuracy of action recognition by using the time sequence information in the complex action.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to these drawings without inventive exercise.
Fig. 1 is a frame flow diagram of a motion recognition method of a multi-stage joint training motion sensing game.
Fig. 2 is a schematic structural diagram of a motion recognition system of a multi-stage joint training motion sensing game.
Fig. 3 is a flowchart illustrating a motion recognition method of a multi-stage joint training motion sensing game according to embodiment 1.
Detailed Description
In order to make the objects, features and advantages of the present invention more obvious and understandable, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is obvious that the embodiments described below are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Example 1
As shown in fig. 1, the method for recognizing motion of multi-stage joint training motion sensing game provided by the present invention comprises three stages, namely, a game host development stage, a game software development stage and a game stage, wherein the game host development stage is completed by a game host hardware manufacturer, the game software development stage is completed by a game software developer, and the game stage is completed by a home game host.
The parts are specifically explained as follows:
1) development stage of game host
In the development stage of the game host, a game host hardware manufacturer completes the data collection and training process, and in this stage, as shown in fig. 2, the game host hardware manufacturer mainly relies on the constructed first-stage training module to complete the whole operation steps. The first-stage training module comprises a meta-motion selection module, a meta-motion data acquisition module, a meta-motion training data set generation module, a model training module, a parameter extraction module and a data broadcasting module.
1.1) meta-action selection module: and the system is responsible for defining basic element actions which can be identified by the motion sensing game, such as basic simple actions of lifting legs, turning bodies and the like.
1.2) a meta-action data acquisition module: the system is responsible for collecting radio frequency signal changes caused by standard element actions in a zone which is as open as possible to obtain element action data which is used as a game platform for developing a body sensing game standard.
1.3) a meta-action training data set generation module: and the system is responsible for carrying out basic filtering and noise reduction on the acquired meta-motion data corresponding to the standard meta-motion and generating a meta-motion training data set. The radio frequency signal change corresponding to the meta-action data is the change of the signal strength and the arrival angle of the radio frequency signal, which respectively represents the distance and angle change caused by the action. The filtering method is a basic radio frequency signal filtering method, such as kalman filtering, high-pass filtering, and the like. The noise reduction method is a basic radio frequency signal noise reduction method, such as mean value noise reduction, wavelet transformation and the like. All metadata in the invention are filtered and denoised, and are pure enough. And determining signal changes corresponding to all the element actions according to sampling frequency and identification requirements, extracting signal intensity characteristics and arrival angle characteristics, recording the signal intensity characteristics and the arrival angle characteristics as H, filling zero with the length being less than H to H, obtaining [2 x H ] two-dimensional characteristic matrixes with the same size corresponding to each element action, marking corresponding element action type labels [ sub-actions ] for each two-dimensional characteristic matrix, and generating an element action training data set. For example, if the maximum number of sampling points of all the collected meta-motion data is 128, then the signal strength and arrival angle sampling values of all the meta-motions are put into a [2 × 128] two-dimensional matrix, and if there are less than 128 bits, the elements of the spare matrix are all set to 0.
1.4) a model training module: and the CNN network is responsible for training the CNN network to obtain the CNN network capable of identifying the standard element action. The CNN network inputs the meta-motion training data set after the label is coded and outputs the classification probability P corresponding to the meta-motion. The structure of the device comprises 1 input layer, 3 convolutional layers, 1 average pooling layer, 2 full-connection layers and 1 output layer which are sequentially connected. The convolutional kernel used by the convolutional layer is determined according to actual requirements, the convolutional layer activation function is PRelu, the fully-connected layer activation function is SoftMax, the Loss of the CNN network is Binary cross entry, and parameters of each layer of the CNN network, such as weights of neurons and the like, are updated by using a gradient back propagation algorithm. And training until the Loss of the whole CNN network converges to obtain the CNN network capable of identifying the standard element action.
1.5) a parameter extraction module: and the CNN network is responsible for extracting and storing the structure and various parameters of the CNN network.
1.6) a data broadcasting module: and the system is responsible for publishing the network structure, parameters, used meta-motion training data set and the like of the CNN network which is trained by the game machine hardware manufacturer and can identify the meta-motion to all game software developers.
2) Game software development phase
In the game software development stage, the game software developer mainly completes the data collection and training process, and in this stage, as shown in fig. 2, the game software developer mainly relies on the constructed second stage training module to complete the whole operation steps. The second-stage training module comprises a complex action acquisition module, a complex action training data set generation module, a model parameter loading module, a model training module and a parameter extraction module.
2.1) Complex action acquisition Module
The system is responsible for collecting the radio frequency signal change caused by the complex action needing to be identified in the motion sensing game to be developed and labeling the label. Wherein, the collection environment is as spacious as possible, reducing the influence of multipath effect. The complex actions are composed of meta-actions provided by game machine hardware manufacturers as much as possible, and sampling points are interrupted for the complex actions according to the meta-actions in the process of signal acquisition. For example, a complex motion consists of 4 elements, and the total number of samples is 1000, which is broken into four signal data segments of 100+300+200+ 400. Labeling the four pieces of data in time sequence, wherein the structure of the four pieces of data is [ location, sub-action, action ], such as 1,1, 1; 2,3, 1; 3,5, 1; 4,6,1. Wherein, the first digit represents the position of the data in the complex action data, the second digit represents the meta-action type label of the data, and the third digit represents the type of the complex action.
2.2) Complex movement training data set generating module
And generating a complex motion training data set. And (3) performing feature extraction on data in each meta-action data segment in the complex action to obtain a [2 multiplied by H ] two-dimensional matrix, and reserving all three labels (positions, meta-action category labels and complex action categories).
2.3) model parameter loading module
The method is used for constructing the CNN-seq2seq-Adaboost model and receiving the CNN model structure and parameters broadcast by a game machine hardware manufacturer. In the constructed CNN-seq2seq-Adaboost model, the network structure of the CNN part is the same as that of a game machine hardware manufacturer, and the structure comprises a 1-layer input layer, a 3-layer convolutional layer, a 1-layer average pooling layer, a 2-layer full-connection layer and a 1-layer output layer which are sequentially connected. The CNN parameters of the first 4 layers are retained and remain unchanged forever (i.e., the first three convolutional layer parameters are not updated by the next gradient backpropagation). This is because the information extracted at the lower level has stronger generalization ability and coarser granularity, and the information extracted at the higher level has weaker generalization ability and finer granularity. The retention of low-level network parameters can greatly accelerate the Loss convergence speed, so that the model training speed is higher, the environmental influence from the meta-action training data set is retained, and the generalization capability of the model is favorably improved.
2.4) model training Module
And the model is responsible for training the CNN-seq2seq-Adaboost model. The network structure of the CNN part is as described above, and its first 4-layer parameters are never changed. The training set is a two-dimensional characteristic matrix and a label [ sub-action ] of each meta-action data segment in the complex action training data set. Inputting a two-dimensional characteristic matrix of the meta-motion data segment, and outputting the prediction category and the probability of the meta-motion data segment. Gradient backpropagation is used to update the data of the layers other than the first four layers. Due to the difference of data set collection environments and the influence of actual complex actions, the two-dimensional feature matrix of the metadata action data segment at the moment is different from the characteristics of the same type of metadata action data collected by a game machine hardware manufacturer, and parameters of the back 8 layers of the CNN network are also changed. At this time, the trained CNN network has the capability of identifying the element action according to the requirement of the motion sensing game.
After the CNN network identifies the meta-motion, the meta-motion category label sequence [ sub-action1, sub-action2, … ] corresponding to the complex motion is used]As input, a one-dimensional matrix, and the complex sequence of motion tags [ action ]1，action2，action3，…]And (4) inputting a seq2seq network as a training target to train. The length of the one-dimensional matrix is longer than the longest element motion sequence in the complex motionThe degree is determined, denoted as L, and the spare bits are filled with 0. The seq2seq network is a multilayer seq2seq network in which the most basic semantic vector participates in all time operations of the sequence, and the specific number of layers can be freely set according to requirements. Its loss is Binary cross entry, the used entry mechanism is Bahdanau entry, and the updating parameters are propagated backward by gradient. The output is a new meta-action category label sequence and a complex action label thereof, and the length of the new meta-action category label sequence is unified to be S. For example, let L =8, S =6, and the input matrix of seq2seq at this time is [1,2,3,4,5,0,0,0]The target matrix is [7,7,7,7]The seq2seq model can then be updated by gradient backpropagation. When testing, if the testing matrix is [1,2,3,4,5,0,0,0](the matrix represented by the standard action that has occurred), then the output of seq2seq is [7,7,7,7]. If the test matrix is [2,2,3,4,5,0,0,0](matrix represented by non-standard actions), the output of seq2seq is [7,7,7,7,6,7]。
The meta-motion category label sequence output by seq2seq is taken as a new label sequence and a complex motion label [ action ] i ]And the Adaboost is trained and the recognition accuracy of the complex motion is output together as a training set. The weak classifier used by Adaboost is CART, the final strong classifier is obtained by weighted averaging of a plurality of CARTs, and a forward distribution learning algorithm is adopted.
2.5) a parameter extraction module: and the model is responsible for extracting and storing the structure and parameters of the trained CNN-seq2seq-Adaboost model.
3) Game stage
In the game stage, the household game host machine mainly completes the identification process of game actions. As shown in fig. 2, the home game console includes a meta motion data collection module, a motion data reconstruction module, a model parameter loading module, a model loading module, and an accuracy prediction module.
3.1) a meta-action data acquisition module: and the system is responsible for acquiring the signal strength characteristic and the arrival angle characteristic of the radio frequency signal corresponding to the meta-action in the game environment for calibration. During specific collection, the sampling point interruption of the complex action is carried out according to the interruption method of the game manufacturer on the sampling point of the complex action.
3.2) an action data reconstruction module: and reconstructing all captured actions by acquiring the difference between the signal strength characteristic and the arrival angle characteristic corresponding to the element actions and the signal strength characteristic and the arrival angle characteristic corresponding to the element actions acquired by a game machine hardware manufacturer under the game environment. For example, if the signal strength characteristic value collected in the game environment is 100, but the signal strength characteristic value of the corresponding element action in the standard environment is 80, it may be determined that the ratio between the game environment and the standard environment is 0.8, and all the characteristics collected in the game environment are adjusted by the ratio to realize data reconstruction.
3.3) a model parameter loading module: and the CNN-seq2seq-Adaboost model is responsible for receiving the CNN-seq2seq-Adaboost model trained and completed by a game software developer. All parameters are not changed.
3.4) model loading module: is responsible for loading the CNN-seq2seq-Adaboost model.
3.5) accuracy prediction module: and the CNN-seq2seq-Adaboost model is responsible for predicting the accuracy of a certain action of a player. Inputting a feature matrix [2 XH ] corresponding to the broken series of element actions, and outputting a label predicted value [ action ] and a predicted accuracy P for the complex action.
As shown in fig. 3, the specific flow steps of the motion sensing game motion recognition method of multi-stage joint training provided by the present invention are as follows:
s201: a game machine hardware manufacturer selects a series of element actions supported by a motion sensing game which can be developed on a host platform of the game machine hardware manufacturer, and collects radio frequency signal changes caused by the element actions in a standard environment;
s202: the game machine hardware manufacturer carries out feature extraction processing on the meta-motion data, generates a meta-motion training data set, and trains a CNN network capable of identifying the meta-motion;
s203: the game machine hardware manufacturer stores the parameters of the trained CNN network and sends the CNN network, the CNN network parameters and the used meta-action information to all game software developers developing the motion sensing game on the platform;
s204: a game software developer collects corresponding complex actions in a standard environment to cause the change of radio frequency signals according to the requirements of the developed motion sensing game, and breaks and labels complex action data according to the meta-actions;
s205: a game software developer generates a complex action training data set, receives a CNN network trained by a game hardware manufacturer, trains a CNN-seq2seq-Adaboost model under the condition that the parameters of the first three layers are not changed, and transmits the model structure and the model parameters to a household game host;
s206: the home game host firstly collects the metadata, determines the difference between the home environment and the standard environment, and reconstructs all collected action data according to the difference;
s207: and the household game host loads a CNN-seq2seq-Adaboost model structure and model parameters which are trained by a game software developer, and uses the model to predict the action accuracy to meet the requirements of the motion sensing game.
Example 2
The embodiment provides an application example, and the specific flow is as follows:
1) a game host hardware manufacturer collects meta-motion data to construct a meta-motion training data set, wherein meta-motion category labels are 1,2,3, …, 9 and 10, 10 meta-motions are totally arranged, and a CNN network capable of identifying the meta-motions is trained;
2) a game software developer designs corresponding complex actions to complete the motion sensing game according to the combination of the meta-actions, for example, designs a complex action with a meta-action sequence of [1346700], and the label of the complex action is 001;
3) and (3) collecting characteristic data of a plurality of complex actions by a game software developer, receiving CNN network parameters from a game host hardware manufacturer, keeping the front three layers unchanged, only updating the rear parameters, and training to obtain a CNN-seq2seq-Adaboost model. For the complex action 001, the tag sequence identified after the complex action passes through the CNN network is [1346700], the tag sequence obtained after the complex action passes through the seq2seq is [111111 ], and the identified action type is 001;
4) and collecting the meta-motion data of the player in the game environment, comparing the meta-motion data with the meta-motion data in the standard environment, and reconstructing all subsequently received motion signals according to the comparison result.
5) The player makes a complex action 001, but not standard, and uses the CNN-seq2seq-Adaboost model for standard degree prediction;
6) the tag sequence identified after the complex motion 001 passes through the CNN network is [1246700], the tag sequence obtained after seq2seq is [111131111], the motion type is 001 after adaboost, and the probability is 0.92;
7) the game host machine determines how to carry out the next game according to the action accuracy of the action of the player.
In the several embodiments provided in the present application, it should be understood that the disclosed system, apparatus and method may be implemented in other manners. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the units is only one logical division, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-only memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.
The above-mentioned embodiments are only used for illustrating the technical solutions of the present invention, and not for limiting the same; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (7)
1. A motion recognition method of a multi-stage joint training motion sensing game realizes motion recognition of the motion sensing game through joint training in a game host development stage and a game software development stage, and is characterized in that: the method comprises the following steps:
first, game host development stage
Selecting and determining a series of element actions supported by a somatosensory game developed on a game running platform, and collecting radio frequency signal changes caused by the element actions in a standard environment to obtain element action data;
extracting the characteristics of the meta-motion data of the series of meta-motions, and generating a meta-motion training data set based on the extracted characteristics;
constructing a CNN network, and training the constructed CNN network by using the meta-action training data set;
second, game software development stage
Acquiring corresponding complex actions in a standard environment to cause the change of radio frequency signals according to the requirements of the developed motion sensing game, obtaining complex action data, and interrupting and labeling the complex action data according to the meta-actions;
extracting features of the complex motion data subjected to interruption and label marking, and generating a complex motion training data set based on the extracted features;
receiving parameters of a CNN network trained in a game host development stage, constructing a CNN-seq2seq-Adaboost model with the CNN parameters consistent with the CNN network parameters trained in the game host development stage, and training the constructed CNN-seq2seq-Adaboost model by using the complex motion training data set; in the training process, parameters of an input layer and a convolutional layer in the CNN-seq2seq-Adaboost model are kept unchanged;
third, game stage
Collecting the change of radio frequency signals caused by game actions in a game stage to obtain game action data; interrupting and labeling game action data according to the meta-action;
extracting characteristics of the game action data subjected to interruption and label marking, and generating action identification data based on the extracted characteristics;
inputting the generated motion recognition data into a trained CNN-seq2seq-Adaboost model, and outputting a game motion recognition result by the CNN-seq2seq-Adaboost model;
in the development stage of the game software, the interrupting and labeling of the complex action data according to the meta-action comprises the following steps:
dividing the complex action data into a plurality of meta-action data segments according to meta-actions, and respectively marking corresponding meta-action data segment labels, wherein the meta-action data segment labels are as follows: location, sub-action, wherein location represents the position of the meta-action data segment in the complex action data, sub-action represents the meta-action category tag of the meta-action data segment, and action represents the category to which the complex action belongs;
in the game stage, the interrupting and labeling of the game action data according to the meta-action comprises: dividing game action data into a plurality of meta-action data segments according to meta-actions, and respectively marking corresponding meta-action data segment labels, wherein the meta-action data segment labels are as follows: [l,s,a]WhereinlIndicating the position of the piece of meta motion data in the game motion data,sa meta-action category tag indicating the piece of meta-action data,aindicating a category to which the game action belongs;
the feature extraction of the complex action data for performing interruption and label marking in the game software development stage comprises the following steps: extracting signal intensity characteristics and arrival angle characteristics in each meta-action data segment, constructing a two-dimensional characteristic matrix according to the signal intensity characteristics and arrival angle characteristics of each meta-action data segment, and constructing elements in a complex action training data set by the two-dimensional characteristic matrices of all the meta-action data segments of the complex action data; the two-dimensional characteristic matrix of each meta-motion data segment is associated with the tag of the meta-motion data segment;
the feature extraction of game action data for performing interruption and label marking in the game stage comprises the following steps: extracting the signal intensity characteristic and the arrival angle characteristic in each section of meta-action data, constructing a two-dimensional characteristic matrix according to the signal intensity characteristic and the arrival angle characteristic of each section of meta-action data, and constructing action identification data by the two-dimensional characteristic matrix of all the meta-action data sections of the game action data;
the training of the constructed CNN-seq2seq-Adaboost model by using the complex motion training data set comprises the following steps:
1) training the CNN part in the CNN-seq2seq-Adaboost model:
taking a two-dimensional characteristic matrix of each meta-motion data segment in the complex motion training data set as the input of a CNN part, and outputting the prediction category and the probability of the meta-motion data segment by the CNN part;
updating parameters of other layers except the input layer and the convolution layer in the CNN part by utilizing gradient back propagation;
2) training the seq2seq part in the CNN-seq2seq-Adaboost model:
the meta-action class label sequence [ sub-action1, sub-action2, … ] in the complex action training data set]As input, a one-dimensional matrix, and a complex sequence of motion labels [ action1，action2，action3，…]Training as a training target input seq2seq part;
the seq2seq part outputs a new meta-action type label sequence and a complex action label thereof;
updating parameters of the seq2seq part using gradient back propagation;
3) training the Adaboost part in the CNN-seq2seq-Adaboost model:
the meta-motion class label sequence output by the seq2seq part is used as a new label sequence and a complex motion label [ action i ]Together as a training set, the Adaboost part is trained and the recognition accuracy of the complex motion is output.
2. The motion sensing game motion recognition method of multi-stage joint training according to claim 1, wherein: and after the metadata action data, the complex action data and the game action data are obtained through collection, filtering and denoising are respectively carried out on the metadata action data, the complex action data and the game action data.
3. The motion sensing game motion recognition method of multi-stage joint training according to claim 2, wherein: in the development stage of the game host, the feature extraction is carried out on the meta-motion data of a series of meta-motions, and the feature extraction comprises the following steps: and extracting the signal intensity characteristic and the arrival angle characteristic in the meta-action data of each meta-action, constructing a characteristic matrix according to the signal intensity characteristic and the arrival angle characteristic of each meta-action, and marking a corresponding meta-action category label for the characteristic matrix.
4. The motion sensing game motion recognition method of multi-stage joint training according to claim 1, wherein: the Adaboost part is obtained by weighted average of a plurality of weak classifiers which are CART and used as a strong classifier.
5. The motion recognition method for a multi-stage co-training motion-sensing game according to any one of claims 1 to 4, wherein: the CNN network and the CNN part in the CNN-seq2seq-Adaboost model have the same structure and respectively comprise a 1-layer input layer, a 3-layer convolutional layer, a 1-layer average pooling layer, a 2-layer full-connection layer and a 1-layer output layer which are sequentially connected.
6. The motion recognition method for a multi-stage co-training motion-sensing game according to any one of claims 1 to 4, wherein: in the game stage, before the change of the radio frequency signal caused by the game action in the game stage is collected, the operation of data reconstruction is carried out:
collecting radio frequency signal changes caused by making a meta-action in the same game environment to obtain meta-action data in the game environment;
extracting the characteristics of the metadata under the game environment;
comparing the extracted characteristics of the meta-motion data under the game environment with the extracted characteristics of the meta-motion data of the same type of meta-motion in the meta-motion training data set, and determining a ratio;
and adjusting the subsequently obtained action identification data by using the determined ratio.
7. A motion recognition system for a multi-stage joint training motion sensing game, which applies the motion recognition method for a multi-stage joint training motion sensing game according to any one of claims 1 to 6, characterized in that: comprises a first-stage training module, a second-stage training module and a game host;
the first-stage training module is used for executing the training step of the game host development stage;
the second stage training module is used for executing the training step of the game software development stage;
the game host is used for executing the steps of the game stage.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111571304.XA CN113952731B (en) | 2021-12-21 | 2021-12-21 | Motion sensing game action recognition method and system based on multi-stage joint training |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111571304.XA CN113952731B (en) | 2021-12-21 | 2021-12-21 | Motion sensing game action recognition method and system based on multi-stage joint training |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113952731A CN113952731A (en) | 2022-01-21 |
| CN113952731B true CN113952731B (en) | 2022-03-22 |
Family
ID=79473478
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111571304.XA Active CN113952731B (en) | 2021-12-21 | 2021-12-21 | Motion sensing game action recognition method and system based on multi-stage joint training |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113952731B (en) |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113476833A (en) * | 2021-06-10 | 2021-10-08 | 深圳市腾讯网域计算机网络有限公司 | Game action recognition method and device, electronic equipment and storage medium |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107291232A (en) * | 2017-06-20 | 2017-10-24 | 深圳市泽科科技有限公司 | A kind of somatic sensation television game exchange method and system based on deep learning and big data |
| US10977872B2 (en) * | 2018-10-31 | 2021-04-13 | Sony Interactive Entertainment Inc. | Graphical style modification for video games using machine learning |
| CN110852436B (en) * | 2019-10-18 | 2023-08-01 | 桂林力港网络科技股份有限公司 | Data processing method, device and storage medium for electronic poker game |
| KR102263438B1 (en) * | 2020-02-06 | 2021-06-10 | 엔에이치엔 주식회사 | Method and apparatus for predicting game difficulty by using deep-learning based game play server |
| CN111598998B (en) * | 2020-05-13 | 2023-11-07 | 腾讯科技（深圳）有限公司 | Three-dimensional virtual model reconstruction method, three-dimensional virtual model reconstruction device, computer equipment and storage medium |
-
2021
- 2021-12-21 CN CN202111571304.XA patent/CN113952731B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113476833A (en) * | 2021-06-10 | 2021-10-08 | 深圳市腾讯网域计算机网络有限公司 | Game action recognition method and device, electronic equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113952731A (en) | 2022-01-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110428428B (en) | Image semantic segmentation method, electronic equipment and readable storage medium | |
| Girdhar et al. | Actionvlad: Learning spatio-temporal aggregation for action classification | |
| Li et al. | Deep reinforcement learning for robust emotional classification in facial expression recognition | |
| de la Cruz et al. | Pre-training with non-expert human demonstration for deep reinforcement learning | |
| CN109891897A (en) | Method for analyzing media content | |
| CN108062505A (en) | Motion detection based on neutral net | |
| Wu et al. | Convolutional reconstruction-to-sequence for video captioning | |
| EP3635590A1 (en) | Systems and methods of prediction of injury risk with a training regime | |
| CN112149602B (en) | Action counting method and device, electronic equipment and storage medium | |
| CN108229300A (en) | Video classification methods, device, computer readable storage medium and electronic equipment | |
| Nurhadiyatna | Deep learning for handwritten Javanese character recognition | |
| Hu et al. | Quantized STDP-based online-learning spiking neural network | |
| Li et al. | Deep representation via convolutional neural network for classification of spatiotemporal event streams | |
| US20210245005A1 (en) | Implementation of machine learning for skill-improvement through cloud computing and method therefor | |
| CN113569805A (en) | Action recognition method and device, electronic equipment and storage medium | |
| Dong et al. | Research on image classification based on capsnet | |
| Wei et al. | Learning facial expression and body gesture visual information for video emotion recognition | |
| CN117237756A (en) | Method for training target segmentation model, target segmentation method and related device | |
| Yao et al. | [Retracted] Facial Expression Recognition Based on Convolutional Neural Network Fusion SIFT Features of Mobile Virtual Reality | |
| CN114266927A (en) | Unsupervised saliency target detection method, system, equipment and medium | |
| Aghaei et al. | Sparse deep LSTMs with convolutional attention for human action recognition | |
| Dey et al. | Umpire’s Signal Recognition in Cricket Using an Attention based DC-GRU Network | |
| CN111860056A (en) | Blink-based in-vivo detection method and device, readable storage medium and equipment | |
| CN113952731B (en) | Motion sensing game action recognition method and system based on multi-stage joint training | |
| Abdullah et al. | Recognition and Evaluation of Stability Movements in Gymnastics Based on Deep Learning |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| PE01 | Entry into force of the registration of the contract for pledge of patent right |
Denomination of invention: A multi-stage joint training method and system for motion recognition in somatosensory games Effective date of registration: 20230825 Granted publication date: 20220322 Pledgee: Guangzhou Rural Commercial Bank Co.,Ltd. Huaxia Branch Pledgor: Guangzhou youkegu Technology Co.,Ltd. Registration number: Y2023980053830 |
|
| PE01 | Entry into force of the registration of the contract for pledge of patent right |