CN106203363A - Human skeleton motion sequence Activity recognition method - Google Patents
Human skeleton motion sequence Activity recognition method Download PDFInfo
- Publication number
- CN106203363A CN106203363A CN201610562181.6A CN201610562181A CN106203363A CN 106203363 A CN106203363 A CN 106203363A CN 201610562181 A CN201610562181 A CN 201610562181A CN 106203363 A CN106203363 A CN 106203363A
- Authority
- CN
- China
- Prior art keywords
- human skeleton
- sequence
- expressed
- motion
- extremity
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 62
- 230000000694 effects Effects 0.000 title claims abstract description 19
- 238000010606 normalization Methods 0.000 claims abstract description 24
- 238000013527 convolutional neural network Methods 0.000 claims abstract description 10
- 210000003414 extremity Anatomy 0.000 claims description 40
- 238000012549 training Methods 0.000 claims description 8
- 238000000605 extraction Methods 0.000 claims description 7
- 238000007476 Maximum Likelihood Methods 0.000 claims description 6
- 210000003141 lower extremity Anatomy 0.000 claims description 6
- 239000011159 matrix material Substances 0.000 claims description 5
- 238000013528 artificial neural network Methods 0.000 claims description 4
- 230000009467 reduction Effects 0.000 claims description 2
- 210000000988 bone and bone Anatomy 0.000 claims 1
- 230000006399 behavior Effects 0.000 description 30
- 230000008569 process Effects 0.000 description 13
- 238000012360 testing method Methods 0.000 description 7
- 230000006870 function Effects 0.000 description 6
- 238000004590 computer program Methods 0.000 description 5
- 230000008859 change Effects 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 4
- 230000003044 adaptive effect Effects 0.000 description 2
- 238000002474 experimental method Methods 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 238000012544 monitoring process Methods 0.000 description 2
- 238000011176 pooling Methods 0.000 description 2
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000015572 biosynthetic process Effects 0.000 description 1
- 239000003086 colorant Substances 0.000 description 1
- 239000012141 concentrate Substances 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000002372 labelling Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000003909 pattern recognition Methods 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
- 238000010008 shearing Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
- G06V40/23—Recognition of whole body movements, e.g. for sport training
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/42—Global feature extraction by analysis of the whole pattern, e.g. using frequency domain transformations or autocorrelation
- G06V10/422—Global feature extraction by analysis of the whole pattern, e.g. using frequency domain transformations or autocorrelation for representing the structure of the pattern or shape of an object therefor
Abstract
The invention discloses a kind of human skeleton motion sequence Activity recognition method.Wherein, the method includes obtaining human skeleton node coordinate；Being concatenated by node corresponding for each for human skeleton limbs, the motion feature forming extremity and trunk is expressed；The motion feature of extremity and trunk is expressed and concatenates, form the vector expression of human skeleton；By the vector expression corresponding to frame each in human skeleton motion sequence, arrange sequentially in time, obtain three-dimensional matrice；Numerical value in three-dimensional matrice is done normalization and dimension normalization, obtains the image expression that human skeleton sequence pair is answered；The textural characteristics using convolutional neural networks to extract in image expression adaptively is expressed；Express based on this textural characteristics and carry out behavior kind judging, and determine the behavior classification belonging to human skeleton sequence in the way of ballot.The behavior of people, without complicated data prediction, accurately can be identified by the embodiment of the present invention according to human skeleton coordinate sequence.
Description
Technical field
The present embodiments relate to computer vision, pattern recognition and degree of depth learning art field, be specifically related to a kind of people
Body skeleton motion sequence Activity recognition method.
Background technology
The recovery of neural network theory has promoted developing rapidly of artificial intelligence technology.Society, intelligent robot, nothing
People drives a car etc. and will enter into the life of people.Intelligent transportation, intelligent video monitoring and smart city etc. are required for computer
Behavior to people automatically analyzes.Currently, the human skeleton algorithm for estimating of depth camera machine technology combined high precision, Ke Yizhi
Connect the frame sequence providing human motion process corresponding, the behavior of people can be identified accurately based on this frame sequence.
Traditional Activity recognition algorithm based on human skeleton sequence is mainly encoded on the basis of manual feature extraction
Rear design grader realizes behavior classification, and the extraction process of manual feature is relatively complicated, and its with feature coding subsequently and point
Class process is generally separated and carries out, though composition system can be cascaded, but is unfavorable for actual application because of inefficient.Additionally, traditional method
Training and test are typically to carry out in small data set, and when data volume increases, model computation complexity is for general hard
Part condition is difficult to bear, and is difficult to play a role in actual applications.
In view of this, the special proposition present invention.
Summary of the invention
In view of the above problems, it is proposed that the present invention is to provide a kind of a kind of human body solving the problems referred to above at least in part
Skeleton motion sequence Activity recognition method.
To achieve these goals, according to an aspect of the invention, it is provided techniques below scheme:
A kind of human skeleton motion sequence Activity recognition method, the method includes:
Obtain described human skeleton node coordinate；Wherein, human skeleton described in a series of continuous moment constitutes described human body
Skeleton motion sequence；
Being concatenated by node corresponding for each for described human skeleton limbs, the motion feature forming extremity and trunk is expressed；
The motion feature of described extremity and described trunk is expressed and concatenates, form the vector expression of human skeleton；
Described vector corresponding to each frame in described human skeleton motion sequence is expressed, arranges sequentially in time,
To three-dimensional matrice；
Numerical value in described three-dimensional matrice is done normalization and dimension normalization, obtains the image that human skeleton sequence pair is answered
Express；
The textural characteristics using convolutional neural networks to extract in described image expression adaptively is expressed；
Described textural characteristics is expressed and is mapped as contained behavior class in dimension and data base by full Connection Neural Network layer
The most total identical vector, and by soft maximization layer normalization, determine the generic probability of described human skeleton；
Minimize following maximum likelihood loss function:
Wherein, described M represents the total amount of the described human skeleton motion sequence samples in data base；Described δ () represents
Kronecker function；Described C represents behavior classification sum；Described k represents behavior item number；Described p (Ck|xm) represent described m
Individual human skeleton motion sequence samples xmIt is under the jurisdiction of kth behavior CkGeneric probability；Described r is human skeleton motion sequence sample
This xmCorresponding real behavior category label；Described L (Ω) represents maximum likelihood loss function；
The image expression answered from described human skeleton sequence pair and flipped image thereof determine four folding corner regions and center
Region, carries out behavior kind judging respectively, and determines the behavior classification belonging to described human skeleton sequence in the way of ballot.
Preferably, method according to claim 1, it is characterised in that described acquisition described human skeleton node is sat
Mark specifically includes:
Described human skeleton motion sequence is obtained by depth camera or motion capture system；
According to human body physical arrangement, single moment human skeleton coordinate is divided into extremity and torso portion；
For described extremity and described torso portion, according to each node physical connection order respectively to three coordinates of x, y, z
Component arranges, and forms described extremity and described trunk and expresses at y-z, z-x, the vector of x-y plane projection, thus
To described human skeleton node coordinate.
Preferably, described node corresponding for each for described human skeleton limbs is concatenated, form extremity and the fortune of trunk
Dynamic feature representation, specifically includes:
According to left arm, right arm, trunk, left lower limb, the order of right lower limb, described vector is expressed and concatenates, thus shape
The motion feature becoming described extremity and described trunk is expressed.
Preferably, described expression by the motion feature of described extremity and described trunk concatenates, and forms human skeleton
Vector is expressed, and specifically includes:
The motion feature of described extremity and described trunk is expressed and is sequentially arranged, form the institute of described human skeleton
State vector expression.
Preferably, described numerical value in described three-dimensional matrice is done normalization and dimension normalization, obtain human skeleton sequence
The image expression that row are corresponding, specifically includes:
To the numerical value in described three-dimensional matrice and dimension, it is normalized operation according to below equation:
Wherein, the pixel value during described p represents the image expression that described human skeleton sequence pair is answered；Described cmaxWith described
cminRepresent the maximum and minimum value of all human skeleton node coordinates in training set respectively；Described floor represents and rounds downwards letter
Number.
Preferably, described method also includes:
During textural characteristics in the described image expression of described extraction is expressed, use and maximize pond method reduction
Convolution output dimension.
Preferably, described method also includes:
The described image expression answering described human skeleton sequence pair is normalized.
To achieve these goals, according to another aspect of the present invention, a kind of human skeleton motion sequence is additionally provided
Activity recognition method, described method includes:
Obtain the node coordinate of human skeleton sequence to be analyzed；
Node corresponding for the described each limbs of human skeleton to be analyzed is concatenated, forms extremity and trunk motion feature table
Reach；
The motion feature of described extremity and described trunk is expressed and concatenates, formed described human skeleton to be analyzed to
Scale reaches；
Described vector corresponding to each frame in described human skeleton motion sequence to be analyzed is expressed, arranges sequentially in time
Row, obtain three-dimensional matrice；
Numerical value in described three-dimensional matrice is done normalization and dimension normalization, obtains described human skeleton sequence to be analyzed
Corresponding image expression；
Four folding corner regions and central area is determined from described image expression and flipped image thereof；Said method is utilized to instruct
Described in the model extraction practiced, the textural characteristics of four folding corner regions and described central area is expressed, and based on described textural characteristics table
Reach, in the way of ballot, determine the behavior classification belonging to described human skeleton sequence to be analyzed.
Compared with prior art, technique scheme at least has the advantages that
The embodiment of the present invention by provide one the most simply, in high precision, high efficiency human body skeleton motion sequence
Activity recognition method.Human skeleton sequence is converted into according to certain rule the image expression of correspondence.Then, after based on converting
Image expression architectural feature when utilizing its textural characteristics of convolutional neural networks model extraction indirectly to obtain frame sequence empty,
And classify, thus the behavior answering raw skeleton sequence pair is identified, it is not necessary to complicated data prediction, can be according to human body
The behavior of people is identified by skeleton coordinate sequence.
The embodiment of the present invention has weight in fields such as intelligent video monitoring, robot vision, man-machine interaction and game controls
Want using value.
Accompanying drawing explanation
Accompanying drawing, as the part of the present invention, is used for providing further understanding of the invention, and the present invention's is schematic
Embodiment and explanation thereof are used for explaining the present invention, but do not constitute inappropriate limitation of the present invention.Obviously, the accompanying drawing in describing below
It is only some embodiments, to those skilled in the art, on the premise of not paying creative work, it is also possible to
Other accompanying drawings are obtained according to these accompanying drawings.In the accompanying drawings:
Fig. 1 is the training method according to the human skeleton motion sequence Activity recognition model shown in an exemplary embodiment
Schematic flow sheet；
Fig. 2 is to concatenate formation four according to each for human skeleton limbs corresponding node being carried out shown in another exemplary embodiment
The schematic diagram that limb and trunk motion feature are expressed；
Fig. 3 is according to obtaining, based on convolutional neural networks, the figure that human skeleton sequence pair is answered shown in an exemplary embodiment
As the schematic diagram expressed；
Fig. 4 is according to synchronizing pond schematic diagram during empty shown in an exemplary embodiment；
Fig. 5 is according to the hierarchical space-time adaptive feature learning schematic diagram shown in an exemplary embodiment；
Fig. 6 is to know according to the human skeleton motion sequence behavior based on convolutional neural networks shown in an exemplary embodiment
The schematic flow sheet of other method.
These accompanying drawings and word describe and are not intended as limiting by any way the concept of the present invention, but pass through reference
Specific embodiment is that those skilled in the art illustrate idea of the invention.
Detailed description of the invention
Below in conjunction with the accompanying drawings and the specific embodiment technical side that the embodiment of the present invention solved the technical problem that, is used
The technique effect of case and realization carries out clear, complete description.Obviously, described embodiment is only of the application
Divide embodiment, be not whole embodiments.Based on the embodiment in the application, those of ordinary skill in the art are not paying creation
Property work on the premise of, the embodiment of other equivalents all of being obtained or substantially modification all falls within protection scope of the present invention.
It should be noted that in the following description, understand for convenience, give many details.But it is the brightest
Aobvious, the realization of the present invention can not have these details.
Also, it should be noted the most clearly limiting or in the case of not conflicting, each embodiment in the present invention and
Technical characteristic therein can be mutually combined and form technical scheme.
In recent years, depth camera rapid technological improvement, combine high efficiency based on its acquired range image sequence
Attitude estimation algorithm, can effectively obtain human skeleton motion sequence, can realize high-precision behavior based on these sequences
Identify.
To this end, the embodiment of the present invention proposes a kind of human skeleton motion sequence Activity recognition method.As it is shown in figure 1, the party
Method may include that
S100: obtain human skeleton node coordinate.Wherein, the human skeleton in a series of continuous moment constitutes human skeleton fortune
Dynamic sequence；
In an optional embodiment, this step can be able to be got by depth camera or motion capture system
Human skeleton motion sequence.Human skeleton motion sequence includes the human skeleton in a series of continuous moment.Then, according to human body thing
Reason structure, is divided into five parts by single moment human skeleton coordinate according to extremity and trunk, and to each several part according to each node
Three coordinate components of x, y, z are arranged by physical connection order respectively, form extremity and trunk three coordinate plane projection
Vector expression.
S110: concatenated by node corresponding for each for human skeleton limbs, forms extremity and trunk motion feature is expressed.
In an optional embodiment, this step can according to left arm, right arm, trunk, left lower limb, right lower limb suitable
Extremity and trunk are concatenated in the vector expression of three coordinate plane projection, thus obtain human skeleton at y-by sequence respectively
The vector of tri-coordinate plane projection of z, z-x, x-y expresses (i.e. motion feature expression).
S120: extremity and trunk motion feature are expressed and concatenate, forms the vector expression of human skeleton.
In an optional embodiment, extremity and trunk motion feature are expressed and is sequentially arranged, form human body
The vector expression of skeleton.
In actual applications, can be respectively at tri-coordinate planes of y-z, z-x, x-y, by time each for human skeleton motion sequence
The vector expression carving human skeleton attitude corresponding is sequentially arranged, and (it is floating-point to constitute three two-dimensional matrixs of red, green, blue
Matrix), as shown in Figure 2.Wherein, the longitudinal direction (arranging) of each matrix represents that each node of single width skeleton is in space y-z, z-x, x-y plane
On the vector expression of projection, laterally (OK) then represents each projection coordinate rule over time, is that single skeleton node exists
Coordinate figure the most in the same time.
S130: by the vector expression corresponding to frame each in human skeleton motion sequence, arrange sequentially in time, obtain three
Dimension matrix.
Wherein, will by human skeleton motion sequence each moment skeleton pose corresponding vector express be sequentially arranged and
Three two-dimensional matrixs (being respectively seen as the RGB three primary colours of image) constituted are stacked into a three-dimensional matrice.These three two-dimensional matrix
Three passages respectively as RGB color image.
S140: the numerical value in three-dimensional matrice is done normalization and dimension normalization, obtains the figure that human skeleton sequence pair is answered
As expressing.
In this step, it is normalized operation according to below equation:
Wherein, p represents the pixel value in the image expression that human skeleton sequence pair is answered, cmaxAnd cminRepresent training set respectively
In the maximum and minimum value of all human skeleton node coordinates, floor represents downward bracket function.
Operated by above-mentioned normalization so that image pixel Distribution value, between 0-255, i.e. obtains human skeleton sequence
The image expression that (namely raw skeleton sequence) is corresponding, as shown in Figure 3.
S150: use convolutional neural networks to extract the texture in the image expression that human skeleton sequence pair is answered adaptively
Feature representation.
In this step, maximization pond (max-pooling) can be used to reduce convolution output dimension, with place simultaneously
Redundant data in reason image, can reduce motion frequency simultaneously and change the impact causing accuracy of identification.Due to input picture
Longitudinally reflection configuration space architectural feature, the most then embody time-varying multidate information, make maximization pond (max-pooling) operate
Scale invariability show as selecting the articulare of more distinction in the vertical, the most then the frequency showing as motion is constant
Property, synchronize pond (maximizing pond) when being sky, as shown in Figure 4.Motion frequency is solved by synchronizing pond operation during sky
The problem of rate change.
Fig. 5 schematically illustrates hierarchical space-time adaptive feature learning process.Specific implementation process can be adopted
With the convolution filter shown in Fig. 5, to carry out two-dimensional convolution operation.
The architectural feature when embodiment of the present invention obtains human skeleton sequence empty indirectly by texture feature extraction information.
The information representation when textural characteristics obtained in this step is expressed namely be empty.
S160: textural characteristics is expressed and is mapped as contained behavior class in dimension and data base by full Connection Neural Network layer
The most total identical vector, and by soft maximization layer normalization, determine the generic probability of human skeleton.
In this step, data base, for example, it may be Berkeley disclosed in Univ California-Berkeley
MHAD data base, the MSR Action3D data base of Microsoft or ChaLearn Gesture Recognition data base.Wherein,
Berkeley MHAD data base is gathered by motion capture system, containing 659 sequences, and totally 11 behavior classifications, frame of video
Rate is that 480 frames are per second, and its human skeleton provided contains 35 nodes.MSR Action3D data base is by class in early days
Kinect device gathers, and frame per second is that 15 frames are per second, totally 557 behavior sequences, 20 behavior classifications, its human skeleton provided
Containing 20 nodes.ChaLearn Gesture Recognition data base comprises the Kinect data of 23 hours, totally 20
Italian gestures classification, it is provided that human skeleton comprise 20 nodes, this data base is multi-modal data storehouse, carries simultaneously
Supply data sequence and the human skeleton sequence of Kinect output after rgb video, range image sequence, foreground segmentation.
The human skeleton sequence i.e. obtaining being currently entered after vector value normalization after mapping is under the jurisdiction of each behavior class
Other probability, then after soft maximization (Softmax) layer normalization, i.e. can get the generic probability of list entries.So, output
Vector dimension is identical with behavior classification number.
Specifically, the generic probability of soft maximization layer output can determine according to below equation:
Wherein, p (Ck) represent generic probability；Ok、OiRepresent kth and i-th element that full articulamentum exports respectively；C represents
Behavior classification sum.
List entries generic can be judged according to the maximum of generic probability.
S170: minimize following maximum likelihood loss function:
Wherein, the total amount of the human skeleton motion sequence samples during M represents data base；δ () represents Kronecker letter
Number；C represents behavior classification sum；K represents behavior item number；p(Ck|xm) represent m-th human skeleton motion sequence samples xmIt is subordinate to
In kth behavior CkGeneric probability；R is human skeleton motion sequence samples xmCorresponding real behavior category label；L (Ω) table
Show maximum likelihood loss function.
The reverse procedure of training uses BPTT (Back-Propagation Through Time) algorithm to carry out.
S180: select four corners and center the image expression answered from human skeleton sequence pair and flipped image thereof
Territory, carries out behavior kind judging respectively, and determines the behavior classification belonging to human skeleton sequence in the way of ballot.
Below by the Sequence Transformed rear image dimension of human skeleton for illustrating as a example by 60x 60.It is cut out from image
The image of 52x 52 is used for testing, respectively with input picture coordinate points (0,0), (0,8), (8,0), (8,8), (4,4) for shearing
The top left corner apex of image, shears the image of 5 52x 52 and respective flipped image totally 10 images are used for testing, last root
Vote according to the test result of these 10 images, to judge the behavior classification belonging to list entries.
Owing to the length of human skeleton sequence is different, then the image table that the human skeleton sequence pair obtained by step S140 is answered
Reach ground width the most different.
In order to unify yardstick, in order to subsequent treatment, in a preferred embodiment, after step s 140, also may be used
To include that the image expression answering human skeleton sequence pair is normalized.
The embodiment of the present invention also proposes a kind of human skeleton motion sequence Activity recognition method.As shown in Figure 6, the method can
To include:
S200: obtain the node coordinate of human skeleton sequence to be analyzed.
Wherein, the acquisition methods that this step relates to is referred to being embodied as of step S100 in above-described embodiment
Journey, does not repeats them here.
S210: the node being analysed to each limbs of human skeleton corresponding concatenates, forms extremity and trunk motion feature
Express.
Specific implementation process about this step is referred to the explanation of step S110 in above-described embodiment, the most superfluous at this
State.
S220: expressed by the motion feature of extremity and trunk and concatenate, forms the vector expression of human skeleton to be analyzed.
Wherein, the specific implementation process about this step is referred to the explanation of step S120 in above-described embodiment, at this
Repeat no more.
S230: be analysed to the vector expression corresponding to each frame in human skeleton motion sequence, arrange sequentially in time,
Obtain three-dimensional matrice.
Wherein, the specific implementation process about this step is referred to the explanation of step S130 in above-described embodiment, at this
Repeat no more.
S240: the numerical value in three-dimensional matrice does normalization and dimension normalization, obtains human skeleton sequence pair to be analyzed
The image expression answered.
This step is Sequence Transformed for corresponding image expression by being analysed to human skeleton, it is achieved that be analysed to human body
The static structure information that during empty in frame sequence, multidate information is converted in image.
Specific implementation process about this step is referred to the explanation of step S140 in above-described embodiment, the most superfluous at this
State.
S250: determine four folding corner regions the image expression answered from human skeleton sequence pair to be analyzed and flipped image thereof
And central area.
S260: utilize above-mentioned human skeleton motion sequence Activity recognition method to extract four folding corner regions and central area
Textural characteristics is expressed, and expresses based on this textural characteristics, determines the row belonging to human skeleton sequence to be analyzed in the way of ballot
For classification.
In this step, a human skeleton motion sequence Activity recognition mould can be built by step S100 to step S180
Type, it can be convolutional neural networks model, utilizes this model (the most hierarchical bank of filters) to extract four comer area
The textural characteristics of territory and central area image is expressed, and indirectly realizes human skeleton sequence institute to be analyzed based on the expression obtained
The identification of genus behavior.
Effectiveness of the invention is verified below by experimental result.
Experiment is carried out on the public data storehouse of three standards, and it is disclosed in Univ California-Berkeley respectively
Berkeley MHAD data base, the MSR Action3D data base of Microsoft, and the most challenging ChaLearn Gesture
Recognition data base.Data base's entirety is divided into training set, checking collects and test set three part, totally 955 videos,
Each video duration 1-2 minute.
Table one schematically illustrates the experimental result on Berkeley MHAD data base.
Table one:
Table two schematically illustrates the experimental result on MSR Action3D data base.
Table two:
Table three schematically illustrates the experimental result on ChaLearn Gesture Recognition data base.
Table three:
Come this model is calculated effect as a example by the experiment on ChaLearn gesture recognition dataset
Rate is analyzed, and wherein, F1-score, also known as balance F mark, is the harmonic mean of accuracy rate and recall rate.The present invention implements
Example realizes based on the convolutional neural networks code ConvNet that increases income.Running on video card with GPU code, training process processes
One sequence probably needs 1.95ms, and test process uses the mode of ballot, general 2.27ms/sequence.This efficiency has been
Entirely can meet real-time application demand.
Totally test result indicate that, the method discrimination on above three public data storehouse has all reached the most high-precision
Degree, and this model manipulation is simple, has the highest computational efficiency, it is simple to actual application.
Although in above-described embodiment, each step is described according to the mode of above-mentioned precedence, but this area
Those of skill will appreciate that, in order to realize the effect of the present embodiment, perform not necessarily in such order between different steps,
It can simultaneously (parallel) perform or perform with reverse order, these simply change all protection scope of the present invention it
In.
The technical scheme provided the embodiment of the present invention above is described in detail.Although applying concrete herein
Individual example principle and the embodiment of the present invention are set forth, but, the explanation of above-described embodiment be only applicable to help reason
Solve the principle of the embodiment of the present invention；For those skilled in the art, according to the embodiment of the present invention, it is being embodied as
All can make a change within mode and range of application.
It should be noted that referred to herein to flow chart be not limited solely to form shown in this article, it is all right
Carry out other to divide and/or combination.
It can further be stated that: labelling and word in accompanying drawing are intended merely to be illustrated more clearly that the present invention, and it is right to be not intended as
The improper restriction of scope.
Term " includes ", " comprising " or any other like term are intended to comprising of nonexcludability, so that
Process, method, article or equipment/device including a series of key elements not only include those key elements, but also include the brightest
Other key element really listed, or also include the key element that these processes, method, article or equipment/device are intrinsic.
Each step of the present invention can realize with general calculating device, and such as, they can concentrate on single
Calculate on device, such as: personal computer, server computer, handheld device or portable set, laptop device or many
Processor device, it is also possible to be distributed on the network that multiple calculating device is formed, they can be to be different from order herein
Step shown or described by execution, or they are fabricated to respectively each integrated circuit modules, or by many in them
Individual module or step are fabricated to single integrated circuit module and realize.Therefore, the invention is not restricted to any specific hardware and soft
Part or its combination.
The method that the present invention provides can use PLD to realize, it is also possible to is embodied as computer program soft
Part or program module (it include performing particular task or realize the routine of particular abstract data type, program, object, assembly or
Data structure etc.), can be such as a kind of computer program according to embodiments of the invention, run this computer program
Product makes computer perform for the method demonstrated.Described computer program includes computer-readable recording medium, should
Comprise computer program logic or code section on medium, be used for realizing described method.Described computer-readable recording medium can
To be the built-in medium being mounted in a computer or the removable medium (example that can disassemble from basic computer
As: use the storage device of hot plug technology).Described built-in medium includes but not limited to rewritable nonvolatile memory,
Such as: RAM, ROM, flash memory and hard disk.Described removable medium includes but not limited to: optical storage media is (such as: CD-
ROM and DVD), magnetic-optical storage medium (such as: MO), magnetic storage medium (such as: tape or portable hard drive), have built-in can
Rewrite the media (such as: storage card) of nonvolatile memory and there are the media (such as: ROM box) of built-in ROM.
Particular embodiments described above, has been carried out the purpose of the present invention, technical scheme and beneficial effect the most in detail
Describe in detail bright, be it should be understood that the specific embodiment that the foregoing is only the present invention, be not limited to the present invention, all
Within the spirit and principles in the present invention, any modification, equivalent substitution and improvement etc. done, should be included in the guarantor of the present invention
Within the scope of protecting.
Claims (8)
1. a human skeleton motion sequence Activity recognition method, it is characterised in that described method at least includes:
Obtain described human skeleton node coordinate；Wherein, the described human skeleton in a series of continuous moment constitutes described human bone
Frame motion sequence；
Being concatenated by node corresponding for each for described human skeleton limbs, the motion feature forming extremity and trunk is expressed；
The motion feature of described extremity and described trunk is expressed and concatenates, form the vector expression of human skeleton；
Described vector corresponding to each frame in described human skeleton motion sequence is expressed, arranges sequentially in time, obtain three
Dimension matrix；
Numerical value in described three-dimensional matrice is done normalization and dimension normalization, obtains the image table that human skeleton sequence pair is answered
Reach；
The textural characteristics using convolutional neural networks to extract in described image expression adaptively is expressed；
Described textural characteristics is expressed by full Connection Neural Network layer be mapped as in dimension and data base contained by behavior classification total
The vector that number is identical, and by soft maximization layer normalization, determine the generic probability of described human skeleton；
Minimize following maximum likelihood loss function:
Wherein, described M represents the total amount of the described human skeleton motion sequence samples in data base；Described δ () represents
Kronecker function；Described C represents behavior classification sum；Described k represents behavior item number；Described p (Ck|xm) represent described m
Individual human skeleton motion sequence samples xmIt is under the jurisdiction of kth behavior CkGeneric probability；Described r is human skeleton motion sequence sample
This xmCorresponding real behavior category label；Described L (Ω) represents maximum likelihood loss function；
The image expression answered from described human skeleton sequence pair and flipped image thereof determine four folding corner regions and central area,
Carry out behavior kind judging respectively, and determine the behavior classification belonging to described human skeleton sequence in the way of ballot.
Method the most according to claim 1, it is characterised in that described acquisition described human skeleton node coordinate specifically wraps
Include:
Described human skeleton motion sequence is obtained by depth camera or motion capture system；
According to human body physical arrangement, single moment human skeleton coordinate is divided into extremity and torso portion；
For described extremity and described torso portion, according to each node physical connection order respectively to three coordinate components of x, y, z
Arrange, form described extremity and described trunk and express at y-z, z-x, the vector of x-y plane projection, thus obtain institute
State human skeleton node coordinate.
Method the most according to claim 2, it is characterised in that described node corresponding for each for described human skeleton limbs is entered
Row concatenation, the motion feature forming extremity and trunk is expressed, and specifically includes:
According to left arm, right arm, trunk, left lower limb, the order of right lower limb, described vector is expressed and concatenates, thus form institute
The motion feature stating extremity and described trunk is expressed.
Method the most according to claim 3, it is characterised in that described by the motion feature table of described extremity and described trunk
Reach and concatenate, form the vector expression of human skeleton, specifically include:
The motion feature of described extremity and described trunk is expressed and is sequentially arranged, formed described human skeleton described to
Scale reaches.
Method the most according to claim 1, it is characterised in that described numerical value in described three-dimensional matrice is done normalization and
Dimension normalization, obtains the image expression that human skeleton sequence pair is answered, specifically includes:
To the numerical value in described three-dimensional matrice and dimension, it is normalized operation according to below equation:
Wherein, the pixel value during described p represents the image expression that described human skeleton sequence pair is answered；Described cmaxWith described cminPoint
Biao Shi the maximum and minimum value of all human skeleton node coordinates in training set；Described floor represents downward bracket function.
Method the most according to claim 1, it is characterised in that described method also includes:
During textural characteristics in the described image expression of described extraction is expressed, use and maximize pond method reduction convolution
Output dimension.
Method the most according to claim 1, it is characterised in that described method also includes:
The described image expression answering described human skeleton sequence pair is normalized.
8. a human skeleton motion sequence Activity recognition method, it is characterised in that described method includes:
Obtain the node coordinate of human skeleton sequence to be analyzed；
Node corresponding for the described each limbs of human skeleton to be analyzed is concatenated, forms extremity and trunk motion feature is expressed；
The motion feature of described extremity and described trunk is expressed and concatenates, form the vector table of described human skeleton to be analyzed
Reach；
Described vector corresponding to each frame in described human skeleton motion sequence to be analyzed is expressed, arranges sequentially in time,
Obtain three-dimensional matrice；
Numerical value in described three-dimensional matrice does normalization and dimension normalization, and obtaining described human skeleton sequence pair to be analyzed should
Image expression；
Four folding corner regions and central area is determined from described image expression and flipped image thereof；
Arbitrary described method in claim 1 to 7 is utilized to extract described four folding corner regions and the texture of described central area
Feature representation, and express based on described textural characteristics, determine belonging to described human skeleton sequence to be analyzed in the way of ballot
Behavior classification.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610562181.6A CN106203363A (en) | 2016-07-15 | 2016-07-15 | Human skeleton motion sequence Activity recognition method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610562181.6A CN106203363A (en) | 2016-07-15 | 2016-07-15 | Human skeleton motion sequence Activity recognition method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN106203363A true CN106203363A (en) | 2016-12-07 |
Family
ID=57475629
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610562181.6A Pending CN106203363A (en) | 2016-07-15 | 2016-07-15 | Human skeleton motion sequence Activity recognition method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106203363A (en) |
Cited By (24)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107203753A (en) * | 2017-05-25 | 2017-09-26 | 西安工业大学 | A kind of action identification method based on fuzzy neural network and graph model reasoning |
| CN107225571A (en) * | 2017-06-07 | 2017-10-03 | 纳恩博（北京）科技有限公司 | Motion planning and robot control method and apparatus, robot |
| CN107392131A (en) * | 2017-07-14 | 2017-11-24 | 天津大学 | A kind of action identification method based on skeleton nodal distance |
| CN108229355A (en) * | 2017-12-22 | 2018-06-29 | 北京市商汤科技开发有限公司 | Activity recognition method and apparatus, electronic equipment, computer storage media, program |
| CN108830215A (en) * | 2018-06-14 | 2018-11-16 | 南京理工大学 | Hazardous act recognition methods based on personnel's framework information |
| CN109145868A (en) * | 2018-09-11 | 2019-01-04 | 广州杰赛科技股份有限公司 | A kind of Activity recognition method and apparatus assisting running training |
| CN109800659A (en) * | 2018-12-26 | 2019-05-24 | 中国科学院自动化研究所南京人工智能芯片创新研究院 | A kind of action identification method and device |
| CN109829422A (en) * | 2019-01-28 | 2019-05-31 | 哈尔滨工业大学 | A kind of video frequency identifying method based on the movement of impulsive neural networks falling over of human body |
| CN110210372A (en) * | 2019-05-29 | 2019-09-06 | 中国科学院自动化研究所 | Based on skeleton Activity recognition method, the system for paying attention to enhancing figure convolutional network |
| CN110222551A (en) * | 2018-03-02 | 2019-09-10 | 杭州海康威视数字技术股份有限公司 | Method, apparatus, electronic equipment and the storage medium of identification maneuver classification |
| CN110215216A (en) * | 2019-06-11 | 2019-09-10 | 中国科学院自动化研究所 | Based on the with different levels Activity recognition method in skeletal joint point subregion, system |
| CN110276768A (en) * | 2019-06-28 | 2019-09-24 | 京东方科技集团股份有限公司 | Image partition method, image segmentation device, image segmentation apparatus and medium |
| CN110309732A (en) * | 2019-06-13 | 2019-10-08 | 浙江大学 | Activity recognition method based on skeleton video |
| CN110348395A (en) * | 2019-07-12 | 2019-10-18 | 电子科技大学 | A kind of skeleton Activity recognition method based on time-space relationship |
| CN110363131A (en) * | 2019-07-08 | 2019-10-22 | 上海交通大学 | Anomaly detection method, system and medium based on human skeleton |
| CN110555349A (en) * | 2018-06-01 | 2019-12-10 | 杭州海康威视数字技术股份有限公司 | working time length statistical method and device |
| CN110765942A (en) * | 2019-10-23 | 2020-02-07 | 睿魔智能科技（深圳）有限公司 | Image data labeling method, device, equipment and storage medium |
| WO2020211242A1 (en) * | 2019-04-15 | 2020-10-22 | 深圳大学 | Behavior recognition-based method, apparatus and storage medium |
| CN111898576A (en) * | 2020-08-06 | 2020-11-06 | 电子科技大学 | Behavior identification method based on human skeleton space-time relationship |
| CN111914807A (en) * | 2020-08-18 | 2020-11-10 | 太原理工大学 | Miner behavior identification method based on sensor and skeleton information |
| CN112381004A (en) * | 2020-11-17 | 2021-02-19 | 华南理工大学 | Framework-based double-flow self-adaptive graph convolution network behavior identification method |
| CN112613436A (en) * | 2020-12-28 | 2021-04-06 | 中国联合网络通信集团有限公司 | Examination cheating detection method and device |
| CN112651360A (en) * | 2020-12-31 | 2021-04-13 | 福州大学 | Skeleton action recognition method under small sample |
| CN112818887A (en) * | 2021-02-08 | 2021-05-18 | 中国科学院自动化研究所 | Human body skeleton sequence behavior identification method based on unsupervised learning |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104615983A (en) * | 2015-01-28 | 2015-05-13 | 中国科学院自动化研究所 | Behavior identification method based on recurrent neural network and human skeleton movement sequences |
| CN104850846A (en) * | 2015-06-02 | 2015-08-19 | 深圳大学 | Human behavior recognition method and human behavior recognition system based on depth neural network |
| CN104899561A (en) * | 2015-05-27 | 2015-09-09 | 华南理工大学 | Parallelized human body behavior identification method |
| CN105320944A (en) * | 2015-10-24 | 2016-02-10 | 西安电子科技大学 | Human body behavior prediction method based on human body skeleton movement information |
-
2016
- 2016-07-15 CN CN201610562181.6A patent/CN106203363A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104615983A (en) * | 2015-01-28 | 2015-05-13 | 中国科学院自动化研究所 | Behavior identification method based on recurrent neural network and human skeleton movement sequences |
| CN104899561A (en) * | 2015-05-27 | 2015-09-09 | 华南理工大学 | Parallelized human body behavior identification method |
| CN104850846A (en) * | 2015-06-02 | 2015-08-19 | 深圳大学 | Human behavior recognition method and human behavior recognition system based on depth neural network |
| CN105320944A (en) * | 2015-10-24 | 2016-02-10 | 西安电子科技大学 | Human body behavior prediction method based on human body skeleton movement information |
Non-Patent Citations (2)
| Title |
|---|
| YONG DU等: "Hierarchical Recurrent Neural Network for Skeleton Based Action Recognition", 《2015 IEEE CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION》 * |
| YONG DU等: "Skeleton based action recognition with convolutional neural network", 《2015 3RD IAPR ASIAN CONFERENCE ON PATTERN RECOGNITION》 * |
Cited By (39)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107203753B (en) * | 2017-05-25 | 2020-09-08 | 西安工业大学 | Action recognition method based on fuzzy neural network and graph model reasoning |
| CN107203753A (en) * | 2017-05-25 | 2017-09-26 | 西安工业大学 | A kind of action identification method based on fuzzy neural network and graph model reasoning |
| CN107225571A (en) * | 2017-06-07 | 2017-10-03 | 纳恩博（北京）科技有限公司 | Motion planning and robot control method and apparatus, robot |
| CN107225571B (en) * | 2017-06-07 | 2020-03-31 | 纳恩博（北京）科技有限公司 | Robot motion control method and device and robot |
| CN107392131A (en) * | 2017-07-14 | 2017-11-24 | 天津大学 | A kind of action identification method based on skeleton nodal distance |
| CN108229355A (en) * | 2017-12-22 | 2018-06-29 | 北京市商汤科技开发有限公司 | Activity recognition method and apparatus, electronic equipment, computer storage media, program |
| CN108229355B (en) * | 2017-12-22 | 2021-03-23 | 北京市商汤科技开发有限公司 | Behavior recognition method and apparatus, electronic device, computer storage medium |
| CN110222551B (en) * | 2018-03-02 | 2021-07-09 | 杭州海康威视数字技术股份有限公司 | Method and device for identifying action type, electronic equipment and storage medium |
| CN110222551A (en) * | 2018-03-02 | 2019-09-10 | 杭州海康威视数字技术股份有限公司 | Method, apparatus, electronic equipment and the storage medium of identification maneuver classification |
| CN110555349A (en) * | 2018-06-01 | 2019-12-10 | 杭州海康威视数字技术股份有限公司 | working time length statistical method and device |
| CN110555349B (en) * | 2018-06-01 | 2023-05-02 | 杭州海康威视数字技术股份有限公司 | Working time length statistics method and device |
| CN108830215A (en) * | 2018-06-14 | 2018-11-16 | 南京理工大学 | Hazardous act recognition methods based on personnel's framework information |
| CN108830215B (en) * | 2018-06-14 | 2021-07-13 | 南京理工大学 | Dangerous behavior identification method based on personnel skeleton information |
| CN109145868A (en) * | 2018-09-11 | 2019-01-04 | 广州杰赛科技股份有限公司 | A kind of Activity recognition method and apparatus assisting running training |
| CN109800659B (en) * | 2018-12-26 | 2021-05-25 | 中国科学院自动化研究所南京人工智能芯片创新研究院 | Action recognition method and device |
| CN109800659A (en) * | 2018-12-26 | 2019-05-24 | 中国科学院自动化研究所南京人工智能芯片创新研究院 | A kind of action identification method and device |
| CN109829422A (en) * | 2019-01-28 | 2019-05-31 | 哈尔滨工业大学 | A kind of video frequency identifying method based on the movement of impulsive neural networks falling over of human body |
| WO2020211242A1 (en) * | 2019-04-15 | 2020-10-22 | 深圳大学 | Behavior recognition-based method, apparatus and storage medium |
| CN110210372A (en) * | 2019-05-29 | 2019-09-06 | 中国科学院自动化研究所 | Based on skeleton Activity recognition method, the system for paying attention to enhancing figure convolutional network |
| CN110215216B (en) * | 2019-06-11 | 2020-08-25 | 中国科学院自动化研究所 | Behavior identification method and system based on skeletal joint point regional and hierarchical level |
| CN110215216A (en) * | 2019-06-11 | 2019-09-10 | 中国科学院自动化研究所 | Based on the with different levels Activity recognition method in skeletal joint point subregion, system |
| CN110309732B (en) * | 2019-06-13 | 2021-04-06 | 浙江大学 | Behavior identification method based on skeleton video |
| CN110309732A (en) * | 2019-06-13 | 2019-10-08 | 浙江大学 | Activity recognition method based on skeleton video |
| US11367195B2 (en) | 2019-06-28 | 2022-06-21 | Beijing Boe Optoelectronics Technology Co., Ltd. | Image segmentation method, image segmentation apparatus, image segmentation device |
| CN110276768A (en) * | 2019-06-28 | 2019-09-24 | 京东方科技集团股份有限公司 | Image partition method, image segmentation device, image segmentation apparatus and medium |
| CN110363131A (en) * | 2019-07-08 | 2019-10-22 | 上海交通大学 | Anomaly detection method, system and medium based on human skeleton |
| CN110348395B (en) * | 2019-07-12 | 2023-01-20 | 电子科技大学 | Skeleton behavior identification method based on space-time relationship |
| CN110348395A (en) * | 2019-07-12 | 2019-10-18 | 电子科技大学 | A kind of skeleton Activity recognition method based on time-space relationship |
| CN110765942A (en) * | 2019-10-23 | 2020-02-07 | 睿魔智能科技（深圳）有限公司 | Image data labeling method, device, equipment and storage medium |
| CN111898576B (en) * | 2020-08-06 | 2022-06-24 | 电子科技大学 | Behavior identification method based on human skeleton space-time relationship |
| CN111898576A (en) * | 2020-08-06 | 2020-11-06 | 电子科技大学 | Behavior identification method based on human skeleton space-time relationship |
| CN111914807A (en) * | 2020-08-18 | 2020-11-10 | 太原理工大学 | Miner behavior identification method based on sensor and skeleton information |
| CN112381004A (en) * | 2020-11-17 | 2021-02-19 | 华南理工大学 | Framework-based double-flow self-adaptive graph convolution network behavior identification method |
| CN112381004B (en) * | 2020-11-17 | 2023-08-08 | 华南理工大学 | Dual-flow self-adaptive graph rolling network behavior recognition method based on framework |
| CN112613436A (en) * | 2020-12-28 | 2021-04-06 | 中国联合网络通信集团有限公司 | Examination cheating detection method and device |
| CN112613436B (en) * | 2020-12-28 | 2023-08-04 | 中国联合网络通信集团有限公司 | Examination cheating detection method and device |
| CN112651360A (en) * | 2020-12-31 | 2021-04-13 | 福州大学 | Skeleton action recognition method under small sample |
| CN112818887A (en) * | 2021-02-08 | 2021-05-18 | 中国科学院自动化研究所 | Human body skeleton sequence behavior identification method based on unsupervised learning |
| CN112818887B (en) * | 2021-02-08 | 2023-04-07 | 中国科学院自动化研究所 | Human skeleton sequence behavior identification method based on unsupervised learning |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106203363A (en) | Human skeleton motion sequence Activity recognition method | |
| CN109325443B (en) | Face attribute identification method based on multi-instance multi-label deep migration learning | |
| CN104182772B (en) | A kind of gesture identification method based on deep learning | |
| CN106407986B (en) | A kind of identification method of image target of synthetic aperture radar based on depth model | |
| CN105956560B (en) | A kind of model recognizing method based on the multiple dimensioned depth convolution feature of pondization | |
| CN106504064A (en) | Clothes classification based on depth convolutional neural networks recommends method and system with collocation | |
| CN109711316A (en) | A kind of pedestrian recognition methods, device, equipment and storage medium again | |
| CN107229904A (en) | A kind of object detection and recognition method based on deep learning | |
| CN106920243A (en) | The ceramic material part method for sequence image segmentation of improved full convolutional neural networks | |
| CN109492529A (en) | A kind of Multi resolution feature extraction and the facial expression recognizing method of global characteristics fusion | |
| CN107844784A (en) | Face identification method, device, computer equipment and readable storage medium storing program for executing | |
| CN108062543A (en) | A kind of face recognition method and device | |
| CN110163813A (en) | A kind of image rain removing method, device, readable storage medium storing program for executing and terminal device | |
| CN109145766A (en) | Model training method, device, recognition methods, electronic equipment and storage medium | |
| CN108846350A (en) | Tolerate the face identification method of change of age | |
| CN106778796A (en) | Human motion recognition method and system based on hybrid cooperative model training | |
| CN107239733A (en) | Continuous hand-written character recognizing method and system | |
| CN105139004A (en) | Face expression identification method based on video sequences | |
| CN109817276A (en) | A kind of secondary protein structure prediction method based on deep neural network | |
| CN107808129A (en) | A kind of facial multi-characteristic points localization method based on single convolutional neural networks | |
| CN104298974A (en) | Human body behavior recognition method based on depth video sequence | |
| CN109919085B (en) | Human-human interaction behavior identification method based on light-weight convolutional neural network | |
| CN102750385A (en) | Correlation-quality sequencing image retrieval method based on tag retrieval | |
| CN110378208A (en) | A kind of Activity recognition method based on depth residual error network | |
| CN109783887A (en) | A kind of intelligent recognition and search method towards Three-dimension process feature |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20161207 |
|
| WD01 | Invention patent application deemed withdrawn after publication |