CN101989326B - Human posture recognition method and device - Google Patents
Human posture recognition method and device Download PDFInfo
- Publication number
- CN101989326B CN101989326B CN200910161452.7A CN200910161452A CN101989326B CN 101989326 B CN101989326 B CN 101989326B CN 200910161452 A CN200910161452 A CN 200910161452A CN 101989326 B CN101989326 B CN 101989326B
- Authority
- CN
- China
- Prior art keywords
- attitude
- module
- human body
- human
- posture
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
- 238000000034 method Methods 0.000 title claims abstract description 70
- 238000012549 training Methods 0.000 claims abstract description 64
- 239000011159 matrix material Substances 0.000 claims abstract description 26
- 230000009466 transformation Effects 0.000 claims abstract description 26
- 239000000284 extract Substances 0.000 claims abstract description 12
- 238000005070 sampling Methods 0.000 claims description 26
- 230000001149 cognitive effect Effects 0.000 claims description 12
- 239000013598 vector Substances 0.000 claims description 9
- 230000008878 coupling Effects 0.000 claims description 7
- 238000010168 coupling process Methods 0.000 claims description 7
- 238000005859 coupling reaction Methods 0.000 claims description 7
- 238000010606 normalization Methods 0.000 claims description 7
- 230000011218 segmentation Effects 0.000 claims description 5
- 238000000605 extraction Methods 0.000 abstract description 10
- 238000007781 pre-processing Methods 0.000 abstract description 3
- 230000009471 action Effects 0.000 description 20
- 230000008569 process Effects 0.000 description 13
- 238000004458 analytical method Methods 0.000 description 11
- 238000002474 experimental method Methods 0.000 description 8
- 238000012360 testing method Methods 0.000 description 6
- 238000004040 coloring Methods 0.000 description 5
- 238000010586 diagram Methods 0.000 description 4
- HUTDUHSNJYTCAR-UHFFFAOYSA-N ancymidol Chemical compound C1=CC(OC)=CC=C1C(O)(C=1C=NC=NC=1)C1CC1 HUTDUHSNJYTCAR-UHFFFAOYSA-N 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 230000002452 interceptive effect Effects 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 230000009467 reduction Effects 0.000 description 2
- 238000005452 bending Methods 0.000 description 1
- 230000000903 blocking effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000012512 characterization method Methods 0.000 description 1
- 239000012141 concentrate Substances 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 239000011521 glass Substances 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000011946 reduction process Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
- G06T7/35—Determination of transform parameters for the alignment of images, i.e. image registration using statistical methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/103—Static body considered as a whole, e.g. static pedestrian or occupant recognition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
- G06T7/262—Analysis of motion using transform domain methods, e.g. Fourier domain methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
- G06T7/37—Determination of transform parameters for the alignment of images, i.e. image registration using transform domain methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
- G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
- G06T7/74—Determining position or orientation of objects or cameras using feature-based methods involving reference images or patches
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Multimedia (AREA)
- Life Sciences & Earth Sciences (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Evolutionary Biology (AREA)
- Probability & Statistics with Applications (AREA)
- Human Computer Interaction (AREA)
- Mathematical Physics (AREA)
- Image Analysis (AREA)
Abstract
The invention provides a human posture recognition method and a human posture recognition device. The device comprises an input module, a preprocessing module, a training module, a feature extraction module, a template database building module, a search module and an output module, wherein the input module captures a human posture and forms an input image; the preprocessing module normalizes the input image to a fixed size and generates a sample in an independent shape; the training module reduces the dimensionality of sample data by a statistical learning method in a training phase to obtain a projection transformation matrix and build a nearest neighbor classifier; the feature extraction module extracts differentiated posture features from the sample data according to the projection transformation matrix in the training phase and a human recognition phase respectively; the template database building module builds a posture template database according to the differentiated posture features extracted in the training phase; the search module compares the differentiated posture features extracted in the human posture recognition phase with the posture template in the posture template database by using the nearest neighbor classifier to perform human posture matching; and the output module outputs an optimal matching posture and repositions a virtual human model.
Description
Technical field
Present invention relates in general to computer vision, more particularly, relate to the estimation of real-time body's gesture recognition and motion analysis.
Background technology
Human motion analysis and human body attitude identification are very important technology, and this technology uses significant human body attitude, to contribute to realizing man-machine interaction, virtual three-dimensional (3D) interactive game, 3D gesture recognition etc.In recent years, due to the promising learning value of its tool and commercial value, human motion capture research receives increasing concern.
Current existence is used for the kinds of schemes of human motion analysis.Some schemes need on object, to stick specific tag block or need specific capturing movement equipment, in general environment (such as home entertaining, 3D interactive game etc.), above-mentioned needs are inconvenient for user, and limit the application of these schemes.For the application of some reality, make a very big effort to make the mark for human motion analysis less.Existing method is mainly divided into two classes, that is, based on the method for human body analysis and the method based on sample.On the other hand, existing method also can be divided into method based on coloured image and 3D laser scanning manikin householder method.
As everyone knows, coloured image can only provide two dimension (2D) information, such as color, texture, shape etc.Therefore, the attitude uncertain problem in 2D information is inevitably caused.Such as, if some positions of human body are from blocking (self-occlusion), then due to the uncertainty of the human body attitude in coloured image, the method based on coloured image can not be used to carry out correct human body attitude identification.Even if employ more advanced attitude estimating method, the probabilistic colouring information of attitude also can cause reduction process speed and inaccurate attitude inferred results.In addition, according to the dress ornament of different seasons, people and ambient lighting change, colouring information is unstable (or not robust), and therefore, in complex environment, the human posture recognition method based on colouring information can not meet the demands.Therefore, some researchists and slip-stick artist use the 3D model of laser scanning to obtain more accurate result.But due to high cost and the large volume of acquisition equipment, laser scanner is impracticable in real environment (such as home entertaining, 3D interactive game etc.).In order to solve this problem, need a kind of method and apparatus carrying out real-time body's gesture recognition in mixed and disorderly environment.
Summary of the invention
The present invention still concentrates on human body attitude identification without the need to tag block or human motion analysis.But, in the present invention, solve the problems of the prior art by new mode.First, the present invention adopts the TOF depth camera of combination (can simultaneously provide depth image and intensity image) and colour TV camera (providing coloured image).Secondly, the invention provides a kind of method and apparatus identifying human body attitude in complex environment, the method and device can effectively utilize depth information and colouring information to carry out human body attitude identification.
According to an aspect of the present invention, provide a kind of human body attitude recognition device, this device comprises: load module, comprises depth camera and colour TV camera, for catching human body attitude simultaneously, forms input picture; Input picture pre-service is applicable form by pretreatment module, and is fixed size by image normalization, produces shape independently attitude sampling, forms sampled data; Training module, the dimension of carrying out sampled data in training stage Using statistics learning method reduces, and to obtain the projective transformation matrix of original image space to feature space, and builds nearest neighbor classifier; Characteristic extracting module, extracts distinguishing posture feature in training stage and human body attitude cognitive phase from sampled data respectively according to described projective transformation matrix; Template database builds module, builds pose template database according to the distinguishing posture feature that characteristic extracting module is extracted in the training stage; Search module, distinguishing posture feature characteristic extracting module extracted in human body attitude cognitive phase by nearest neighbor classifier is compared with the pose template in pose template database, to carry out human body attitude coupling; Output module, exports the attitude of optimum matching, and reorientates virtual human model based on the attitude of optimum matching.
According to a further aspect in the invention, provide a kind of human posture recognition method, the method comprises: (a) utilizes depth camera and colour TV camera to catch human body attitude simultaneously, forms input picture; B input picture pre-service is applicable form by (), and be fixed size by image normalization, produces shape independently attitude sampling, forms sampled data; C dimension that () carries out sampled data in training stage Using statistics learning method reduces, and to obtain the projective transformation matrix of original image space to feature space, and builds nearest neighbor classifier; D () extracts distinguishing posture feature in training stage and human body attitude cognitive phase from sampled data respectively according to described projective transformation matrix; E () builds pose template database according to the distinguishing posture feature extracted in the training stage; F the distinguishing posture feature extracted in human body attitude cognitive phase is compared with the pose template in pose template database by nearest neighbor classifier by (), to carry out human body attitude coupling; G () exports the attitude of optimum matching, and reorientate virtual human model based on the attitude of optimum matching.
Accompanying drawing explanation
In conjunction with the drawings, from the description of the following examples, the present invention these and/or other side and advantage will become clear, and are easier to understand, wherein:
Fig. 1 is the block diagram of the human body attitude recognition device according to the embodiment of the present invention;
Fig. 2 shows the sampled images of catching according to the load module of the embodiment of the present invention;
Fig. 3 is the process flow diagram of the human posture recognition method according to the embodiment of the present invention;
Fig. 4 shows the Image semantic classification process of the pretreatment module according to the embodiment of the present invention;
Fig. 5 shows the example of the location shoulder point according to the embodiment of the present invention;
Fig. 6 shows the sorter training process of the training module according to the embodiment of the present invention;
Fig. 7 shows the template database building process building module according to the template database of the embodiment of the present invention.
Fig. 8 shows the characteristic extraction procedure of the characteristic extracting module according to the embodiment of the present invention;
Fig. 9 shows the human body attitude output procedure of characteristic matching according to the search module of the embodiment of the present invention and output module;
Figure 10 to Figure 13 shows the experiment 1 and experiment 2 carried out according to the present invention.
Embodiment
Below, embodiments of the invention are described in detail with reference to accompanying drawing.
Fig. 1 is the block diagram of the human body attitude recognition device according to the embodiment of the present invention.As shown in Figure 1, this human body attitude recognition device comprises load module 101, pretreatment module 102, training module 103, template database (DB) build module 104, characteristic extracting module 105, search module 106 and output module 107.
Load module 101 comprises two video cameras, that is, depth camera and colour TV camera, and depth camera can be such as TOF (Time of Flight) depth camera.TOF depth camera and colour TV camera are used for catching human body attitude simultaneously, form input picture.Input picture pre-service is applicable form by pretreatment module 102, and is fixed size by image normalization, produces shape and independently samples.The raw data of normalized sampling is high-dimensional.After pre-processing, training module 103 the training stage (namely, learning phase) Using statistics learning method (such as PCA (pivot analysis), LLE (local linear embedding) etc.), the dimension of carrying out sampled data reduces, with obtain original image space to feature space projective transformation matrix (namely, obtain the Feature Selection mechanism being used for feature extraction), and build nearest neighbor classifier.In order to identify human body attitude, template DB builds the previous pose template database that module 104 builds off-line.Build in module 104 at template DB, different human body attitudes is manually marked.Then, characteristic extracting module 105 extracts distinguishing posture feature in the training stage from sampled data according to projective transformation matrix, so that template DB structure module 104 finally sets up the attitude corresponding relation between posture feature and relevant attitude.
In the online gesture recognition stage, characteristic extracting module 105 only extracts distinguishing posture feature according to projective transformation matrix.Search module 106 receives described distinguishing posture feature, distinguishing posture feature characteristic extracting module 105 extracted in human body attitude cognitive phase by nearest neighbor classifier is compared with the pose template in pose template database, to carry out human body attitude coupling.Afterwards, output module 107 provides the attitude of optimum matching, reorientates virtual human model.Thus, whole human body attitude identifying is just completed.
In the present invention, use two video cameras to catch identical scene simultaneously.A video camera is TOF depth camera, and another video camera is colour TV camera.Colour TV camera can be traditional CCD/CMOS video camera, can provide coloured image.TOF depth camera can provide depth image and intensity image.Depth image represents the distance between reference object and TOF depth camera.Intensity image represents the light intensity energy that TOF depth camera receives.
Fig. 2 shows the sampled images of catching according to the load module 101 of the embodiment of the present invention.As can be seen from Figure 2, intensity image provides background image clearly, and this background image is very suitable for carrying out foreground extraction and outline (silhouette) segmentation.Intuitively, can easily use clearly background intensity image to locate head and the trunk of human body.When the glasses reflection that people wears is very serious, if want to locate eye locations, then intensity image may not be optimal selection.Therefore, coloured image can be used to locate eye locations.There is multiple diverse ways and locate eye locations in coloured image.In addition, in some cases, coloured image and sketch figure picture have ambiguity for human body attitude analysis, therefore can make full use of depth image to alleviate the ambiguousness of human body attitude.
After the input picture (coloured image, depth image and intensity image) obtaining three types, needing these Image semantic classification is applicable form.Utilize the input picture of this three types to carry out Image semantic classification.
Fig. 3 is the process flow diagram of the human posture recognition method according to the embodiment of the present invention.
With reference to Fig. 3, in operation 301, the depth camera in load module 101 and colour TV camera catch human body attitude simultaneously, form input picture.In operation 302, input picture pre-service is applicable form by pretreatment module 102, and is fixed size by image normalization, produces shape and independently samples.In operation 303, the dimension that training module 103 carries out sampled data in training stage Using statistics learning method reduces dimension reduction, to obtain the projective transformation matrix of original image space to feature space, and builds nearest neighbor classifier.In operation 304, levy extraction module 104 and extract distinguishing posture feature in training stage and human body attitude cognitive phase from sampled data respectively according to projective transformation matrix.In operation 305, template database (DB) builds module and builds pose template database according to the distinguishing posture feature in the training stage.In operation 306, the distinguishing posture feature that characteristic extracting module 105 is extracted by nearest neighbor classifier by search module 106 in human body attitude cognitive phase compares with the pose template in pose template database, to carry out human body attitude coupling.In operation 307, output module 107 exports the attitude of optimum matching, and reorientates virtual human model based on the attitude of optimum matching.
Describe according to Image semantic classification of the present invention referring to Fig. 4 and Fig. 5.Fig. 4 shows the Image semantic classification process of the pretreatment module 102 according to the embodiment of the present invention.
With reference to Fig. 4, in operation 401, pretreatment module 102 working strength image is split human region and extracts outline.In this process, threshold segmentation method can be used.In operation 402, pretreatment module 102 uses the human region of segmentation as the mask (mask) in coloured image, so that head and trunk.Head and trunk are detected, the detecting device training that pretreatment module 102 can use existing AdaBoost algorithm to provide and local feature.Image normalization is fixed size by pretreatment module 102, therefore needs some reference point.In operation 403, pretreatment module 102 select eye locations and shoulder position as a reference point, this is because for the front view of human body, eye locations is strong reference point in head zone, and shoulder position is strong reference point in torso area.In order to locate eye locations con vigore, pretreatment module 102 can use the eye detecting device of existing training, and this eye detecting device also can be trained based on AdaBoost algorithm and local characterization method.In order to location shoulder position (comprises left shoulder point P con vigore
lSwith right shoulder point P
rS), pretreatment module 102 adopts a kind of simple method, and the method has the advantage of the depth image of mask as shown in Figure 4.Pretreatment module 102 detects bending point in the horizontal projection and vertical projection of torso area as shoulder point.
After located eye locations and shoulder position, in operation 404, pretreatment module 102 carries out shape normalized.The normalized object of shape produces shape independently to sample.Suppose P
1represent the center between left eye and right eye, P
2represent left shoulder point P
lSwith right shoulder point P
rSbetween center, D
1represent P
1and P
2between distance, D
2represent left shoulder point P
lSwith right shoulder point P
rSbetween distance, then adopt D
1as the reference length of height of sampling h, adopt D
2as the reference length of sampling width w.Shape normalized part 1024 uses equation below sampling pruned and be normalized to the size of 80 × 48: D
2/ D
1=5: 2 (this ratio is used for being normalized shape); W=4 × D
2and h=6 × D
1(for sample area size).For punch action, sampling is pruned and is normalized to the size of 80 × 80 by pretreatment module 102, and arranges w=h=6 × D
1, because the image gathered does not comprise complicated punch action.
Fig. 5 shows the example of the location shoulder point according to the embodiment of the present invention.Particularly, (a) in Fig. 5 is the outline of human body foreground area.(b) in Fig. 5 is this histogram in this image (this outline) vertical direction, the position of the horizontal direction of horizontal ordinate representative image (namely, the row coordinate of image, span is 0 ~ picture traverse), ordinate implication is in some row coordinate points, the aggregate-value (that is, the vertical direction projection value of this row coordinate points) of all pixel values of these row in image.(c) in Fig. 5 is image histogram in the horizontal direction, the position of the vertical direction of horizontal ordinate representative image (namely, the row-coordinate of image, span is 0 ~ picture altitude), ordinate implication is at some row-coordinate points, the aggregate-value (i.e. the horizontal direction projection value of this row-coordinate point) of all pixel values of this row in image.(d) in Fig. 5 is the result of location shoulders of human body point (region detection).
Describe sorter according to the present invention referring to Fig. 6 to train.Fig. 6 shows the sorter training process of the training module 103 according to the embodiment of the present invention.
Training module 103 adopts PCA (pivot analysis) and LLE (local linear embedding) learning method to obtain the projective transformation matrix of original image space to feature space.
With reference to Fig. 6, in operation 601, training module 103 creates training dataset.The standard that training dataset is selected makes to train sampling (that is, the attitude sampling in training stage) diversified and representative, makes training dataset comprise human action as much as possible.Training module 103 mainly selects diversified training sampling according to different punch action, and training sampling is uniformly distributed in image space.Then, in operation 602, training sampled data is transformed to suitable input vector, to learn by training module 103.That is, 2D data are directly expanded into one dimension (1D) vector by training module 103.Then, in operation 603, training module 103 adopts the statistical learning methods such as PCA (pivot analysis) and LLE (local linear embedding) to carry out dimension reduction, to obtain projective transformation matrix.Those skilled in the art can obtain the concrete introduction about PCA and LLE from prior art, are therefore no longer described in greater detail here.After this, in operation 604, training module 103 builds has L
1nN (arest neighbors) sorter of distance (measuring similarity value), L
1be defined in and hereinafter describe.
Describe referring to Fig. 7 and build according to template DB of the present invention.Fig. 7 shows the template DB building process building module 104 according to the template DB of the embodiment of the present invention.It is part and parcel that template DB builds for the motion analysis based on sample.
With reference to Fig. 7, build module 104 in operation 701, template DB and select different attitude samplings.Then, build module 104 pairs of attitude sampled images in operation 702, template DB manually to mark.Preferably, template DB structure module 104 uses and produces based on the motion capture system marked or suitable computer graphical software the data set be marked.Because present apparatus and layout limit, acquire the punch action attitude of 8 types in the present invention, and eliminate the process of mark.Characteristic extracting module 105 extracts the distinguishing feature of low dimension according to the projective transformation matrix that training module 103 obtains from sampling.Then, the corresponding relation between described distinguishing characteristics and attitude (skeleton) is set up in operation 703, template DB structure module 104 based on the distinguishing feature extracted.In the present invention, the corresponding relation between the index establishing the punch action of described distinguishing characteristics and 8 types.After this, build module 104 in operation 704, template DB and produce the template of skeleton (or action) index comprising eigenvector and be associated based on the corresponding relation set up.
Describe according to online gesture recognition of the present invention referring to Fig. 8 and Fig. 9.After the suitable template DB that trained sorter and foundation, online gesture recognition can be carried out.Similar with the training stage, first pre-service is carried out to input picture.Process subsequently comprises feature extraction, characteristic matching and human body attitude and exports.
Fig. 8 shows the characteristic extraction procedure of the characteristic extracting module 105 according to the embodiment of the present invention, and Fig. 9 shows the human body attitude output procedure of characteristic matching according to the search module 106 of the embodiment of the present invention and output module 107.
The object of feature extraction is to extract distinguishing feature to mate.With reference to Fig. 8, in operation 801, the depth data transform of input picture is applicable image vector by characteristic extracting module 105, that is, directly 2D data are expanded into 1D vector.Then, in operation 802, characteristic extracting module 105 is used in the projective transformation matrix that obtains in the training stage by the data projection from image space to feature space.In the present invention, PCA and the LLE projective transformation matrix of training can be used.
Suppose X={x
1, x
2... x
nrepresent the 1D view data of input (wherein, N=w × h, w are sampling width, and h is height of sampling), W represents the PCA/LLE projective transformation matrix (dimension of W is: N × M, M < < N) of training.Therefore, in operation 803, characteristic extracting module 105 can obtain eigenvector V, V=W
tthe dimension of X, eigenvector V is M.
After having carried out feature extraction, utilize NN (arest neighbors) sorter in template database, take out top-n optimum matching attitude.Namely, the distinguishing posture feature extracted in human body attitude cognitive phase is compared with the pose template in pose template database 104 by nearest neighbor classifier by search module 106, to carry out human body attitude coupling.Specifically, with reference to Fig. 9, in operation 901, the distance between the eigenvector that search module 106 utilizes nearest neighbor classifier to calculate in current eigenvector and template database.Suppose V
0current eigenvector (that is, the eigenvector of input), V
ibe eigenvector in template DB (i=1 ..., N), S
ibe be associated skeleton (attitude) index (i=1 ..., N).Service range measures L
1=| V
0-V
i| (i=1 ..., N), the eigenvector V of input
0by with all N number of template V in template DB
imate, obtain a series of L
1measuring similarity value.In operation 902, search module 106 is based on described L
1distance can obtain the index of top-n optimum matching in template database.In operation 903, output module 107 obtains the attitude (skeleton) of optimum matching according to the index profit of optimum matching in template database.Then, in operation 904, output module 107 reorientates virtual human model based on the attitude (skeleton) of optimum matching.
Such as, establish pose template database 104 in the off-line learning stage, pose template database 104 comprises the maneuver library of a set of taijiquan, has the image of 500 actions.When setting up pose template database 104, be extracted the eigenvector of each human action respectively, and the position of each articulation point is marked (being convenient to output module 107 drives virtual portrait to show).In the stage of the online actions identification of reality, user has done an action, captures the image of this action, and carried out pre-service by pretreatment module 102, characteristic extracting module 105 extracts distinguishing posture feature, then obtains the eigenvector of this action; 500 stack features vectors in this eigenvector and pose template database 104 are compared by nearest neighbor classifier by search module 106 respectively, calculate similarity, the n finding a similarity maximum action, and this process is exactly the process of top-n arest neighbors classification; If n=1, find a action the most close exactly; Output module 107 exports the human synovial dot information corresponding with this action, carries out driving or the display of virtual portrait.
The experiment 1 and experiment 2 carried out according to the present invention are described referring to Figure 10 to Figure 13.
With reference to Figure 10, in experiment 1, for specific people.The attitude data of the people of test is included in training data.In the training stage, relate to 4 people, have the punch action of 8 kinds of attitudes, have 1079 samplings (each sample size is 80 × 80), carry out reorientating manikin according to 100 dimensions.At test phase, relate to 4 people identical with the training stage, have the punch action of 8 kinds of attitudes, test 1079 samplings.
Figure 11 shows the result of experiment 1.(a) in Figure 11 shows the Search Results adopting LLE method to obtain, (b) in Figure 11 shows the Search Results adopting PCA method to obtain, (a) is in fig. 11 with in (b), an image in the upper left corner is transfused to as inquiry, and other image is output as rreturn value.
With reference to Figure 12, in experiment 2, for unspecific people.The attitude data of the people of test is not comprised in training data.In the training stage, relate to 4 people, have the punch action of 8 kinds of attitudes, have 1079 samplings, carry out reorientating manikin according to 100 dimensions.At test phase, relate to 2 people different from the training stage, have the punch action of 8 kinds of attitudes, test 494 samplings.
Figure 13 shows the result of experiment 2.(a) in Figure 13 shows the Search Results adopting LLE method to obtain, (b) in Figure 13 shows the Search Results adopting PCA method to obtain, (a) is in fig. 13 with in (b), an image in the upper left corner is transfused to as inquiry, and other image is output as rreturn value.
Therefore, compared with traditional method based on coloured image, the present invention can solve fuzzy problem in outline owing to using depth data.Present invention utilizes depth information and colouring information, can provide a kind of shape method for normalizing, the method can obtain shape independently gesture recognition.In addition, present invention employs statistical learning method and method for fast searching, make human body attitude recognition device structure simply and more effective.
Although the present invention is described in detail with reference to its exemplary embodiment and shows, but will be understood by those skilled in the art that, when not departing from the spirit and scope of the present invention be defined by the claims, the various changes of form and details can be carried out to it.
Claims (14)
1. a human body attitude recognition device, comprising:
Load module, comprises depth camera and colour TV camera, for catching human body attitude simultaneously, forms input picture;
Input picture pre-service is applicable form by pretreatment module, and is fixed size by image normalization, produces shape independently attitude sampling, forms sampled data;
Training module, the dimension of carrying out sampled data in training stage Using statistics learning method reduces, and to obtain the projective transformation matrix of original image space to feature space, and builds nearest neighbor classifier;
Characteristic extracting module, extracts distinguishing posture feature in training stage and human body attitude cognitive phase from sampled data respectively according to described projective transformation matrix;
Template database builds module, builds pose template database according to the distinguishing posture feature that characteristic extracting module is extracted in the training stage;
Search module, distinguishing posture feature characteristic extracting module extracted in human body attitude cognitive phase by nearest neighbor classifier is compared with the pose template in pose template database, to carry out human body attitude coupling;
Output module, exports the attitude of optimum matching, and reorientates virtual human model based on the attitude of optimum matching,
Wherein, training module creates training dataset, attitude is sampled and is uniformly distributed in image space, sampled data is transformed to input vector, and the dimension adopting statistical learning method to carry out sampled data reduces, to obtain described projective transformation matrix,
Wherein, the depth data transform of input picture is one-dimensional data vector by characteristic extracting module, is used in the projective transformation matrix that obtains in the training stage by the data projection from image space to feature space, to obtain eigenvector.
2. human body attitude recognition device according to claim 1, wherein, depth camera forms depth image and the intensity image of human body attitude, and colour TV camera forms the coloured image of human body attitude.
3. human body attitude recognition device according to claim 2, wherein, pretreatment module working strength image is split human body attitude and is extracted outline, use human region head and the trunk of segmentation, select eye locations and shoulder position is as a reference point carries out shape normalized, produce shape independently attitude sampling.
4. human body attitude recognition device according to claim 3, wherein, described statistical learning method comprises pca method or local linear embedding grammar.
5. human body attitude recognition device according to claim 4, wherein, template database builds the different attitude sampling of model choice, manually marks attitude sampled images; Characteristic extracting module extracts the distinguishing feature of low dimension from attitude sampling according to described projective transformation matrix; Template database structure module sets up the corresponding relation between described distinguishing characteristics and attitude based on the distinguishing feature extracted, and produce the template of attitude index comprising eigenvector and be associated based on the corresponding relation set up, to build pose template database.
6. human body attitude recognition device according to claim 5, wherein, search module calculates the distance between the eigenvector in current eigenvector and pose template database by nearest neighbor classifier, obtains the index of optimum matching based on described distance in pose template database.
7. human body attitude recognition device according to claim 6, wherein, output module obtains the attitude of optimum matching in pose template database according to the index of optimum matching, and reorientates virtual human model based on the attitude of optimum matching.
8. a human posture recognition method, comprises the following steps:
A () utilizes depth camera and colour TV camera to catch human body attitude simultaneously, form input picture;
B input picture pre-service is applicable form by (), and be fixed size by image normalization, produces shape independently attitude sampling, forms sampled data;
C dimension that () carries out sampled data in training stage Using statistics learning method reduces, and to obtain the projective transformation matrix of original image space to feature space, and builds nearest neighbor classifier;
D () extracts distinguishing posture feature in training stage and human body attitude cognitive phase from sampled data respectively according to described projective transformation matrix;
E () builds pose template database according to the distinguishing posture feature extracted in the training stage;
F the distinguishing posture feature extracted in human body attitude cognitive phase is compared with the pose template in pose template database by nearest neighbor classifier by (), to carry out human body attitude coupling;
G () exports the attitude of optimum matching, and reorientate virtual human model based on the attitude of optimum matching,
Wherein, step (c) comprising: create training dataset, attitude is sampled and is uniformly distributed in image space; Sampled data is transformed to input vector; The dimension adopting statistical learning method to carry out sampled data reduces, to obtain described projective transformation matrix,
Wherein, step (d) comprising: be one-dimensional data vector by the depth data transform of input picture; Be used in the projective transformation matrix that obtains in the training stage by the data projection from image space to feature space, to obtain eigenvector.
9. human posture recognition method according to claim 8, wherein, depth camera forms depth image and the intensity image of human body attitude, and colour TV camera forms the coloured image of human body attitude.
10. human posture recognition method according to claim 9, wherein, step (b) comprising:
Working strength image is split human body attitude and is extracted outline;
Use the human region head after segmentation and trunk,
Select eye locations and shoulder position is as a reference point carries out shape normalized, produce shape independently attitude sampling.
11. human posture recognition methods according to claim 10, wherein, described statistical learning method comprises pca method or local linear embedding grammar.
12. human posture recognition methods according to claim 11, wherein, step (e) comprising:
Select different attitude samplings, attitude sampled images is manually marked;
The corresponding relation between described distinguishing characteristics and attitude is set up based on the distinguishing feature extracted in the training stage;
The template of attitude index comprising eigenvector and be associated is produced, to build pose template database based on the corresponding relation set up.
13. human posture recognition methods according to claim 12, wherein, step (f) comprising:
The distance between the eigenvector in current eigenvector and pose template database is calculated by nearest neighbor classifier;
In pose template database, the index of optimum matching is obtained based on described distance.
14. human posture recognition methods according to claim 13, wherein, step (g) comprising:
In pose template database, the attitude of optimum matching is obtained according to the index of optimum matching;
Attitude based on optimum matching reorientates virtual human model.
Priority Applications (3)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN200910161452.7A CN101989326B (en) | 2009-07-31 | 2009-07-31 | Human posture recognition method and device |
| KR1020100036589A KR20110013200A (en) | 2009-07-31 | 2010-04-20 | Identifying method of human attitude and apparatus of the same |
| US12/805,457 US20110025834A1 (en) | 2009-07-31 | 2010-07-30 | Method and apparatus of identifying human body posture |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN200910161452.7A CN101989326B (en) | 2009-07-31 | 2009-07-31 | Human posture recognition method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN101989326A CN101989326A (en) | 2011-03-23 |
| CN101989326B true CN101989326B (en) | 2015-04-01 |
Family
ID=43745858
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN200910161452.7A Expired - Fee Related CN101989326B (en) | 2009-07-31 | 2009-07-31 | Human posture recognition method and device |
Country Status (2)
| Country | Link |
|---|---|
| KR (1) | KR20110013200A (en) |
| CN (1) | CN101989326B (en) |
Families Citing this family (75)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8908995B2 (en) | 2009-01-12 | 2014-12-09 | Intermec Ip Corp. | Semi-automatic dimensioning with imager on a portable device |
| CN102184008A (en) * | 2011-05-03 | 2011-09-14 | 北京天盛世纪科技发展有限公司 | Interactive projection system and method |
| US9208571B2 (en) * | 2011-06-06 | 2015-12-08 | Microsoft Technology Licensing, Llc | Object digitization |
| KR101815975B1 (en) * | 2011-07-27 | 2018-01-09 | 삼성전자주식회사 | Apparatus and Method for Detecting Object Pose |
| CN102436301B (en) * | 2011-08-20 | 2015-04-15 | Tcl集团股份有限公司 | Human-machine interaction method and system based on reference region and time domain information |
| CN102324041B (en) * | 2011-09-09 | 2014-12-03 | 深圳泰山在线科技有限公司 | Pixel classification method, joint body gesture recognition method and mouse instruction generating method |
| CN102509074B (en) * | 2011-10-18 | 2014-01-29 | Tcl集团股份有限公司 | Target identification method and device |
| KR101919831B1 (en) | 2012-01-11 | 2018-11-19 | 삼성전자주식회사 | Object Recognition Apparatus, Classification Tree Learning Apparatus and Method thereof |
| KR101908284B1 (en) | 2012-01-13 | 2018-10-16 | 삼성전자주식회사 | Apparatus and method for analysising body parts association |
| US9779546B2 (en) * | 2012-05-04 | 2017-10-03 | Intermec Ip Corp. | Volume dimensioning systems and methods |
| US9007368B2 (en) | 2012-05-07 | 2015-04-14 | Intermec Ip Corp. | Dimensioning system calibration systems and methods |
| CN103390174A (en) * | 2012-05-07 | 2013-11-13 | 深圳泰山在线科技有限公司 | Physical education assisting system and method based on human body posture recognition |
| CN103390150B (en) * | 2012-05-08 | 2019-01-08 | 北京三星通信技术研究有限公司 | human body part detection method and device |
| US10007858B2 (en) | 2012-05-15 | 2018-06-26 | Honeywell International Inc. | Terminals and methods for dimensioning objects |
| US10321127B2 (en) | 2012-08-20 | 2019-06-11 | Intermec Ip Corp. | Volume dimensioning system calibration systems and methods |
| US9939259B2 (en) | 2012-10-04 | 2018-04-10 | Hand Held Products, Inc. | Measuring object dimensions using mobile computer |
| US9841311B2 (en) | 2012-10-16 | 2017-12-12 | Hand Held Products, Inc. | Dimensioning system |
| US9080856B2 (en) | 2013-03-13 | 2015-07-14 | Intermec Ip Corp. | Systems and methods for enhancing dimensioning, for example volume dimensioning |
| CN103310193B (en) * | 2013-06-06 | 2016-05-25 | 温州聚创电气科技有限公司 | A kind of method that records sportsman's important technology action moment in gymnastics video |
| US10228452B2 (en) | 2013-06-07 | 2019-03-12 | Hand Held Products, Inc. | Method of error correction for 3D imaging device |
| CN103366160A (en) * | 2013-06-28 | 2013-10-23 | 西安交通大学 | Objectionable image distinguishing method integrating skin color, face and sensitive position detection |
| US9239950B2 (en) | 2013-07-01 | 2016-01-19 | Hand Held Products, Inc. | Dimensioning system |
| US9464885B2 (en) | 2013-08-30 | 2016-10-11 | Hand Held Products, Inc. | System and method for package dimensioning |
| CN104573612B (en) * | 2013-10-16 | 2019-10-22 | 北京三星通信技术研究有限公司 | The device and method of the posture for the multiple human objects being overlapped in estimating depth image |
| CN104463089A (en) * | 2013-11-25 | 2015-03-25 | 安徽寰智信息科技股份有限公司 | Human body posture recognizing device |
| CN104748750B (en) * | 2013-12-28 | 2015-12-02 | 华中科技大学 | A kind of model constrained under the Attitude estimation of Three dimensional Targets in-orbit method and system |
| CN103778436B (en) * | 2014-01-20 | 2017-04-05 | 电子科技大学 | A kind of pedestrian's attitude detecting method based on image procossing |
| CN104125396B (en) * | 2014-06-24 | 2018-06-08 | 小米科技有限责任公司 | Image capturing method and device |
| US9823059B2 (en) | 2014-08-06 | 2017-11-21 | Hand Held Products, Inc. | Dimensioning system with guided alignment |
| US10775165B2 (en) | 2014-10-10 | 2020-09-15 | Hand Held Products, Inc. | Methods for improving the accuracy of dimensioning-system measurements |
| US9779276B2 (en) | 2014-10-10 | 2017-10-03 | Hand Held Products, Inc. | Depth sensor based auto-focus system for an indicia scanner |
| US10810715B2 (en) | 2014-10-10 | 2020-10-20 | Hand Held Products, Inc | System and method for picking validation |
| US9762793B2 (en) | 2014-10-21 | 2017-09-12 | Hand Held Products, Inc. | System and method for dimensioning |
| US10060729B2 (en) | 2014-10-21 | 2018-08-28 | Hand Held Products, Inc. | Handheld dimensioner with data-quality indication |
| US9752864B2 (en) | 2014-10-21 | 2017-09-05 | Hand Held Products, Inc. | Handheld dimensioning system with feedback |
| US9557166B2 (en) | 2014-10-21 | 2017-01-31 | Hand Held Products, Inc. | Dimensioning system with multipath interference mitigation |
| US9897434B2 (en) | 2014-10-21 | 2018-02-20 | Hand Held Products, Inc. | Handheld dimensioning system with measurement-conformance feedback |
| US9786101B2 (en) | 2015-05-19 | 2017-10-10 | Hand Held Products, Inc. | Evaluating image values |
| US10066982B2 (en) | 2015-06-16 | 2018-09-04 | Hand Held Products, Inc. | Calibrating a volume dimensioner |
| US20160377414A1 (en) | 2015-06-23 | 2016-12-29 | Hand Held Products, Inc. | Optical pattern projector |
| US9857167B2 (en) | 2015-06-23 | 2018-01-02 | Hand Held Products, Inc. | Dual-projector three-dimensional scanner |
| US9835486B2 (en) | 2015-07-07 | 2017-12-05 | Hand Held Products, Inc. | Mobile dimensioner apparatus for use in commerce |
| CN104952221B (en) * | 2015-07-09 | 2017-06-13 | 深圳大学 | Myopia-proof intelligent desk lamp |
| EP3396313B1 (en) | 2015-07-15 | 2020-10-21 | Hand Held Products, Inc. | Mobile dimensioning method and device with dynamic accuracy compatible with nist standard |
| US20170017301A1 (en) | 2015-07-16 | 2017-01-19 | Hand Held Products, Inc. | Adjusting dimensioning results using augmented reality |
| US10094650B2 (en) | 2015-07-16 | 2018-10-09 | Hand Held Products, Inc. | Dimensioning and imaging items |
| US10249030B2 (en) | 2015-10-30 | 2019-04-02 | Hand Held Products, Inc. | Image transformation for indicia reading |
| US10225544B2 (en) | 2015-11-19 | 2019-03-05 | Hand Held Products, Inc. | High resolution dot pattern |
| CN105631861B (en) * | 2015-12-21 | 2019-10-01 | 浙江大学 | Restore the method for 3 D human body posture from unmarked monocular image in conjunction with height map |
| US10025314B2 (en) | 2016-01-27 | 2018-07-17 | Hand Held Products, Inc. | Vehicle positioning and object avoidance |
| CN105997094B (en) * | 2016-05-09 | 2019-03-29 | 北京科技大学 | A kind of gesture recognition device and method |
| US10339352B2 (en) | 2016-06-03 | 2019-07-02 | Hand Held Products, Inc. | Wearable metrological apparatus |
| US9940721B2 (en) | 2016-06-10 | 2018-04-10 | Hand Held Products, Inc. | Scene change detection in a dimensioner |
| US10163216B2 (en) | 2016-06-15 | 2018-12-25 | Hand Held Products, Inc. | Automatic mode switching in a volume dimensioner |
| CN106780569A (en) * | 2016-11-18 | 2017-05-31 | 深圳市唯特视科技有限公司 | A kind of human body attitude estimates behavior analysis method |
| CN108154161A (en) * | 2016-12-05 | 2018-06-12 | 上海西门子医疗器械有限公司 | The method of training grader, the method and medical instrument for determining detected object position |
| US10909708B2 (en) | 2016-12-09 | 2021-02-02 | Hand Held Products, Inc. | Calibrating a dimensioner using ratios of measurable parameters of optic ally-perceptible geometric elements |
| CN106780530B (en) * | 2016-12-15 | 2019-06-14 | 广州视源电子科技股份有限公司 | A kind of figure prediction technique and equipment |
| CN108629345B (en) | 2017-03-17 | 2021-07-30 | 北京京东尚科信息技术有限公司 | High-dimensional image feature matching method and device |
| US11047672B2 (en) | 2017-03-28 | 2021-06-29 | Hand Held Products, Inc. | System for optically dimensioning |
| CN107358149B (en) * | 2017-05-27 | 2020-09-22 | 深圳市深网视界科技有限公司 | Human body posture detection method and device |
| US10733748B2 (en) | 2017-07-24 | 2020-08-04 | Hand Held Products, Inc. | Dual-pattern optical 3D dimensioning |
| CN108121963B (en) * | 2017-12-21 | 2021-08-24 | 北京奇虎科技有限公司 | Video data processing method and device and computing equipment |
| CN108345869B (en) * | 2018-03-09 | 2022-04-08 | 南京理工大学 | Driver posture recognition method based on depth image and virtual data |
| US10584962B2 (en) | 2018-05-01 | 2020-03-10 | Hand Held Products, Inc | System and method for validating physical-item security |
| CN109101866B (en) * | 2018-06-05 | 2020-12-15 | 中国科学院自动化研究所 | Pedestrian re-identification method and system based on segmentation silhouette |
| CN110362843B (en) * | 2018-11-20 | 2022-06-17 | 莆田学院 | Virtual human overall attitude approximate generation method based on typical attitude |
| TWI707249B (en) * | 2018-11-27 | 2020-10-11 | 美律實業股份有限公司 | System and method for generating label data |
| CN109657631B (en) * | 2018-12-25 | 2020-08-11 | 上海智臻智能网络科技股份有限公司 | Human body posture recognition method and device |
| CN110020630B (en) * | 2019-04-11 | 2020-12-18 | 成都乐动信息技术有限公司 | Method and device for evaluating action completion degree, storage medium and electronic equipment |
| CN112288798A (en) * | 2019-07-24 | 2021-01-29 | 鲁班嫡系机器人（深圳）有限公司 | Posture recognition and training method, device and system |
| US11639846B2 (en) | 2019-09-27 | 2023-05-02 | Honeywell International Inc. | Dual-pattern optical 3D dimensioning |
| US11564651B2 (en) | 2020-01-14 | 2023-01-31 | GE Precision Healthcare LLC | Method and systems for anatomy/view classification in x-ray imaging |
| CN111353543B (en) * | 2020-03-04 | 2020-09-11 | 镇江傲游网络科技有限公司 | Motion capture data similarity measurement method, device and system |
| CN115294375B (en) * | 2022-10-10 | 2022-12-13 | 南昌虚拟现实研究院股份有限公司 | Speckle depth estimation method and system, electronic device and storage medium |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101079103A (en) * | 2007-06-14 | 2007-11-28 | 上海交通大学 | Human face posture identification method based on sparse Bayesian regression |
| CN101332362A (en) * | 2008-08-05 | 2008-12-31 | 北京中星微电子有限公司 | Interactive delight system based on human posture recognition and implement method thereof |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| GB0308943D0 (en) * | 2003-04-17 | 2003-05-28 | Univ Dundee | A system for determining the body pose of a person from images |
-
2009
- 2009-07-31 CN CN200910161452.7A patent/CN101989326B/en not_active Expired - Fee Related
-
2010
- 2010-04-20 KR KR1020100036589A patent/KR20110013200A/en not_active Application Discontinuation
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101079103A (en) * | 2007-06-14 | 2007-11-28 | 上海交通大学 | Human face posture identification method based on sparse Bayesian regression |
| CN101332362A (en) * | 2008-08-05 | 2008-12-31 | 北京中星微电子有限公司 | Interactive delight system based on human posture recognition and implement method thereof |
Also Published As
| Publication number | Publication date |
|---|---|
| CN101989326A (en) | 2011-03-23 |
| KR20110013200A (en) | 2011-02-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN101989326B (en) | Human posture recognition method and device | |
| CN107688391B (en) | Gesture recognition method and device based on monocular vision | |
| US7831087B2 (en) | Method for visual-based recognition of an object | |
| Liu et al. | Hand gesture recognition using depth data | |
| Xu et al. | Online dynamic gesture recognition for human robot interaction | |
| US20110025834A1 (en) | Method and apparatus of identifying human body posture | |
| CN108898063A (en) | A kind of human body attitude identification device and method based on full convolutional neural networks | |
| US20050271279A1 (en) | Sign based human-machine interaction | |
| CN103839040A (en) | Gesture identification method and device based on depth images | |
| Choi et al. | Human body orientation estimation using convolutional neural network | |
| CN104102904B (en) | A kind of static gesture identification method | |
| Wang et al. | Head pose estimation with combined 2D SIFT and 3D HOG features | |
| Munaro et al. | 3D flow estimation for human action recognition from colored point clouds | |
| CN110751097B (en) | Semi-supervised three-dimensional point cloud gesture key point detection method | |
| Zhang et al. | A survey on human pose estimation | |
| CN109325408A (en) | A kind of gesture judging method and storage medium | |
| CN112101208A (en) | Feature series fusion gesture recognition method and device for elderly people | |
| Chen et al. | Silhouette-based object phenotype recognition using 3D shape priors | |
| Sapp et al. | A Fast Data Collection and Augmentation Procedure for Object Recognition. | |
| Tian et al. | Human Detection using HOG Features of Head and Shoulder Based on Depth Map. | |
| Liang et al. | 3D motion trail model based pyramid histograms of oriented gradient for action recognition | |
| Kanaujia et al. | Part segmentation of visual hull for 3d human pose estimation | |
| Liang et al. | Egocentric hand pose estimation and distance recovery in a single RGB image | |
| CN113971801A (en) | Target multi-dimensional detection method based on four-type multi-modal data fusion | |
| Weber et al. | Leveraging Saliency-Aware Gaze Heatmaps for Multiperspective Teaching of Unknown Objects |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20150401 Termination date: 20170731 |