CN109902593A - A kind of gesture occlusion detection method and system based on Kinect - Google Patents
A kind of gesture occlusion detection method and system based on Kinect Download PDFInfo
- Publication number
- CN109902593A CN109902593A CN201910094427.5A CN201910094427A CN109902593A CN 109902593 A CN109902593 A CN 109902593A CN 201910094427 A CN201910094427 A CN 201910094427A CN 109902593 A CN109902593 A CN 109902593A
- Authority
- CN
- China
- Prior art keywords
- gesture
- kinect
- image
- color image
- posture
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Abstract
The gesture occlusion detection method and system based on Kinect that the present invention provides a kind of, comprising: S1, obtain depth image and color image respectively using Kinect；S2, by depth image binaryzation, and with color image carry out and operation, obtain the color image in only manpower region；The ratio in judgement of S3, the largest connected region of statistics and area of skin color are blocked with the presence or absence of gesture；S4, calculating acquisition presence is blocked or there is no prediction gesture postures when blocking.The present invention obtains the depth image and its corresponding color image of gesture by Kinect, colored Hand Gesture Segmentation is come out by image and operation, then the case where being blocked with the ratio in judgement for counting largest connected region and area of skin color with the presence or absence of gesture, and gesture posture is predicted by gesture algorithm for estimating, to solve the problems, such as that the gesture as caused by occlusion issue is unrecognized, realize that gesture posture can be effectively predicted out especially in the case where blocking by improving gesture identification accuracy rate.
Description
Technical field
The present invention relates to technical field of virtual reality, more particularly to a kind of gesture occlusion detection method based on Kinect
And system.
Background technique
With the rapid development of virtual reality, each neck has been widely used in using the gesture natural interaction of Kinect
Domain.But in the actual use process often exist caused due to occlusion issue gesture can not identified phenomenon, thus
The phenomenon that leading to system fault.
Therefore, it is badly in need of a kind of gesture occlusion detection method of high-accuracy, to realize the occlusion detection to gesture, Lai Tigao
The accuracy of gesture identification.
Summary of the invention
The gesture occlusion detection method and system based on Kinect that the object of the present invention is to provide a kind of, it is intended to solve existing
The unrecognized problem of the gesture as caused by occlusion issue in technology is realized and improves gesture identification accuracy rate, is effectively predicted out
Gesture posture.
To reach above-mentioned technical purpose, the gesture occlusion detection method based on Kinect that the present invention provides a kind of is described
Method the following steps are included:
S1, hand depth image and color image are obtained respectively using Kinect；
S2, by depth image binaryzation, and with color image carry out and operation, obtain the cromogram in only manpower region
Picture；
The ratio in judgement of S3, the largest connected region of statistics and area of skin color are blocked with the presence or absence of gesture；
S4, calculating acquisition presence is blocked or there is no prediction gesture postures when blocking.
Preferably, the method also includes:
Pre-training is carried out to data set using deep neural network, obtains to predict manpower appearance by manpower depth image
The deep neural network of state；
By data all in data set carry out a forward-propagating, 256 dimensional features that layer second from the bottom is extracted to
Amount is stored in feature database with corresponding posture coordinate data.
Preferably, the step S1 concrete operations are as follows:
Human depth's figure, cromogram and human skeleton information are obtained using Kinect；
The manpower part in depth map is split by the manpower depth information in skeleton information；
Corresponding hand color image is obtained using Zhang Zhengyou calibration method.
Preferably, the step S3 concrete operations are as follows:
It is YCbCr model by RGB model conversion, traverses the pixel in largest connected region Area, if the Cr of pixel points
For amount between [140,170], the quantity of skin pixel point Skin is then added 1 between [100,120] by Cb component；
Ratio D shared by skin pixel in largest connected region is calculated, is judged as if D is less than 80% and blocks, it is no
Then it is judged as that there is no block.
Preferably, the calculation formula of the ratio D is as follows:
It is preferably, described that there is no the calculation method of prediction gesture posture when blocking is as follows:
Hand depth image is zoomed into 96*96, is passed to deep neural network, feature extraction is carried out and obtains the spy of 256 dimensions
Levy vector v1；
Calculate feature vector v1With the Euclidean distance of each feature in feature database, the smallest n spy of Euclidean distance is acquired
Levy vector D1∈Rn×256And corresponding whole manpower posture P1∈Rn×m, then have:
Assuming that M is low-rank matrix, then predict that the calculating of gesture posture is as follows:
P=v1(D1)-1P1。
Preferably, the calculation method that there is prediction gesture posture when blocking is as follows:
Using the feature vector of r frame before deep neural network structure extraction history, obtain per one-dimensional original series:
X(0)=(x(0)(1),x(0)(2),....,x(0)(r))
Single order accumulation process is carried out to every one-dimensional initial data, obtains cumulative sequence:
X(1)=(x(1)(1),x(1)(2),....,x(1)(r))
Wherein, x(1)=x(0)(1),
System GM (1,1), albinism differential equation form are established by cumulative single order sequence are as follows:
Wherein a is the development coefficient of system, and b is grey actuating quantity, corresponding Grey Differential Equation are as follows:
a(0)(k)+az(1)(k)=b
Wherein, z(1)It (k) is average generation sequence,
To parameter a, b evaluation is obtained by least square method:
[a,b]T=(BTB)-1BTY
Wherein, Y=(x(0)(2),x(0)(3),....,x(0)(r))T,
It is in primary conditionIn the case where, obtain the data mould of single order Accumulating generation sequence
Type:
It is as follows to obtain original data storage power model:
It is v using the feature vector that initial data stores power model prediction present frame2, by v2Instead of feature vector v1, and be put into
Matrix fill-in calculating is carried out in matrix M, finally obtains the gesture posture p predicted when blocking.
The present invention also provides a kind of gesture sheltering detection system based on Kinect, the system comprises:
Image collection module, for obtaining hand depth image and color image respectively using Kinect；
Image operation module for by depth image binaryzation, and carries out with color image and operation, obtains only manpower
The color image in region；
Shadowing module, the ratio in judgement for counting largest connected region and area of skin color are hidden with the presence or absence of gesture
Gear；
Predict gesture module, acquisition presence is blocked or there is no prediction gesture postures when blocking for calculating.
The effect provided in summary of the invention is only the effect of embodiment, rather than invents all whole effects, above-mentioned
A technical solution in technical solution have the following advantages that or the utility model has the advantages that
Compared with prior art, the present invention obtains the depth image and its corresponding color image of gesture by Kinect,
Colored Hand Gesture Segmentation is come out by image and operation, is then with the ratio in judgement for counting largest connected region and area of skin color
No the case where being blocked there are gesture, and gesture posture is predicted by gesture algorithm for estimating, to solve due to occlusion issue
The caused unrecognized problem of gesture realizes that raising gesture identification accuracy rate can be effective especially in the case where blocking
Predict gesture posture.
Detailed description of the invention
Fig. 1 is a kind of gesture occlusion detection method flow signal based on Kinect provided in the embodiment of the present invention
Figure；
Fig. 2 is a kind of gesture occlusion detection total algorithm logical schematic provided in the embodiment of the present invention；
Fig. 3 is a kind of pre-training CNN schematic network structure provided in the embodiment of the present invention；
Fig. 4 is a kind of gesture algorithm for estimating logical schematic provided in the embodiment of the present invention；
Fig. 5 is a kind of gesture sheltering detection system structural block diagram based on Kinect provided in the embodiment of the present invention.
Specific embodiment
In order to clearly illustrate the technical characterstic of this programme, below by specific embodiment, and its attached drawing is combined, to this
Invention is described in detail.Following disclosure provides many different embodiments or example is used to realize different knots of the invention
Structure.In order to simplify disclosure of the invention, hereinafter the component of specific examples and setting are described.In addition, the present invention can be with
Repeat reference numerals and/or letter in different examples.This repetition is that for purposes of simplicity and clarity, itself is not indicated
Relationship between various embodiments and/or setting is discussed.It should be noted that illustrated component is not necessarily to scale in the accompanying drawings
It draws.Present invention omits the descriptions to known assemblies and treatment technology and process to avoid the present invention is unnecessarily limiting.
Be provided for the embodiments of the invention with reference to the accompanying drawing a kind of gesture occlusion detection method based on Kinect and
System is described in detail.
As shown in Figure 1, 2, the embodiment of the invention discloses a kind of gesture occlusion detection method based on Kinect, the side
Method the following steps are included:
S1, hand depth image and color image are obtained respectively using Kinect；
S2, by depth image binaryzation, and with color image carry out and operation, obtain the cromogram in only manpower region
Picture；
The ratio in judgement of S3, the largest connected region of statistics and area of skin color are blocked with the presence or absence of gesture；
S4, calculating acquisition presence is blocked or there is no prediction gesture postures when blocking.
The embodiment of the present invention obtains the corresponding color image of depth image of gesture by Kinect, by image with
Operation comes out colored Hand Gesture Segmentation, then using the ratio for counting largest connected region and area of skin color to determine whether in the presence of
The case where gesture is blocked, and gesture posture is predicted by gesture algorithm for estimating.
Pre-training is carried out to public data collection NYU using network structure shown in Fig. 3, obtaining can be by the depth of manpower
The deep neural network of image prediction manpower posture m.
The forward-propagating that all data in public data collection NYU are carried out to primary network, layer second from the bottom is extracted
256 dimensional feature vectors come are stored with corresponding posture coordinate data into feature database.
Images of gestures, including depth image and color image are obtained using Kinect, in addition also obtains skeleton letter
Breath, the manpower depth information in skeleton information that will acquire split the manpower part in depth image.
Corresponding color image is obtained using Zhang Zhengyou calibration method, by depth image carry out binaryzation and and color image into
Capable and operation, obtains the color image I in only manpower region.
For color image I, largest connected region Area in I is sought, and calculates the point number Skin of the skin pixel in Area,
Specific operation process is as follows:
It is YCbCr model by RGB model conversion, the pixel in Area is traversed, if the Cr component of pixel is in [140,170]
Between, colour of skin Skin quantity is then added 1 between [100,120] by Cb component.
Ratio D shared by skin pixel in largest connected region is calculated, is judged as if D is less than 80% and blocks, institute
The calculation formula of accounting rate D is as follows:
As shown in figure 4, that is, D > 0.8, then the manpower depth image that will acquire zoom to 96*96 if there is no blocking,
It is passed in the neural network of Fig. 3 again and carries out feature extraction, obtain the feature vector v of 256 dimensions1。
Calculate feature vector v1The Euclidean distance of each feature in feature database gone out with said extracted, acquires Euclidean distance
The smallest n feature vector D1∈Rn×256And corresponding whole manpower posture P1∈Rn×m:
Wherein, p ∈ R1×mIt is the estimation gesture attitude value of present frame, is unknown parameter.Assuming that M is a low-rank matrix, lead to
Solving matrix filling problem is crossed, the gesture estimation of present frame can be calculated:
P=v1(D1)-1P1
If there is D < 0.8 is blocked, then the feature vector of r frame before history is extracted using the network structure of Fig. 3, is obtained each
The original series X of dimension(0)=(x(0)(1),x(0)(2),....,x(0)(r)) the cumulative place of single order, is carried out to every one-dimensional initial data
Reason obtains cumulative sequence:
X(1)=(x(1)(1),x(1)(2),....,x(1)(r))
Wherein, x(1)=x(0)(1),
System GM (1,1), albinism differential equation form are established by cumulative single order sequence are as follows:
Wherein a is the development coefficient of system, and b is grey actuating quantity, corresponding Grey Differential Equation are as follows:
a(0)(k)+az(1)(k)=b
Wherein, z(1)It (k) is average generation sequence,
To parameter a, b evaluation is obtained by least square method:
[a,b]T=(BTB)-1BTY
Wherein, Y=(x(0)(2),x(0)(3),....,x(0)(r))T,
It is in primary conditionIn the case where, obtain the data mould of single order Accumulating generation sequence
Type:
It is as follows to obtain original data storage power model:
Feature vector using original data model prediction present frame is v2, by v2Instead of feature vector v1, and it is put into matrix
Matrix fill-in calculating is carried out in M, finally obtains the gesture posture p predicted when blocking.
The embodiment of the present invention by Kinect obtain gesture depth image and its corresponding color image, by image with
Operation comes out colored Hand Gesture Segmentation, then whether there is gesture with the ratio in judgement for counting largest connected region and area of skin color
The case where being blocked, and gesture posture is predicted by gesture algorithm for estimating, to solve the gesture as caused by occlusion issue
Unrecognized problem realizes that gesture can be effectively predicted out especially in the case where blocking by improving gesture identification accuracy rate
Posture.
As shown in figure 5, the embodiment of the invention also discloses a kind of gesture sheltering detection system based on Kinect, the system
System includes:
Image collection module, for obtaining hand depth image and color image respectively using Kinect；
Image operation module for by depth image binaryzation, and carries out with color image and operation, obtains only manpower
The color image in region；
Shadowing module, the ratio in judgement for counting largest connected region and area of skin color are hidden with the presence or absence of gesture
Gear；
Predict gesture module, acquisition presence is blocked or there is no prediction gesture postures when blocking for calculating.
The corresponding color image of depth image that gesture is obtained by Kinect, by image and operation by colored hand
Gesture is split, then using the ratio for counting largest connected region and area of skin color to determine whether being blocked there are gesture
Situation, and gesture posture is predicted by gesture algorithm for estimating.
The foregoing is merely illustrative of the preferred embodiments of the present invention, is not intended to limit the invention, all in essence of the invention
Made any modifications, equivalent replacements, and improvements etc., should all be included in the protection scope of the present invention within mind and principle.
Claims (8)
1. a kind of gesture occlusion detection method based on Kinect, which is characterized in that the described method comprises the following steps:
S1, hand depth image and color image are obtained respectively using Kinect；
S2, by depth image binaryzation, and with color image carry out and operation, obtain the color image in only manpower region；
The ratio in judgement of S3, the largest connected region of statistics and area of skin color are blocked with the presence or absence of gesture；
S4, calculating acquisition presence is blocked or there is no prediction gesture postures when blocking.
2. a kind of gesture occlusion detection method based on Kinect according to claim 1, which is characterized in that the method
Further include:
Pre-training is carried out to data set using deep neural network, obtains to predict manpower posture by manpower depth image
Deep neural network；
By data all in data set carry out a forward-propagating, 256 dimensional feature vectors that layer second from the bottom is extracted with
In corresponding posture coordinate data deposit feature database.
3. a kind of gesture occlusion detection method based on Kinect according to claim 1, which is characterized in that the step
S1 concrete operations are as follows:
Human depth's figure, cromogram and human skeleton information are obtained using Kinect；
The manpower part in depth map is split by the manpower depth information in skeleton information；
Corresponding hand color image is obtained using Zhang Zhengyou calibration method.
4. a kind of gesture occlusion detection method based on Kinect according to claim 1, which is characterized in that the step
S3 concrete operations are as follows:
It is YCbCr model by RGB model conversion, the pixel in largest connected region Area is traversed, if the Cr component of pixel exists
Between [140,170], the quantity of skin pixel point Skin is then added 1 between [100,120] by Cb component；
Ratio D shared by skin pixel in largest connected region is calculated, is judged as if D is less than 80% and blocks, otherwise sentence
Break as there is no block.
5. a kind of gesture occlusion detection method based on Kinect according to claim 4, which is characterized in that the ratio
The calculation formula of D is as follows:
6. a kind of gesture occlusion detection method based on Kinect according to claim 2, which is characterized in that described not deposit
The calculation method of prediction gesture posture when blocking is as follows:
Hand depth image is zoomed into 96*96, is passed to deep neural network, carry out feature extraction obtain the features of 256 dimensions to
Measure v1；
Calculate feature vector v1With the Euclidean distance of each feature in feature database, the smallest n feature vector of Euclidean distance is acquired
D1∈Rn×256And corresponding whole manpower posture P1∈Rn×m, then have:
Assuming that M is low-rank matrix, then predict that the calculating of gesture posture is as follows:
P=v1(D1)-1P1。
7. a kind of gesture occlusion detection method based on Kinect according to claim 6, which is characterized in that the presence
The calculation method of prediction gesture posture when blocking is as follows:
Using the feature vector of r frame before deep neural network structure extraction history, obtain per one-dimensional original series:
X(0)=(x(0)(1),x(0)(2),....,x(0)(r))
Single order accumulation process is carried out to every one-dimensional initial data, obtains cumulative sequence:
X(1)=(x(1)(1),x(1)(2),....,x(1)(r))
Wherein, x(1)=x(0)(1),
System GM (1,1), albinism differential equation form are established by cumulative single order sequence are as follows:
Wherein a is the development coefficient of system, and b is grey actuating quantity, corresponding Grey Differential Equation are as follows:
a(0)(k)+az(1)(k)=b
Wherein, z(1)It (k) is average generation sequence,
To parameter a, b evaluation is obtained by least square method:
[a,b]T=(BTB)-1BTY
Wherein, Y=(x(0)(2),x(0)(3),....,x(0)(r))T,
It is in primary conditionIn the case where, obtain the data model of single order Accumulating generation sequence:
It is as follows to obtain original data storage power model:
It is v using the feature vector that initial data stores power model prediction present frame2, by v2Instead of feature vector v1, and it is put into matrix
Matrix fill-in calculating is carried out in M, finally obtains the gesture posture p predicted when blocking.
8. a kind of gesture sheltering detection system based on Kinect, which is characterized in that the system comprises:
Image collection module, for obtaining hand depth image and color image respectively using Kinect；
Image operation module for by depth image binaryzation, and carries out with color image and operation, obtains only manpower region
Color image；
Shadowing module, the ratio in judgement for counting largest connected region and area of skin color are blocked with the presence or absence of gesture；
Predict gesture module, acquisition presence is blocked or there is no prediction gesture postures when blocking for calculating.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910094427.5A CN109902593A (en) | 2019-01-30 | 2019-01-30 | A kind of gesture occlusion detection method and system based on Kinect |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910094427.5A CN109902593A (en) | 2019-01-30 | 2019-01-30 | A kind of gesture occlusion detection method and system based on Kinect |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109902593A true CN109902593A (en) | 2019-06-18 |
Family
ID=66944409
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910094427.5A Pending CN109902593A (en) | 2019-01-30 | 2019-01-30 | A kind of gesture occlusion detection method and system based on Kinect |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109902593A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111259749A (en) * | 2020-01-10 | 2020-06-09 | 上海大学 | Real-time human body posture recognition method in complex environment based on bidirectional LSTM |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103400110A (en) * | 2013-07-10 | 2013-11-20 | 上海交通大学 | Abnormal face detection method in front of ATM (automatic teller machine) |
| CN105930784A (en) * | 2016-04-15 | 2016-09-07 | 济南大学 | Gesture recognition method |
| CN107871180A (en) * | 2017-10-17 | 2018-04-03 | 淮阴工学院 | GM based on Newton-Cotes formulas tectonic setting value（1,1）Model prediction method |
| CN108460319A (en) * | 2017-02-22 | 2018-08-28 | 浙江宇视科技有限公司 | Abnormal face detecting method and device |
| CN109033978A (en) * | 2018-06-28 | 2018-12-18 | 济南大学 | A kind of CNN-SVM mixed model gesture identification method based on error correction strategies |
-
2019
- 2019-01-30 CN CN201910094427.5A patent/CN109902593A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103400110A (en) * | 2013-07-10 | 2013-11-20 | 上海交通大学 | Abnormal face detection method in front of ATM (automatic teller machine) |
| CN105930784A (en) * | 2016-04-15 | 2016-09-07 | 济南大学 | Gesture recognition method |
| CN108460319A (en) * | 2017-02-22 | 2018-08-28 | 浙江宇视科技有限公司 | Abnormal face detecting method and device |
| CN107871180A (en) * | 2017-10-17 | 2018-04-03 | 淮阴工学院 | GM based on Newton-Cotes formulas tectonic setting value（1,1）Model prediction method |
| CN109033978A (en) * | 2018-06-28 | 2018-12-18 | 济南大学 | A kind of CNN-SVM mixed model gesture identification method based on error correction strategies |
Non-Patent Citations (4)
| Title |
|---|
| ZHOU XIAOYAN ET AL: "《the design of man-machine finger-gussing game based on the hand gesture of the intellisense》", 《2016 INTERNATIONAL CONFERENCE ON CYBERWORLDS(CW)》 * |
| 王长有等: "《基于灰色预测和Mean-Shift的抗遮挡跟踪算法》", 《控制工程》 * |
| 肖寒君: "《基于深度传感器的手势追踪系统》", 《中国优秀硕士学位论文全文数据库信息科技辑》 * |
| 胡宏宇等: "《基于视频监控的运动目标跟踪算法》", 《北京工业大学学报》 * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111259749A (en) * | 2020-01-10 | 2020-06-09 | 上海大学 | Real-time human body posture recognition method in complex environment based on bidirectional LSTM |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110276316B (en) | Human body key point detection method based on deep learning | |
| CN106874894B (en) | Human body target detection method based on regional full convolution neural network | |
| CN109816641B (en) | Multi-scale morphological fusion-based weighted local entropy infrared small target detection method | |
| CN104392468B (en) | Based on the moving target detecting method for improving visual background extraction | |
| CN110378844A (en) | Motion blur method is gone based on the multiple dimensioned Image Blind for generating confrontation network is recycled | |
| CN103268481B (en) | A kind of Text Extraction in complex background image | |
| CN113379771B (en) | Hierarchical human body analysis semantic segmentation method with edge constraint | |
| CN106529419B (en) | The object automatic testing method of saliency stacking-type polymerization | |
| CN106599805A (en) | Supervised data driving-based monocular video depth estimating method | |
| CN103735269B (en) | A kind of height measurement method followed the tracks of based on video multi-target | |
| CN107679503A (en) | A kind of crowd's counting algorithm based on deep learning | |
| CN109558815A (en) | A kind of detection of real time multi-human face and tracking | |
| CN109948593A (en) | Based on the MCNN people counting method for combining global density feature | |
| CN109858389A (en) | Vertical ladder demographic method and system based on deep learning | |
| CN102194443A (en) | Display method and system for window of video picture in picture and video processing equipment | |
| CN111860398A (en) | Remote sensing image target detection method and system and terminal equipment | |
| CN110197185B (en) | Method and system for monitoring space under bridge based on scale invariant feature transform algorithm | |
| CN111160111A (en) | Human body key point detection method based on deep learning | |
| CN109902613A (en) | A kind of human body feature extraction method based on transfer learning and image enhancement | |
| CN108805057A (en) | A kind of SAR image oil depot area detection method based on joint significance analysis | |
| CN109146925A (en) | Conspicuousness object detection method under a kind of dynamic scene | |
| CN116311412A (en) | Mask wearing detection method integrating 3D attention mechanism and cavity convolution | |
| CN106056078A (en) | Crowd density estimation method based on multi-feature regression ensemble learning | |
| CN117197388A (en) | Live-action three-dimensional virtual reality scene construction method and system based on generation of antagonistic neural network and oblique photography | |
| CN115272691A (en) | Training method, recognition method and equipment for steel bar binding state detection model |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190618 |
|
| RJ01 | Rejection of invention patent application after publication |