CN108182728A - A kind of online body-sensing three-dimensional modeling method and system based on Leap Motion - Google Patents
A kind of online body-sensing three-dimensional modeling method and system based on Leap Motion Download PDFInfo
- Publication number
- CN108182728A CN108182728A CN201810054234.2A CN201810054234A CN108182728A CN 108182728 A CN108182728 A CN 108182728A CN 201810054234 A CN201810054234 A CN 201810054234A CN 108182728 A CN108182728 A CN 108182728A
- Authority
- CN
- China
- Prior art keywords
- gesture
- data
- hmm
- dimensional modeling
- model
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0487—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
- G06F3/0488—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures
- G06F3/04883—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures for inputting data by handwriting, e.g. gesture or text
Abstract
The invention discloses a kind of online body-sensing three-dimensional modeling method and system based on Leap Motion, method is predefined including gesture information and gesture is trained and identifies；Gesture information is predefined including defining three-dimensional modeling gesture, gesture data pretreatment and structure gesture data model；Gesture training and identification include building gesture training sample set, structure HMM gestures training pattern and gesture identification three-dimensional modeling.System includes software interactive unit, collecting unit, Real Time Communication Unit, data processing unit, storage unit, computing unit and execution unit；It is operated present invention is primarily based on technical principles such as virtual reality, human-computer interaction and computer graphics user to be assisted to carry out the online three-dimensional modeling based on browser end by body feeling interaction mode.
Description
Technical field
A kind of existed the invention belongs to virtual reality and human-computer interaction technique field more particularly to based on Leap Motion
Wire body sense three-dimensional modeling method and system.
Background technology
With the continuous development of information technology, CAD (Computer Aided Design) software by
Gradually developed from traditional C/S (Client/Server) frameworks to B/S (Browser/Server), directly transported in browser end
Row supports the online three-dimensional modeling in high in the clouds and product design, and distributed collaboration design, information communication and knowledge can be provided for enterprise and is total to
The service enjoyed.At present, designer still relies on the three-dimensional of " mouse+keyboard " with the interactive mode of online 3 d modeling software and builds
Mould mode, additional three-dimensional modeling need the professional knowledge of long period to learn, cause the study of existing three-dimensional modeling interactive mode into
This is higher, is unfavorable for reducing three-dimensional modeling threshold, layman is attracted to carry out three-dimensional modeling and product design.Such as towards machine
The online 3 d modeling software Onshape of tool design and the online 3 d modeling software innovated towards 3D printing masses
TinkerCAD, although modeling function is very powerful, especially Onshape is even more to support Parametric three-dimensional modeling and Virtual assemble, so
And above two software does not support that somatosensory device accesses, thus its three-dimensional modeling interactivity is not powerful enough, three-dimensional modeling and production
Product design learning threshold is still higher.
In recent years, along with the birth of virtual reality technology, limit of the mouse with keyboard to human-computer interaction is effectively alleviated
System more can intuitively reflect the three-dimensional modeling thought of designer, the human-computer interaction three-dimensional modeling side for realize nature, immersing
Formula.Hu Hong etc. exists《Leap Motion Critical point model hand Attitude estimation methods》In one text, propose a kind of based on Leap
The Critical point model Attitude estimation method of Motion establishes key point hand skeleton pattern based on hand-type structure, and which part is crucial
Dot position information can be obtained by Leap Motion sensors, and the space coordinate of remaining key point can be calculated according to model, right whereby
Gesture posture is estimated.However, key point hand skeleton pattern is relatively simple constructed by this method, and the model merely with
Leap Motion capture hand point spatial positional information, not using normal vector and rate information, thus to not homochirality certainly
Adaptability is poor.In addition, this method realizes that platform is C/S (Client/Server) framework client software, can not be suitable for
Line 3 d modeling software.Liu Quan etc. exists《Adaptive dynamic hand gesture recognition based on Leap Motion sensors》In one text, propose
A kind of gesture identification method that dynamic gesture track self study is carried out using hidden Markov model (HMM), defines small letter 26
English alphabet handwriting trace utilizes Leap Motion sensors extraction gesture artis institute as 26 kinds of Pre-defined gesture classifications
Streak track data, establish sample set, and sample set is trained using HMM, can Dynamic Recognition Pre-defined gesture, and know
Other average accuracy is up to 92.95%.However, the data that this method acquired and inputted HMM are only tracing point space bit confidence
Breath, not using normal vector and rate information, and HMM self-learning algorithms are not optimized, thus recognition correct rate still have compared with
Big room for promotion, and this method is still realized based on local side, and online gesture identification is not implemented.Wu Fuli etc. exists《It is based on
The virtual crops Design of man-machine Conversation of Leap Motion》In one text, propose a kind of based on the online of Leap Motion
Virtual crops body feeling interaction method acquires gesture attitude data by Leap Motion and is encapsulated as JSON files, by
WebSocket agreements are uploaded to server, and the virtual crops man-machine interactive system tune for being developed based on HTML5 at any time in real time
With.The system is developed based on WebGL, can carry out three-dimensional visible to all kinds of crops by gesture data based entirely on browser
Change interactive operation.However, this method operates virtual crops threedimensional model merely with acquired gesture data, it is not implemented
Gesture self study, thus its performance accuracy is relatively low, and precision can not be promoted with the increase of system frequency of use.The gloomy grade of Japanese plum exists
Patent《A kind of body feeling interaction quick three-dimensional modeling auxiliary system and its method》In, disclose a kind of body-sensing three-dimensional interactive modeling
Method and system receives depth image data capture user modeling using Kinect sensor and is intended to, including types of models, model
Size, and the corresponding API of SolidWorks is called to realize the three-dimensional modeling of basic body in the software.However, this method
On the one hand it can only realize the interactive three-dimensional modeling based on SolidWorks, can not realize online three-dimensional modeling, on the other hand
User can not interact created threedimensional model formula operation, and including scaling, translating, rotating, man-machine interaction is not strong enough
Greatly.
Invention content
The present invention is poor for being supported based on the 3 d modeling software of Web the access of body-sensing hardware device, causes in Web rings
The problem of three-dimensional modeling process man-machine interaction is insufficient is carried out in border, proposes a kind of online body-sensing three based on Leap Motion
Tie up modeling method and system, by defining different types of gesture, create 6 kinds of basic bodies, and to create primitives into
Row translation, rotation, zoom operations.Great amount of samples is acquired for different gestures, builds gesture training sample set, and by hidden Ma Er
Can husband's model (HMM) be trained, realize gesture identification three-dimensional modeling.Moreover, it relates to a kind of be based on Leap
The online body-sensing 3 d modeling system of Motion, by the vertical virtual light curtain of Leap Motion sensors, utilizes right hand forefinger
Mouse operation and control computer mode is simulated, realizes the online 3 d modeling software of motion sensing manipulation.
Technical solution is used by the method for the present invention：A kind of online body-sensing three-dimensional modeling based on Leap Motion
Method, which is characterized in that include the following steps：
Step 1：Gesture information predefines；
Step 1.1：Define three-dimensional modeling gesture；
Definition of gesture is carried out to creating basic body model and model manipulation mode；
Step 1.2：Gesture data pre-processes；
Leap Motion sensors are captured gesture space coordinate data, vector data and speed data to carry out in advance
Processing, acquisition as hidden Markov model (HMM) can input and complete effective gesture data of gesture identification function；
Step 1.3：Build gesture data model；
The gesture data model trained to effective gesture data structure of acquisition for HMM.
Step 2：Gesture training and identification；
Step 2.1：Build gesture training sample set；
The gesture data model trained to effective gesture data structure of acquisition for HMM；
Step 2.2：Build HMM gesture training patterns；
Sample set is built using effective gesture data of all captures, each gesture is trained for HMM；
Step 2.3：Gesture identification threedimensional model is built, for identifying certain given gesture sequence, and corresponding three-dimensional is performed and builds
Modulo operation.
Preferably, in step 1.1, the basic body includes square, cuboid, cylinder, cone, ball
Body, torus；The model manipulation mode includes translation, rotate around X-axis, rotated around Y-axis, rotate about the z axis, scales.
Preferably, step 1.2, gesture data is pre-processed including the valid frame screening technique based on centre of the palm rate, effectively
Frame three-dimensional coordinate data normalization processing method and available frame count are according to method for resampling；The valid frame based on centre of the palm rate
Screening technique judges whether captured frame data are that palm rate travel is thin less than 5mm/s by setting centre of the palm rate-valve value
Micromotion causes, and then judges the beginning of gesture with terminating state；The valid frame three-dimensional coordinate data normalization processing method
Make unified specificationization by the three-dimensional coordinate data to acquired valid frame to handle, effectively meet HMM model observation for integer
Requirement；The available frame count is according to method for resampling by set point away from threshold value, it is ensured that in each gesture between available frame count strong point
Away from approximately equal.
It is comprising three-dimensional in effective frame data by structure preferably, building gesture data model described in step 1.3
Coordinate data and the Formal Representation model of corresponding normal vector or direction vector, the training as HMM gesture training patterns are defeated
Enter data.
It is for 6 kinds of basic bodies and 3 kinds of moulds preferably, building gesture training sample set described in step 2.1
Type mode of operation, altogether 11 kinds operation, respectively to each operating gesture data collecting sample several, build gesture training sample
Collection.
It is to utilize preceding backward algorithm in HMM theories preferably, building HMM gesture training patterns described in step 2.2,
According to the hidden state corresponding to given gesture training sample sequence and each gesture training sample, structure HMM gesture instructions
Practice model.
It is HMM gestures training mould constructed by judgement preferably, building gesture identification threedimensional model described in step 2.3
Whether type is the best match model of certain given gesture sequence, and then realizes the identification of certain gestures and corresponding three-dimensional modeling behaviour
Make.
Technical solution is used by the system of the present invention：A kind of online body-sensing three-dimensional modeling based on Leap Motion
System, it is characterised in that：Including software interactive unit, collecting unit, Real Time Communication Unit, data processing unit, storage unit,
Computing unit and execution unit；
The software interactive unit is used to simulate the online 3 d modeling software of mouse operation and control by right hand forefinger；
The collecting unit is for collecting sample gesture data and needs the gesture data judged；
The Real Time Communication Unit is used to acquired gesture data being encapsulated as JSON files, is assisted by real-time Communication for Power Network
View is uploaded to Cloud Server；
The data processing unit is used to carry out acquired gesture data the pretreatment for meeting HMM model training；
The storage unit builds gesture training sample set for storing pretreated gesture data；
The computing unit is used to gesture training sample set inputting HMM gesture training patterns, and gesture is carried out to the model
Training can effectively identify 11 kinds of corresponding gesture sequences of three-dimensional modeling operation；
The execution unit performs corresponding three-dimensional modeling behaviour in a browser for identifying the gesture sequence that user gives
Make.
Beneficial effects of the present invention can be summarized as at following 3 points：
(1) Gesture Recognition with CAD (CAD) is combined, three is carried out by predefined 11 kinds of gestures
Modelling operability is tieed up, effectively reduces three-dimensional modeling and the study threshold of product design, and hanging down using Leap Motion sensors
Straight virtual light curtain, simulates the online 3 d modeling software of mouse operation and control, improves the man-machine interaction of three-dimensional modeling；
(2) it is poor to hardware access support for web application, it is unfavorable for somatosensory device and online 3 d modeling software
The problem of into row data communication, builds the gesture data real-time Transmission frame based on WebSocket agreements, by Leap Motion
Captured gesture data is encapsulated as JSON files, and it is soft to be uploaded to by WebSocket agreements the online three-dimensional modeling of deployment in real time
The Cloud Server of part realizes that user and online three-dimensional modeling are soft, it can be achieved that the real-time Communication for Power of gesture data and Cloud Server
The body feeling interaction of part；
(3) acquired gesture data is pre-processed, and builds gesture data model and gesture training sample set, stored
Effective gesture data of gesture identification function can be inputted and completed as HMM, effectively prevent being less than because of palm rate travel
The slight movement of 5mm/s causes to start gesture different with the erroneous judgement for terminating state and different user hand-type and operating habit
Lead to problems such as gesture identification accuracy rate relatively low.Gesture data amount is not effectively reduced only by pre-treatment step, meet network
Requirement of the environment to data transmission high efficiency, real-time, and the speed and accuracy rate of gesture identification are effectively improved, it realizes
Good man-machine interaction experience sense.
Description of the drawings
Fig. 1 is the method schematic of the embodiment of the present invention；
Fig. 2 is that basic body model and model manipulation schematic diagram are created in the embodiment of the present invention；
Fig. 3 (a) is virtual three-dimensional space coordinate system signal constructed by the Leap Motion sensors of the embodiment of the present invention
Figure；
Fig. 3 (b) captures certain frame data schematic diagram by the Leap Motion sensors of the embodiment of the present invention；
Fig. 4 is the HMM gesture training pattern schematic diagrames of the embodiment of the present invention；
Fig. 5 is the system architecture diagram of the embodiment of the present invention；
Fig. 6 is that operation chart is clicked in the simulation of the embodiment of the present invention by mouse right button；
Fig. 7 (a) is " mouse+keyboard " mode three-dimensional modeling human-computer interaction schematic diagram in the embodiment of the present invention；
Fig. 7 (b) is Leap Motion body-sensing three-dimensional modeling human-computer interaction schematic diagrames in the embodiment of the present invention；
Fig. 8 is that gesture data uploads rendering frame per second (FPS) variation tendency schematic diagram in real time in the embodiment of the present invention；
Fig. 9 (a) is the gesture identification confusion matrix based on gesture path Similarity Match Method in the embodiment of the present invention；
Fig. 9 (b) is the gesture identification confusion matrix based on HMM model in the embodiment of the present invention.
Specific embodiment
Understand for the ease of those of ordinary skill in the art and implement the present invention, with reference to the accompanying drawings and embodiments to this hair
It is bright to be described in further detail, it should be understood that implementation example described herein is merely to illustrate and explain the present invention, not
For limiting the present invention.
Referring to Fig.1, a kind of online body-sensing three-dimensional modeling method based on Leap Motion provided by the invention, including with
Lower step：
Step 1：Gesture information predefines；
It is modeled for 6 kinds of basic bodies, including square, cuboid, cylinder, cone, sphere, torus, with
And 3 kinds of model manipulation modes, including translating, rotating (around 3 reference axis), scaling, 11 kinds of operations altogether, by defining 11 kind three
Dimension modeling gesture will be carried out the process of three-dimensional modeling using mouse operation and control, be converted to and carry out three-dimensional modeling by gesture, lead to simultaneously
It crosses and gesture data is captured to Leap Motion sensors pre-processes, effectively reducing gesture data amount, adapt to network and pass
It is defeated while shift the scenes, it ensure that the validity of data, and pass through and build the gesture that gesture data model storage has been subjected to pretreatment
Data, structure formalization Data Storage Models, convenient for its input as the training of HMM model gesture identification.
(1) three-dimensional modeling gesture is defined；
11 kinds of three-dimensional modeling gestures are defined, as shown in Fig. 2, referring to table 1 to the process description of 11 kinds of gestures.
1 three-dimensional modeling gesture of table describes
(2) gesture data pre-processes；
Gesture data pretreatment is carried out by acquiring gesture data to Leap Motion sensors at screening and normalization
Reason, can both reduce gesture data amount with meet gesture data be uploaded in real time Cloud Server processing efficient, real-time will
It asks, and HMM model can be met to requirement of the input data for integer.As Fig. 3 (a) show Leap Motion sensors institute
Virtual three-dimensional space coordinate system is built, sensor captures the information such as spatial value, vector, the rate in frame data and is based on
The coordinate system calculates.Certain frame data schematic diagram is captured as Fig. 3 (b) show Leap Motion sensors.
1. the valid frame screening technique based on centre of the palm rate；
Leap Motion sensor data acquisitions frame per second is 200fps, and equipment is more sensitive, thus when user's subjective consciousness
When driving the hand to remain static, the slight movement data of hand can also be captured by equipment, lead to the beginning and end of gesture
State can not be identified.Therefore, must setting rate-valve value ε judge that a certain frame is captured produced by whether data be gesture motion
Data, meanwhile, by the setting of the threshold value, may be such that the beginning of computer identification gesture and end state, i.e. the moment frame
Data are judged as in vain, and gesture remains static.Rule of thumb, ε is set as 5mm/s.If in the i-th frame data, right hand HRi
Centre of the palm PRiRate isWhen the rate is less than threshold epsilon, it is believed that the right hand is stationary state, and the frame data are invalid, when the rate is big
In threshold epsilon, it is believed that the right hand is kept in motion, and the frame data are effective.
In addition, define the decision rule that gesture starts and terminates state.Rule of thumb, speed difference threshold value is set | ε | be
9mm/s judges gesture to start or terminating state according to the speed difference of the (i-1)-th frame and the i-th frame.
The processing method 2. valid frame three-dimensional coordinate data is standardized；
The size of track in gesture data is captured for unified each valid frame, avoids differing influence because of gesture path size
Gesture identification effect must make normalization processing to the D coordinates value of track in all available frame counts evidence.With right hand HRiCentre of the palm PRi
Space coordinateFor, illustrate processing procedure of standardizing.To each gesture path, find out respectively its
X-axis, Y-axis, the maximal and minmal value on Z axis, and compression ratio of the track in three reference axis is calculated using the value, according to the pressure
Contracting by track than being transformed into 10 × 10 × 10 space (unit：mm).
It in addition, need to be to being acquiredCarry out rounding operation, it is ensured that coordinate data is distributed in three-dimensional
On the integer crosspoint of reference axis, and the value after rounding is assigned to again
3. available frame count is according to method for resampling；
To ensure contained effective frame data point spacing approximately equal in each gesture, using equidistant method for resampling to passing through
The gesture sample of normalization processing carries out resampling.By taking the right hand centre of the palm as an example, by empirically determined point away from threshold value lmax=0.03mm,
It calculatesWithThe distance betweenJudgeWhether l is more thanmax：IfThen retainIt calculatesWithDistanceAnd continue to judge whether it is more than lmax；IfThen deleteIt calculatesWithDistanceAnd judge whether it is more than lmax.Iterative calculation is until the last one point, it is ensured that in each gesture effectively
Frame number strong point is distributed approaches uniformity.
(3) gesture data model is built；
To pretreated valid frame gesture data, Formal Representation model, the input number as follow-up HMM model are built
According to referring to Fig. 3 (b).Define the i-th frame gesture HiIt is expressed as：
Hi={ HLi,HRi}
Wherein HLiRepresent Left-hand gesture data, HRiRepresent right-hand gesture data, the two is represented by：
Wherein P represents centre of the palm data, F1,F2,F3,F4,F5Thumb, forefinger, middle finger, the third finger and little finger of toe are represented successively
Finger tip data.
For palm data, by taking left-hand palm as an example, PLiIt is represented by：
Wherein,Represent left hand centre of the palm coordinate,Represent left hand centre of the palm normal vector,Represent the left hand palm
Heart rate.
For finger data, by taking left hand thumb as an example,It is represented by：
Wherein,Represent left hand thumb tip coordinate,Represent left hand thumb tip direction vector.
Step 2：Gesture training and identification；
By building gesture training sample set, the user gesture sample that Leap Motion sensors are acquired is converged
Always, structure can be as the gesture sample set of HMM model input data, while by building HMM gesture training patterns, to gesture
Different classes of gesture carries out classification processing in sample set, and then the gesture data newly inputted can be carried out at correct classification
Gesture information is converted into three-dimensional modeling operation information by reason, i.e. gesture identification, and pass through gesture identification three-dimensional modeling step, real
Existing corresponding threedimensional model creates and operating function.
(1) gesture training sample set is built；
If the single gesture sample for completing a certain gesture isT represents the total of valid frame
Number.Then establish the sample set of the gestureN represents sample size.11 samples are established in total
SetThe input data of gesture identification training is carried out as HMM model.
(2) HMM gesture training patterns are built；
HMM model structure is carried out to gesture in 11 in gesture training sample set, i.e., according to gesture sample data, for every
Kind gesture establishes corresponding HMM model.The process can be considered as gesture learning process, by known gesture sample sequence
And corresponding gesture classification, train the corresponding initial vector π=[π of each HMM modeli], state-transition matrix A=
[aij] and confusion matrix B=[bjk], which uses Baum-Welch algorithms, as shown in Figure 4.By gesture training sample set
In all gesture sequence data corresponding to per class gesture sequentially input the model, whenever inputting new gesture sequence data, just
Initial vector, state-transition matrix and confusion matrix are reevaluated, calculation formula is as follows：
Wherein, δt(i, j) is forward variable, and it is q to represent t moment stateiAnd t+1 moment state is qjProbability, ψt(i)
For posterior probability, when representing given gesture sample sequence and HMM, t moment state is the probability of i.For iterative process is avoided to be absorbed in
Local optimum and globally optimal solution can not be obtained, in every time iterative calculation π, A, B after, using simulated annealing (SA) to obscuring
Matrix B=[bjk] be modified, its step are as follows：
1. it initializes：Initial temperature of annealing T0, cooling ratio k, final temperature Te, condition of convergence ρ,
2. set cooling function Th+1=kTh, k ∈ [0,1]；
3. generating N × M independently of each other and meeting the stochastic variable X of normal distribution, E (X)=0, variance it is expectedIt enablesIfThen enable
It is 4. rightIt is normalized,
5. judge whether P (O | λ) meets condition of convergence ρ or whether reached final temperature Te：If satisfied, then algorithm knot
Beam takes current P (O | λ) as optimal solution；If not satisfied, then going to step 2., continue iteration.
By iterating to calculate 3 matrixes, a globally optimal solution of the HMM model can be obtained, i.e., sample is trained by gesture
This collection trains the HMM gesture training patterns corresponding to each gesture.
(3) gesture identification three-dimensional modeling；
After HMM gesture training patterns are successfully built, respectively by the corresponding HMM gestures training pattern of 11 kinds of gestures calculate by
The probability of the new gesture sequence data of model generation observable, chooses the gesture classification corresponding to maximum probability, i.e., successful real
Existing gesture identification.The process uses the forwards algorithms in HMM theories, and recursion seeks each probability value.Expressed by gesture to be obtained
After user's three-dimensional modeling is intended to, then corresponding three-dimensional modelling operability is performed, realize the operation that three-dimensional modeling is carried out by gesture.
See Fig. 5, a kind of online body-sensing 3 d modeling system based on Leap Motion provided by the invention, including soft
Part interactive unit, collecting unit, Real Time Communication Unit, data processing unit, storage unit, computing unit and execution unit；
Software interactive unit：For simulating the online 3 d modeling software of mouse operation and control by right hand forefinger；
Collecting unit：For collecting sample gesture data and need the gesture data judged；
Real Time Communication Unit：For acquired gesture data to be encapsulated as JSON files, by real-time Communication for Power Network agreement
It is uploaded to the Cloud Server for disposing online 3 d modeling software；
Data processing unit：For carrying out the pretreatment for meeting HMM model training to acquired gesture data；
Storage unit：For storing pretreated gesture data, and build gesture training sample set；
Computing unit：For gesture training sample set to be inputted HMM gesture training patterns, gesture instruction is carried out to the model
Practice, can effectively identify 11 kinds of corresponding gesture sequences of three-dimensional modeling operation；
Execution unit：For identifying gesture sequence that user gives, and held in the web browser of the subscriber computer
The corresponding three-dimensional modeling operation of row.
User PC ends connect Leap Motion sensors, and pass through browser access be deployed in it is online with server
3 d modeling software.Collecting unit acquires user gesture information and operates the action message of software in real time, and is sent in real time
Communication unit；Acquired Information encapsulation is lightweight data storage file-JSON files by Real Time Communication Unit, to adapt to net
The light-weighted requirement to data of network Ambient Transfer, and pass through 6437 ends at user PC ends by real-time communication protocol WebSocket
Mouth carries out data real-time, interactive with Cloud Server, i.e. Leap Motion sensors capture data and are uploaded to Cloud Server in real time；
Software interactive unit disposes online 3 d modeling software and Leap.js library files is configured, and by the library file and Leap
Motion sensors carry vertical virtual light curtain, simulate the clicking operation of right mouse button, user is facilitated directly to be manipulated by gesture
Online 3 d modeling software；Data processing unit parses JSON files, and all data are regular for gesture training sample set, full
The requirement that sufficient HMM model standardizes to training data；Storage unit is using the processed hand of high in the clouds relational data library storage
Gesture training sample set；Computing unit is based on HMM gestures training pattern and develops, the gesture concentrated by receiving gesture training sample
Data simultaneously carry out repetition training to it, and online 3 d modeling software is enable effectively to identify 11 kinds of three-dimensional modeling gestures；Perform list
Member response specific three dimensional modeling gesture, and perform corresponding three-dimensional modeling operation.
It is illustrated in addition, clicking operation by mouse right button for the simulation in software interactive unit, as shown in Figure 6.
The vertical virtual light curtain carries attribute for Leap Motion sensors, is located at three-dimensional constructed by Leap Motion sensors sit
On the XOY plane for marking system.The auxiliary magnet that navigates is the X of user's right hand index finger tip, the mapping of Y coordinate on a pc screen, and auxiliary is used
Family observation right hand index finger tip position, there are purple, yellow, red three kinds of color states, correspond to right button release, right button respectively
It hovers, click three kinds of events by right key.When right hand index finger tip Z coordinate is more than 10, in right button releasing orientation, navigate auxiliary magnet
Become purple, finger can move freely at the arbitrary button for wishing to click at this time.When right hand index finger tip Z coordinate between -1 to
When between 1, in right button floating state, navigation auxiliary magnet becomes yellow, and the auxiliary magnet that navigates at this time can not move, and represents that inquiry is used
Whether family confirms the button clicked and navigated to.When right hand index finger tip Z coordinate is less than 10, in state is clicked by right key, lead
Boat auxiliary magnet becomes red, and system, which performs, at this time clicks button event, is equal to and is clicked on by mouse.The function can mould
Intend mouse click button event, click the button in online 3 d modeling software using right hand forefinger, software is operated.
For the advantageous effect (1) of Summary, as Fig. 7 (a) show using traditional " mouse+keyboard " mode into
The schematic diagram of row three-dimensional modeling man-machine interactive operation carries out body-sensing three-dimensional modeling as Fig. 7 (b) is shown using Leap Motion
The schematic diagram of man-machine interactive operation.Body-sensing three-dimensional modeling is carried out using Leap Motion, only needs to create three-dimensional mould by gesture
Type, without operating mouse, keyboard so that human-computer interaction is more natural.
For the advantageous effect (2) of Summary, be illustrated in figure 8 be respectively adopted traditional AJAX polling techniques with
Leap Motion are acquired gesture data and are uploaded to Cloud Server progress gesture three-dimensional modeling in real time by WebSocket agreements,
Line 3 d modeling software real-time rendering frame per second (FPS) variation tendency, testing time are 2 minutes, and a FPS value is acquired every 1s.
As shown in Figure 8, it is communicated using AJAX polling techniques, FPS values about fluctuate between 45-60, and WebSocket is used to assist
View communicate, FPS values about fluctuate between 65-85, it was demonstrated that using WebSocket agreements carry out gesture data it is real-time on
It passes, efficiency of transmission higher leads to gesture identification process faster, and three-dimensional modeling process is more smooth.
For the advantageous effect (3) of Summary, gesture path Similarity Match Method and HMM is respectively adopted
Method of model identification, to 11 kinds, test is identified in predefined three-dimensional modeling gesture.11 kinds of gestures of 20 users are acquired in total
Sample, wherein boy student 10, schoolgirl 10, boy student's height is differed from 165mm to 180mm, and schoolgirl's height is from 155cm to 170cm
It differs, to test influence of the different size hand-types to gesture recognition effect.As Fig. 9 (a) show it is similar based on gesture path
The gesture identification confusion matrix of matching process is spent, the gesture identification confusion matrix based on HMM model is shown such as Fig. 9 (b).By scheming
It is found that the gesture identification Average Accuracy based on gesture path Similarity Match Method is only 79.1%, and based on HMM model
Gesture identification accuracy rate is then up to 90%, and it is at most 1 time that arbitrary gesture, which is misidentified as other gesture numbers, represents HMM moulds
Type has certain self-learning capability, and gesture identification effect is relatively preferable.
The present invention the online body-sensing three-dimensional modeling method and system based on Leap Motion, be based primarily upon virtual reality,
The technical principles such as human-computer interaction and computer graphics assist the user to be carried out by body feeling interaction mode based on browser end
Online three-dimensional modeling operation.It should be noted that this system is suitable for online 3 d modeling software being deployed in cloud platform, it is many
Multi-user can access the software simultaneously by respective PC and Leap Motion sensors and carry out three-dimensional modeling.
It should be understood that the part that this specification does not elaborate belongs to the prior art.
It should be understood that the above-mentioned description for preferred embodiment is more detailed, can not therefore be considered to this
The limitation of invention patent protection range, those of ordinary skill in the art are not departing from power of the present invention under the enlightenment of the present invention
Profit is required under protected ambit, can also be made replacement or deformation, be each fallen within protection scope of the present invention, this hair
It is bright range is claimed to be determined by the appended claims.
Claims (8)
1. a kind of online body-sensing three-dimensional modeling method based on Leap Motion, which is characterized in that include the following steps：
Step 1：Gesture information predefines；
Step 1.1：Define three-dimensional modeling gesture；
Definition of gesture is carried out to creating basic body model and model manipulation mode；
Step 1.2：Gesture data pre-processes；
Leap Motion sensors are captured gesture space coordinate data, vector data and speed data to pre-process,
Obtaining as hidden Markov model (HMM) can input and complete effective gesture data of gesture identification function；
Step 1.3：Build gesture data model；
The gesture data model trained to effective gesture data structure of acquisition for HMM；
Step 2：Gesture training and identification；
Step 2.1：Build gesture training sample set；
The gesture data model trained to effective gesture data structure of acquisition for HMM；
Step 2.2：Build HMM gesture training patterns；
Sample set is built using effective gesture data of all captures, each gesture is trained for HMM；
Step 2.3：Gesture identification threedimensional model for identifying certain given gesture sequence, and performs corresponding three-dimensional modelling operability.
2. the online body-sensing three-dimensional modeling method according to claim 1 based on Leap Motion, it is characterised in that：Step
In rapid 1.1, the basic body includes square, cuboid, cylinder, cone, sphere, torus；The model behaviour
Make mode to include translation, rotate around X-axis, rotate around Y-axis, rotate about the z axis, scale.
3. the online body-sensing three-dimensional modeling method according to claim 1 based on Leap Motion, it is characterised in that：Step
Rapid 1.2, gesture data pretreatment is standardized including the valid frame screening technique based on centre of the palm rate, valid frame three-dimensional coordinate data
Processing method and available frame count are according to method for resampling；It is described based on the valid frame screening technique of centre of the palm rate by setting the centre of the palm
Rate-valve value judges whether captured frame data are that slight movement of the palm rate travel less than 5mm/s causes, and then judge hand
The beginning of gesture is with terminating state；The valid frame three-dimensional coordinate data normalization processing method passes through three to acquired valid frame
Dimension coordinate data make unified specificationization processing, effectively meet the requirement that HMM model observation is integer；The available frame count is according to weight
The method of sampling is by set point away from threshold value, it is ensured that available frame count strong point spacing approximately equal in each gesture.
4. the online body-sensing three-dimensional modeling method according to claim 1 based on Leap Motion, it is characterised in that：Step
Gesture data model is built described in rapid 1.3, is that three-dimensional coordinate data and correspondent method in effective frame data are included by structure
The Formal Representation model of vector or direction vector, the training input data as HMM gesture training patterns.
5. the online body-sensing three-dimensional modeling method according to claim 1 based on Leap Motion, it is characterised in that：Step
Gesture training sample set is built described in rapid 2.1, is to be directed to 6 kinds of basic bodies and 3 kinds of model manipulation modes, 11 kinds altogether
Operation, respectively to each operating gesture data collecting sample several, build gesture training sample set.
6. the online body-sensing three-dimensional modeling method according to claim 1 based on Leap Motion, it is characterised in that：Step
HMM gesture training patterns are built described in rapid 2.2, are using preceding backward algorithm in HMM theories, according to given gesture training sample
Hidden state corresponding to this sequence and each gesture training sample builds HMM gesture training patterns.
7. the online body-sensing three-dimensional modeling method according to claim 1 based on Leap Motion, it is characterised in that：Step
Gesture identification threedimensional model is built described in rapid 2.3, is whether HMM gestures training pattern constructed by judgement is certain given gesture sequence
The best match model of row, and then realize the identification of certain gestures and corresponding three-dimensional modelling operability.
8. a kind of online body-sensing 3 d modeling system based on Leap Motion, it is characterised in that：Including software interactive unit,
Collecting unit, Real Time Communication Unit, data processing unit, storage unit, computing unit and execution unit；
The software interactive unit is used to simulate the online 3 d modeling software of mouse operation and control by right hand forefinger；
The collecting unit is for collecting sample gesture data and needs the gesture data judged；
The Real Time Communication Unit is used to acquired gesture data being encapsulated as JSON files, by real-time Communication for Power Network agreement
Reach Cloud Server；
The data processing unit is used to carry out acquired gesture data the pretreatment for meeting HMM model training；
The storage unit builds gesture training sample set for storing pretreated gesture data；
The computing unit is used to gesture training sample set inputting HMM gesture training patterns, and gesture training is carried out to the model,
It can effectively identify 11 kinds of corresponding gesture sequences of three-dimensional modeling operation；
The execution unit performs corresponding three-dimensional modeling operation in a browser for identifying the gesture sequence that user gives.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810054234.2A CN108182728A (en) | 2018-01-19 | 2018-01-19 | A kind of online body-sensing three-dimensional modeling method and system based on Leap Motion |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810054234.2A CN108182728A (en) | 2018-01-19 | 2018-01-19 | A kind of online body-sensing three-dimensional modeling method and system based on Leap Motion |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN108182728A true CN108182728A (en) | 2018-06-19 |
Family
ID=62550886
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810054234.2A Pending CN108182728A (en) | 2018-01-19 | 2018-01-19 | A kind of online body-sensing three-dimensional modeling method and system based on Leap Motion |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108182728A (en) |
Cited By (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109116984A (en) * | 2018-07-27 | 2019-01-01 | 冯仕昌 | A kind of tool box for three-dimension interaction scene |
| CN109410319A (en) * | 2018-09-30 | 2019-03-01 | Oppo广东移动通信有限公司 | A kind of data processing method, server and computer storage medium |
| CN109492558A (en) * | 2018-10-29 | 2019-03-19 | 华南理工大学 | A kind of dynamic gesture identification method based on Leap Motion |
| CN109753876A (en) * | 2018-12-03 | 2019-05-14 | 西北工业大学 | A kind of construction method of the extraction identification and three-dimensional gesture interaction system of three-dimension gesture |
| CN109948592A (en) * | 2019-04-04 | 2019-06-28 | 北京理工大学 | A kind of design idea method of discrimination and system based on hand signal identification |
| CN110060350A (en) * | 2019-02-28 | 2019-07-26 | 浙江华越设计股份有限公司 | A kind of architectural design method based on VR technology |
| CN110070777A (en) * | 2019-06-13 | 2019-07-30 | 大连民族大学 | A kind of Hezhe's fish-skin draws simulation training system and implementation method |
| CN110837792A (en) * | 2019-11-04 | 2020-02-25 | 东南大学 | Three-dimensional gesture recognition method and device |
| CN111290574A (en) * | 2020-01-19 | 2020-06-16 | 山东超越数控电子股份有限公司 | Method and device for controlling unmanned aerial vehicle by using gestures and readable storage medium |
| CN112069873A (en) * | 2020-07-16 | 2020-12-11 | 上海大学 | LeapMotion gesture recognition-based screen control system and method |
| CN113223344A (en) * | 2021-05-25 | 2021-08-06 | 湖南汽车工程职业学院 | Big data-based professional teaching display system for art design |
| CN113296604A (en) * | 2021-05-24 | 2021-08-24 | 北京航空航天大学 | True 3D gesture interaction method based on convolutional neural network |
| CN113391754A (en) * | 2021-06-28 | 2021-09-14 | 昆明理工大学 | Wrist rotation menu interaction technology based on LeapMotion sensor |
| CN113496168A (en) * | 2020-04-02 | 2021-10-12 | 百度在线网络技术（北京）有限公司 | Sign language data acquisition method, sign language data acquisition equipment and storage medium |
| CN113961080A (en) * | 2021-11-09 | 2022-01-21 | 南京邮电大学 | Three-dimensional modeling software framework based on gesture interaction and design method |
| CN114625255A (en) * | 2022-03-29 | 2022-06-14 | 北京邮电大学 | Free-hand interaction method for visual view construction, visual view construction device and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102142055A (en) * | 2011-04-07 | 2011-08-03 | 上海大学 | True three-dimensional design method based on augmented reality interactive technology |
| CN104007819A (en) * | 2014-05-06 | 2014-08-27 | 清华大学 | Gesture recognition method and device and Leap Motion system |
| CN104915009A (en) * | 2015-06-25 | 2015-09-16 | 深圳先进技术研究院 | Gesture prediction method and system |
| CN107024989A (en) * | 2017-03-24 | 2017-08-08 | 中北大学 | A kind of husky method for making picture based on Leap Motion gesture identifications |
-
2018
- 2018-01-19 CN CN201810054234.2A patent/CN108182728A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102142055A (en) * | 2011-04-07 | 2011-08-03 | 上海大学 | True three-dimensional design method based on augmented reality interactive technology |
| CN104007819A (en) * | 2014-05-06 | 2014-08-27 | 清华大学 | Gesture recognition method and device and Leap Motion system |
| CN104915009A (en) * | 2015-06-25 | 2015-09-16 | 深圳先进技术研究院 | Gesture prediction method and system |
| CN107024989A (en) * | 2017-03-24 | 2017-08-08 | 中北大学 | A kind of husky method for making picture based on Leap Motion gesture identifications |
Non-Patent Citations (3)
| Title |
|---|
| 任磊: "基于Leap Motion的动态手势识别研究", 《中国优秀硕士学位论文全文数据库 信息科技辑》 * |
| 吴福理 等: "基于Leap Motion的虚拟农作物人机交互系统设计", 《农业工程学报》 * |
| 黄俊: "基于Leap Motion的手势交互在虚拟场景中的研究", 《中国优秀硕士学位论文全文数据库 信息科技辑》 * |
Cited By (26)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109116984A (en) * | 2018-07-27 | 2019-01-01 | 冯仕昌 | A kind of tool box for three-dimension interaction scene |
| CN109410319B (en) * | 2018-09-30 | 2021-02-02 | Oppo广东移动通信有限公司 | Data processing method, server and computer storage medium |
| CN109410319A (en) * | 2018-09-30 | 2019-03-01 | Oppo广东移动通信有限公司 | A kind of data processing method, server and computer storage medium |
| US11044457B2 (en) | 2018-09-30 | 2021-06-22 | Guangdong Oppo Mobile Telecommunications Corp., Ltd. | Method for processing data, server and computer storage medium |
| CN109492558A (en) * | 2018-10-29 | 2019-03-19 | 华南理工大学 | A kind of dynamic gesture identification method based on Leap Motion |
| CN109753876A (en) * | 2018-12-03 | 2019-05-14 | 西北工业大学 | A kind of construction method of the extraction identification and three-dimensional gesture interaction system of three-dimension gesture |
| CN110060350A (en) * | 2019-02-28 | 2019-07-26 | 浙江华越设计股份有限公司 | A kind of architectural design method based on VR technology |
| CN109948592A (en) * | 2019-04-04 | 2019-06-28 | 北京理工大学 | A kind of design idea method of discrimination and system based on hand signal identification |
| CN110070777B (en) * | 2019-06-13 | 2021-07-09 | 大连民族大学 | Huchizhui fish skin painting simulation training system and implementation method |
| CN110070777A (en) * | 2019-06-13 | 2019-07-30 | 大连民族大学 | A kind of Hezhe's fish-skin draws simulation training system and implementation method |
| CN110837792A (en) * | 2019-11-04 | 2020-02-25 | 东南大学 | Three-dimensional gesture recognition method and device |
| CN110837792B (en) * | 2019-11-04 | 2023-07-25 | 东南大学 | Three-dimensional gesture recognition method and device |
| CN111290574A (en) * | 2020-01-19 | 2020-06-16 | 山东超越数控电子股份有限公司 | Method and device for controlling unmanned aerial vehicle by using gestures and readable storage medium |
| CN111290574B (en) * | 2020-01-19 | 2022-09-09 | 超越科技股份有限公司 | Method and device for controlling unmanned aerial vehicle by using gestures and readable storage medium |
| CN113496168A (en) * | 2020-04-02 | 2021-10-12 | 百度在线网络技术（北京）有限公司 | Sign language data acquisition method, sign language data acquisition equipment and storage medium |
| CN112069873A (en) * | 2020-07-16 | 2020-12-11 | 上海大学 | LeapMotion gesture recognition-based screen control system and method |
| CN113296604B (en) * | 2021-05-24 | 2022-07-08 | 北京航空航天大学 | True 3D gesture interaction method based on convolutional neural network |
| CN113296604A (en) * | 2021-05-24 | 2021-08-24 | 北京航空航天大学 | True 3D gesture interaction method based on convolutional neural network |
| CN113223344A (en) * | 2021-05-25 | 2021-08-06 | 湖南汽车工程职业学院 | Big data-based professional teaching display system for art design |
| CN113223344B (en) * | 2021-05-25 | 2022-08-23 | 湖南汽车工程职业学院 | Big data-based professional teaching display system for art design |
| CN113391754A (en) * | 2021-06-28 | 2021-09-14 | 昆明理工大学 | Wrist rotation menu interaction technology based on LeapMotion sensor |
| CN113391754B (en) * | 2021-06-28 | 2024-03-12 | 昆明理工大学 | Wrist rotation menu interaction technology based on leap motion sensor |
| CN113961080A (en) * | 2021-11-09 | 2022-01-21 | 南京邮电大学 | Three-dimensional modeling software framework based on gesture interaction and design method |
| CN113961080B (en) * | 2021-11-09 | 2023-08-18 | 南京邮电大学 | Three-dimensional modeling software framework based on gesture interaction and design method |
| CN114625255A (en) * | 2022-03-29 | 2022-06-14 | 北京邮电大学 | Free-hand interaction method for visual view construction, visual view construction device and storage medium |
| CN114625255B (en) * | 2022-03-29 | 2023-08-01 | 北京邮电大学 | Freehand interaction method oriented to visual view construction, visual view construction device and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108182728A (en) | A kind of online body-sensing three-dimensional modeling method and system based on Leap Motion | |
| CN108776773B (en) | Three-dimensional gesture recognition method and interaction system based on depth image | |
| Zhou et al. | Virtual reality: A state-of-the-art survey | |
| CN100407798C (en) | Three-dimensional geometric mode building system and method | |
| US11410464B2 (en) | Detection of hand gestures using gesture language discrete values | |
| CN107423398A (en) | Exchange method, device, storage medium and computer equipment | |
| CN111488824A (en) | Motion prompting method and device, electronic equipment and storage medium | |
| CN105051755A (en) | Part and state detection for gesture recognition | |
| CN103823554A (en) | Digital virtual-real interaction system and digital virtual-real interaction method | |
| CN112241715A (en) | Model training method, expression recognition method, device, equipment and storage medium | |
| CN116092120B (en) | Image-based action determining method and device, electronic equipment and storage medium | |
| WO2020195017A1 (en) | Path recognition method, path recognition device, path recognition program, and path recognition program recording medium | |
| Shi et al. | Grasping 3d objects with virtual hand in vr environment | |
| CN112596659B (en) | Drawing method and device based on intelligent voice and image processing | |
| CN112435316B (en) | Method and device for preventing mold penetration in game, electronic equipment and storage medium | |
| CN107122042A (en) | The Chinese-character writing method and system that a kind of quiet dynamic gesture is combined | |
| CN110991237A (en) | Grasping taxonomy-based virtual hand natural grasping action generation method | |
| Yan et al. | AGRMTS: A virtual aircraft maintenance training system using gesture recognition based on PSO‐BPNN model | |
| Taylor et al. | Interacting with real objects in virtual worlds | |
| CN117932713A (en) | Cloud native CAD software gesture interaction geometric modeling method, system, device and equipment | |
| Tang | Application of Virtual 3D Reproduction Technology in Interactive Intelligence of Game Design | |
| Yohannes et al. | Virtual reality in puppet game using depth sensor of gesture recognition and tracking | |
| KR102568699B1 (en) | Floor-aware Post-processing method for Point Cloud Generated from 360-degree Panoramic Indoor Images | |
| Albakri et al. | 3D Keyframe Motion Extraction from Zapin Traditional Dance Videos | |
| Xia | An interactive algorithm for augmented reality experiment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20180619 |