CN117932713A

CN117932713A - Cloud native CAD software gesture interaction geometric modeling method, system, device and equipment

Info

Publication number: CN117932713A
Application number: CN202410307605.9A
Authority: CN
Inventors: 赵飞宇; 万弘辉; 刘猛; 李广能
Original assignee: South Central University for Nationalities
Current assignee: South Central Minzu University
Filing date: 2024-03-18
Publication date: 2024-04-26

Abstract

The invention discloses a method, a system, a device and equipment for modeling gesture interaction geometry of cloud native CAD software, which are characterized in that firstly gestures for simulating a mouse to control the cloud native CAD software are collected, and corresponding geometry model data are identified; preprocessing gesture data; inputting the preprocessed gesture data into a Hidden Markov Model (HMM) to identify a gesture sequence corresponding to the three-dimensional modeling operation; and finally, based on the recognized gesture sequence, executing corresponding geometric modeling operation in the cloud-native CAD software. According to the invention, the gesture recognition technology is combined with the CAD technology, so that the learning threshold of geometric modeling and product design is effectively reduced, and the man-machine interaction of three-dimensional modeling is improved by utilizing the vertical virtual light curtain of the Leap Motion sensor; the method and the device not only effectively reduce the gesture data volume by means of the preprocessing step and meet the requirements of the network environment on the high efficiency and the real-time performance of data transmission, but also effectively improve the speed and the accuracy of gesture recognition and realize good human-computer interaction experience.

Description

Cloud native CAD software gesture interaction geometric modeling method, system, device and equipment

Technical Field

The invention belongs to the technical field of virtual reality and man-machine interaction, relates to a gesture interaction geometric modeling method, a gesture interaction geometric modeling system, a gesture interaction geometric modeling device and gesture interaction geometric modeling equipment, and particularly relates to a cloud native CAD software gesture interaction geometric modeling method, a cloud native CAD software gesture interaction geometric modeling system, a cloud native CAD software gesture interaction geometric modeling device and gesture interaction geometric modeling equipment.

Background

With the continuous development of information technology, computer aided design (Computer AIDED DESIGN) software has gradually developed from a traditional C/S (Client/Server) architecture to a B/S (Browser/Server) architecture, and gradually developed into cloud-based CAD software which directly runs at a Browser end, supports cloud online three-dimensional modeling, and can provide services of distributed collaborative design, information communication and knowledge sharing for enterprises. At present, the interaction mode of designers and cloud native CAD software still depends on a mode of a mouse and a keyboard, and professional CAD software has higher requirements on professional knowledge of the designers, is not beneficial to reducing the geometric modeling design threshold of products, and attracts non-professional persons to participate. For example, cloud native CAD software Onshape, crownCAD facing the manufacturing field, and cloud native CAD software TINKERCAD facing the 3D printing field developed by Autodesk corporation, although the geometric modeling function is very powerful, but virtual reality device access is not supported, so that the geometric modeling interactivity is not strong enough, and the three-dimensional modeling and product design learning thresholds are still high.

In recent years, with the advent of virtual reality technology, the limitation of mouse and keyboard on man-machine interaction is effectively relieved, the three-dimensional modeling thought of a designer can be reflected more intuitively, and a natural and immersed man-machine interaction three-dimensional modeling mode is realized. Hu Hong et al, in the text of Leap Motion key point model hand gesture estimation method, propose a key point model gesture estimation method based on Leap Motion, establish a key point hand skeleton model based on a hand structure, wherein part of key point position information can be obtained by a Leap Motion sensor, and the space coordinates of other key points can be calculated according to the model, so that the gesture is estimated. However, the key point hand skeleton model constructed by the method is simpler, and the model only uses the space position information of the hand joint points captured by the Leap Motion, and does not use normal vector and speed information, so that the adaptability to different hands is poor. In addition, the method realizes that the platform is C/S architecture client software and cannot be suitable for online three-dimensional modeling software. Dragon and the like in a text of 'CAD model gesture control technique based on scenes', propose a technical method for describing gesture control of a human-computer interaction scene model in a CAD modeling process, according to gesture preference of a user, a mapping relation between gestures and CAD instructions under a specific situation is established by using a decision tree model, one-to-many flexible mapping of the gestures and the CAD instructions is realized, the number of the mappable CAD instructions of the gestures is increased by 100%, and the design intent of the user can be expressed rapidly by using a gesture interaction mode. However, according to the method, aiming at specific scenes and personal preferences of users, hidden Markov Models (HMM) are adopted for gesture recognition, a decision tree algorithm is utilized for realizing gesture and CAD instruction mapping, gesture self-learning is not realized, and universality is not strong. And the accuracy of gesture control in the fine control process is still lower than that of keyboard and mouse control. Liu Quan et al in the text of adaptive dynamic gesture recognition based on a Leap Motion sensor, propose a gesture recognition method for performing dynamic gesture track self-learning by using an HMM, define 26 English letter writing tracks as 26 predefined gesture categories, extract track data drawn by gesture nodes by using the Leap Motion sensor, establish a sample set, train the sample set by using the HMM, and dynamically recognize the predefined gestures, wherein the average recognition accuracy can reach 92.95%. However, the data collected and input by the method is only track point space position information, normal vector and speed information are not utilized, and the self-learning algorithm of the HMM is not optimized, so that the recognition accuracy still has a large improvement space, and the method is still realized based on a local terminal and does not realize online gesture recognition. Wu Fuli et al, in the text of the figure of man-machine interaction system of virtual crops based on Leap Motion, propose an online virtual farming object sense interaction method based on Leap Motion, the gesture posture data collected by Leap Motion is packaged into a JSON file, and the JSON file is uploaded to a server in real time by means of WebSocket protocol and is used for calling the man-machine interaction system of virtual crops developed based on HTML5 at any time. The system is developed based on WebGL, and can perform three-dimensional visual interactive operation on various crops based on a browser completely by means of gesture data. However, the method only uses the collected gesture data to operate the three-dimensional model of the virtual crop, and gesture self-learning is not realized, so that the operation accuracy is low, and the accuracy cannot be improved along with the increase of the use frequency of the system. Gu Jing et al, real-time hand gestures system based on Leap Motion, disclose a Real-time gesture recognition system based on Leap Motion. Static gestures and dynamic gestures may be recognized and real-time may be achieved. For the identification of static gestures, a Support Vector Machine (SVM) classifier is used for extracting finger tip characteristic points for identification; for the identification of dynamic gestures, an algorithm based on a time sequence analysis technology is used for identifying information such as movement tracks and speeds of the gestures. Meanwhile, in order to improve the robustness of the system, the length of the finger is also introduced as a factor. However, this approach focuses only on the recognition of gestures, and does not relate to the meaning and application of gestures. Inam Ur Rehman et al in "Fingertip Gestures Recognition Using Leap Motionand Camera for Interaction with Virtual Environment", propose a simple gesture recognition method based on color fingertips, based on a Leap Motion and camera using single fingertip gestures to navigate, select, translate and release interactions of multiple interaction modes such as objects with a virtual environment. However, this method requires placing a color cap on the fingertip, which places an additional burden on the user, while requiring mapping of fingertip colors during the recognition process, increasing the complexity of the work, and the recognition process is affected by light and background colors. Li Shusen et al in patent, "a somatosensory interaction rapid three-dimensional modeling auxiliary system and method thereof", disclose a somatosensory three-dimensional interactive modeling method and system, which captures user modeling intention including model type, model size by using a Kinect sensor to receive depth image data, and invokes corresponding API function of SolidWorks to realize three-dimensional modeling of basic geometry in the software. However, the method can only realize the interactive three-dimensional modeling based on SolidWorks, but cannot realize the online three-dimensional modeling, and on the other hand, a user cannot interactively operate the created three-dimensional model, including scaling, translation and rotation, so that the man-machine interaction is not strong enough.

Disclosure of Invention

Aiming at the problem that the human-computer interaction in the three-dimensional geometric modeling process in a Web environment is insufficient due to the fact that the existing cloud-based CAD software has poor access support to virtual reality equipment, the invention provides a gesture interaction geometric modeling method, a gesture interaction geometric modeling system, gesture interaction geometric modeling device and gesture interaction geometric modeling equipment for the cloud-based CAD software.

The technical scheme adopted by the method is as follows: a cloud native CAD software gesture interaction geometric modeling method comprises the following steps:

step 1: acquiring gestures of a simulated mouse for controlling cloud native CAD software, and identifying corresponding geometric modeling operation;

Step 2: preprocessing gesture data;

Step 3: inputting the preprocessed gesture data into a Hidden Markov Model (HMM) to identify a gesture sequence corresponding to the three-dimensional modeling operation;

Step 4: based on the recognized gesture sequence, a corresponding geometric modeling operation is performed in cloud-native CAD software.

Preferably, in step 1, the geometric modeling operation includes creating a cube model, creating a cuboid model, creating a cylinder model, creating a cone model, creating a sphere model, creating a torus model, translating a model, rotating a model around an X-axis, rotating a model around a Y-axis, rotating a model around a Z-axis, and scaling a model.

Preferably, in step 2, the start and end states of the gesture are recognized, and the rate of the palm P _Ri of the right hand H _Ri in the i-th frame data is recorded asWhen the rate is smaller than a set threshold epsilon, the right hand is considered to be in a static state, the frame data is invalid, and when the rate is larger than the threshold epsilon, the right hand is considered to be in a motion state, and the frame data is valid;

defining judgment rules of gesture starting and ending states, and setting a rate difference threshold value Judging whether the gesture is in a starting or ending state according to the difference of the speeds of the i-1 th frame and the i-th frame, and when/>When it is in a starting state whenThe time is the end state;

Normalizing three-dimensional coordinate values of tracks in all effective frame data, and recording the space coordinate of palm P _Ri of right hand H _Ri as For each gesture track, find out the maximum of the gesture tracks on the X axis, the Y axis and the Z axisAnd minimum/>And uses the value to calculate the compression ratio/>, of the track on three coordinate axesTransforming the trajectory into a space of 10 ³;

For the obtained Performing rounding operation to ensure that coordinate data are distributed on integer crossing points of three-dimensional coordinate axes, and reassigning the rounded values to/>；

And resampling the normalized gesture samples by adopting an equidistant resampling method, so as to ensure that the effective frame data points in each gesture are distributed approximately uniformly.

Preferably, in step2, the i-th frame gesture H _i after preprocessing is recorded as:

；

wherein, Representing left hand gesture data,/>Representing right hand gesture data; /(I)Data representing left and right palmar heart respectively,/>Finger tip data of the thumb, the index finger, the middle finger, the ring finger and the little finger of the left hand are sequentially represented; /(I)Sequentially representing fingertip data of a thumb, an index finger, a middle finger, a ring finger and a little finger;

；

wherein, Representing left palm heart coordinates,/>Representing the left palm center normal vector,/>Representing left palm heart rate; /(I)And obtaining the product by the same way;

；

wherein, Representing the finger tip coordinates of the left hand thumb,/>Representing a left thumb fingertip direction vector; /(I)And obtaining the product in the same way.

Preferably, in step 4, based on the recognized gesture sequence, selecting a gesture category corresponding to the maximum probability as the recognized gesture; after the three-dimensional modeling intention of the user expressed by the gesture is obtained, executing corresponding three-dimensional modeling operation to realize three-dimensional modeling through the gesture.

Preferably, the HMM model in step 3 is a trained model;

the specific training comprises the following substeps:

Step 3.1: constructing a gesture training sample set;

a single gesture sample recorded as completing a certain gesture is T represents the total number of valid frames, and a sample set/>, of the gesture is establishedN represents the number of samples; altogether build L sample sets/>As input data for performing gesture recognition training by the HMM model; wherein L represents the total number of gestures;

Step 3.2: training a model;

for each HMM model corresponding to the gesture, training an initial vector corresponding to each HMM model through a known gesture sample sequence and a gesture category corresponding to the known gesture sample sequence State transition matrix/>Confusion matrix；

Sequentially inputting all gesture sequence data corresponding to each type of gesture in the gesture training sample set into the HMM model, and re-estimating the initial vector every time new gesture sequence data is inputThe state transition matrix A and the confusion matrix B have the following calculation formulas:

；

wherein, The posterior probability is the probability that the state is i at the moment t when a gesture sample sequence and an HMM model are given; /(I)As a forward variable, the probability that the state at time t is q _i and the state at time t+1 is q _j is expressed; n represents the number of possible states and M represents the number of possible gesture observations.

Preferably, each iteration is calculatedThen, adopting a simulated annealing algorithm to mix matrix/>The correction is carried out, specifically comprising the following substeps:

(1) Initializing parameters including An annealing initial temperature T ₀, a cooling coefficient k, a termination temperature T _e, and a convergence condition ρ;

(2) Setting a cooling function ；

(3) Generating N×M mutually independent random variables X satisfying normal distribution, which expects E (X) =0, varianceLet/>If/>Let/>；

(4) For a pair ofNormalization processing is carried out,/>；

(5) JudgingWhether the convergence condition ρ is satisfied or whether the termination temperature T _e has been reached: if yes, the algorithm is ended, and the current/> istakenAn optimal solution; if not, turning to the step (1) and continuing iteration; wherein O represents the observed sequence,/>Representing the current hidden markov model.

The system of the invention adopts the technical proposal that: a cloud-native CAD software gesture interaction geometric modeling system comprises the following modules:

The data acquisition and identification module is used for acquiring gestures for simulating a mouse to control the cloud native CAD software and identifying corresponding geometric modeling operation;

the data preprocessing module is used for preprocessing gesture data;

the gesture sequence recognition module is used for inputting the preprocessed gesture data into the HMM model and recognizing a gesture sequence corresponding to the three-dimensional modeling operation;

and the geometric modeling module is used for executing corresponding geometric modeling operation in the cloud native CAD software based on the recognized gesture sequence.

The technical scheme adopted by the device is as follows: a cloud native CAD software gesture interaction geometry modeling apparatus, comprising: the system comprises a software interaction unit, an acquisition unit, a real-time communication unit, a data processing unit, a storage unit, a calculation unit and an execution unit;

The acquisition unit is used for acquiring gesture information of a user and the gesture information in real time and sending the gesture information to the real-time communication unit; the real-time communication unit is used for packaging the acquired information into a lightweight data storage file-JSON file so as to adapt to the requirement of network environment transmission on data weight reduction, and carrying out data real-time interaction with the cloud server through a 6437 port of the client by means of a WebSocket protocol, namely uploading the data captured by the Leap Motion sensor to the cloud server in real time; the software interaction unit is used for deploying cloud native CAD software, configuring a Leap. Js library file, simulating clicking operation of a right button of a mouse by means of the library file and a vertical virtual light curtain of a Leap Motion sensor, and facilitating a user to directly control the cloud native CAD software through gestures; the data processing unit is used for analyzing the JSON file, and rectifying all data into a gesture training sample set, so that the requirement of the HMM model on training data standardization is met; the storage unit is used for storing the processed gesture training sample set by adopting a relational database; the computing unit is used for performing scientific computation based on the HMM gesture training model, and through receiving gesture data in the gesture training sample set and repeatedly training the gesture data, the cloud native CAD software can effectively recognize 11 geometric modeling gestures; the execution unit is used for responding to the specific geometric modeling gesture and executing corresponding geometric modeling operation.

The technical scheme adopted by the equipment is as follows: a cloud native CAD software gesture interaction geometry modeling apparatus, comprising:

One or more processors;

And the storage device is used for storing one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors realize the cloud native CAD software gesture interaction geometric modeling method.

The beneficial effects of the invention include:

(1) The gesture recognition technology is combined with the CAD technology, three-dimensional modeling operation is carried out through predefining 11 gestures, so that the learning threshold of geometric modeling and product design is effectively reduced, the vertical virtual light curtain of the Leap Motion sensor is utilized, the cloud primary CAD software is controlled by a simulated mouse, and the man-machine interaction of the three-dimensional modeling is improved;

(2) Aiming at the problems that the access support of a Web application program to a virtual reality device is poor and data communication is difficult, a gesture data real-time transmission frame based on a WebSocket protocol is constructed, gesture data captured by a Leap Motion is packaged into a JSON file, and the JSON file is uploaded to a cloud server for deploying cloud native CAD software in real time by means of the WebSocket protocol, so that the real-time communication of the gesture data and the cloud server can be realized, and further the somatosensory interaction of a user and online three-dimensional modeling software is realized;

(3) Preprocessing the collected gesture data, constructing a gesture data model and a gesture training sample set to store the preprocessed gesture data, and training the gesture by means of the HMM model, so that the problems of misjudgment of the starting and ending states of the gesture caused by the fine movement of the hand, low gesture recognition accuracy caused by different hand types and operation habits of different users and the like are effectively avoided. The gesture data volume is effectively reduced by means of the preprocessing step, the requirements of the network environment on the high efficiency and the real-time performance of data transmission are met, the speed and the accuracy of gesture recognition are effectively improved, and good human-computer interaction experience is realized.

Drawings

The following examples, as well as specific embodiments, are used to further illustrate the technical aspects of the present invention. In addition, in the course of describing the technical solutions, some drawings are also used. Other figures and the intent of the present invention can be derived from these figures without inventive effort for a person skilled in the art.

FIG. 1 is a flow chart of a method according to an embodiment of the present invention;

FIG. 2 is a diagram of geometric model data corresponding to gesture data in embodiment 11;

FIG. 3 (a) is a schematic diagram of a virtual three-dimensional coordinate system constructed by a Leap Motion sensor according to an embodiment of the present invention;

FIG. 3 (b) is a schematic diagram of a diagram of capturing a frame of data by a Leap Motion sensor according to an embodiment of the present invention;

FIG. 4 is a schematic diagram of an HMM gesture training model according to an embodiment of the present invention;

FIG. 5 is a diagram of a real-time gesture data transmission frame based on the WebSocket protocol according to an embodiment of the present invention;

FIG. 6 is a schematic diagram of a simulated mouse right click operation according to an embodiment of the present invention;

FIG. 7 is a diagram illustrating a real-time upload frame rate (FPS) change of gesture data according to an embodiment of the present invention;

FIG. 8 (a) is a gesture recognition confusion matrix based on a gesture trajectory similarity matching method according to an embodiment of the present invention;

FIG. 8 (b) is a schematic diagram of a HMM model-based gesture recognition confusion matrix according to an embodiment of the invention.

Detailed Description

In order to facilitate the understanding and practice of the invention, those of ordinary skill in the art will now make further details with reference to the drawings and examples, it being understood that the examples described herein are for the purpose of illustration and explanation only and are not intended to limit the invention thereto.

Aiming at the problem that the existing cloud-based CAD software has poor access support to virtual reality equipment and causes insufficient man-machine interactivity in the three-dimensional geometric modeling process in a Web environment, the embodiment provides a gesture interaction geometric modeling method for the cloud-based CAD software, 6 basic geometric bodies are created by defining gestures of different types, and translation, rotation and scaling operations are carried out on the created geometric bodies. A large number of samples are collected for different gestures, a gesture training sample set is constructed, and training is carried out by means of a Hidden Markov Model (HMM), so that gesture recognition three-dimensional modeling is realized. In addition, the embodiment also relates to a gesture interaction geometric modeling device of the cloud native CAD software, and by means of the vertical virtual light curtain of the Leap Motion sensor, the on-line three-dimensional modeling software for somatosensory manipulation is realized by using a mode that a right index finger simulates a mouse to control a computer.

Please refer to fig. 1, a method for modeling gesture interaction geometry of cloud native CAD software provided in this embodiment includes the following steps:

Step 1: acquiring gestures for simulating mouse to control on-line three-dimensional modeling software, and identifying corresponding geometric model data;

In one embodiment, the geometric modeling operation includes creating a cube model, creating a cuboid model, creating a cylinder model, creating a cone model, creating a sphere model, creating a torus model, translating a model, rotating about an X-axis, rotating about a Y-axis, rotating about a Z-axis, and scaling a model.

In one embodiment, gesture definition is performed on a total of 11 three-dimensional modeling operations for creating 6 basic geometric models, controlling the geometric models to translate, rotate (around 3 coordinate axes), and zoom; by defining 11 three-dimensional modeling gestures, a three-dimensional modeling process by mouse control is converted into three-dimensional modeling by gestures, gesture data captured by a Leap Motion sensor is preprocessed, the gesture data amount is effectively reduced, a network transmission scene is adapted, the validity of the data is ensured, the preprocessed gesture data is stored by constructing a gesture data model, and a formalized data storage model is constructed, so that the gesture data storage model is conveniently used as input of gesture recognition training of an HMM model.

In one embodiment, as shown in FIG. 2, a detailed description of the process of 11 gestures is given in Table 1.

Table 1 three-dimensional modeling gesture description

Step 2: preprocessing gesture data;

In one embodiment, the space coordinate data, vector data and speed data captured by the Leap Motion sensor are preprocessed to obtain effective gesture data required by gesture recognition.

In one embodiment, the gesture data acquired by the Leap Motion sensor is screened and normalized, so that the gesture data volume can be reduced to meet the requirements of high efficiency and instantaneity in the process of uploading the gesture data to the cloud server in real time, and the requirement of the HMM model on the input data as an integer can be met. As shown in fig. 3 (a), a virtual three-dimensional space coordinate system constructed by the Leap Motion sensor is shown, and the information of space coordinate values, vectors, velocity and the like in the current frame data captured by the sensor is calculated based on the coordinate system. Fig. 3 (b) shows a schematic diagram of a frame of data captured by the Leap Motion sensor.

The Leap Motion sensor data acquisition frame rate is 200 fps, and the device is sensitive, so that when the subjective consciousness of a user drives the hand to be in a static state, the fine Motion data of the hand can be captured by the device, and the beginning and ending states of the gesture can not be identified. Therefore, a rate threshold epsilon is set to determine whether the captured data of a certain frame is the data generated by the gesture motion, and the computer can recognize the starting and ending states of the gesture by setting the threshold, i.e. the frame data is determined to be invalid at the moment, and the gesture is in a static state. Empirically, ε is set to 5 mm/s. Let the palm P _Ri rate of the right hand H _Ri in the ith frame data beWhen the rate is smaller than a set threshold epsilon, the right hand is considered to be in a static state, the frame data is invalid, and when the rate is larger than the threshold epsilon, the right hand is considered to be in a motion state, and the frame data is valid;

。

further, decision rules for gesture start and end states are defined. Empirically, a rate difference threshold is set 9 Mm/s, and judging that the gesture is in a starting or ending state according to the difference of the speeds of the i-1 frame and the i frame;

。

in order to unify the track sizes in the gesture data captured by each effective frame and avoid the influence on the gesture recognition effect due to the fact that the track sizes of the gestures are different, the three-dimensional coordinate values of the tracks in all the effective frame data are required to be normalized. Taking the space coordinate of the palm center P _Ri of the right hand H _Ri as For example, the normalization process is described. And (3) for each gesture track, respectively searching the maximum and minimum values of the gesture track on the X axis, the Y axis and the Z axis, calculating the compression ratio of the track on three coordinate axes by using the values, and transforming the track into a space (unit: mm) of 10 ³.

；

In addition, the obtainedPerforming rounding operation to ensure that coordinate data are distributed on integer crossing points of three-dimensional coordinate axes, and reassigning the rounded values to/>；

。

In order to ensure that the distances between the effective frame data points contained in each gesture are approximately equal, an equidistant resampling method is adopted to resample the normalized gesture samples. Taking the right hand palm as an example, the threshold of the point distance is determined empiricallyCalculation/>And/>Distance between/>Judge/>Whether or not to be greater than/>: If/>Then reserveCalculation/>And/>Distance of/>And continue to determine if it is greater than/>; If/>Delete/>Calculation/>And/>Distance/>And determine if it is greater than/>. Iterative computations are performed until the last point, ensuring that the effective frame data points in each gesture are distributed approximately uniformly.

And (3) constructing a formalized expression model for the preprocessed valid frame gesture data, wherein the formalized expression model is used as input data of a subsequent HMM model, and the details are shown in fig. 3 (b). Defining an i-th frame gesture H _i is expressed as:

；

Wherein H _Li represents left hand gesture data, H _Ri represents right hand gesture data, and both may be represented as:

；

Wherein P _Li、P_Ri represents left and right palm center data respectively, Finger tip data of the thumb, the index finger, the middle finger, the ring finger and the little finger of the left hand are sequentially represented; /(I)Sequentially representing fingertip data of a thumb, an index finger, a middle finger, a ring finger and a little finger;

For palm data, taking the left hand palm as an example, P _Li can be expressed as:

；

wherein, Representing left palm heart coordinates,/>Representing the left palm center normal vector,/>Representing left palm heart rate;

For finger data, taking the thumb of the left hand as an example, Can be expressed as:

；

wherein, Representing the finger tip coordinates of the left hand thumb,/>Representing the finger tip direction vector of the left hand.

Step 3: inputting the preprocessed gesture data into an HMM model, and identifying a gesture sequence corresponding to the three-dimensional modeling operation;

step 4: based on the recognized gesture sequence, executing corresponding geometric modeling operation in cloud-native CAD software;

in one embodiment, based on the recognized gesture sequence, selecting a gesture category corresponding to the maximum probability as the recognized gesture; after the three-dimensional modeling intention of the user expressed by the gesture is obtained, executing corresponding three-dimensional modeling operation to realize three-dimensional modeling through the gesture.

In one embodiment, the HMM model of step 3 is a trained model; the gesture training sample set can be constructed, user gesture samples collected by the Leap Motion sensor are summarized, a gesture sample set which can be used as input data of an HMM model is constructed, meanwhile, different types of gestures in the gesture sample set are classified by constructing the HMM gesture training model, and further, correct classification processing, namely gesture recognition, can be carried out on newly input gesture data, gesture information is converted into three-dimensional modeling operation information through a gesture recognition three-dimensional modeling step, and corresponding three-dimensional model creation and operation functions are realized.

The specific training comprises the following substeps:

Step 3.1: constructing a gesture training sample set;

Let a single gesture sample completing a certain gesture be T represents the total number of valid frames. Then establish the sample set/>, of the gestureN represents the number of samples. Totally establish 11 sample sets/>As input data for the gesture recognition training of the HMM model.

Step 3.2: constructing an HMM gesture training model;

And (3) carrying out HMM model construction on the gestures in the 11-stage gesture training sample set, namely building a corresponding HMM model for each gesture according to gesture sample data. The process can be regarded as a gesture learning process, and the initial vector corresponding to each HMM model is trained by using a known gesture sample sequence and gesture categories corresponding to the known gesture sample sequence State transition matrixConfusion matrix/>; This step uses the Baum-Welch algorithm, as shown in FIG. 4. All gesture sequence data corresponding to each type of gesture in the gesture training sample set are sequentially input into the model, and each time new gesture sequence data are input, the initial vector, the state transition matrix and the confusion matrix are re-estimated, and the calculation formula is as follows:

；

wherein, The posterior probability is the probability that the state is i at the moment t when a gesture sample sequence and an HMM model are given; /(I)As a forward variable, the probability that the state at time t is q _i and the state at time t+1 is q _j is expressed; n represents the number of possible states and M represents the number of possible gesture observations. /(I)

To avoid the problem that the iteration process falls into local optimum and cannot acquire a global optimum solution, the method calculates at each iterationThen, adopting a simulated annealing algorithm to mix matrix/>The correction is carried out, and the steps are as follows:

Initializing: An annealing initial temperature T ₀, a cooling coefficient k, a termination temperature T _e, and a convergence condition ρ;

(1) Setting a cooling function ；

(4) For a pair ofNormalization processing is carried out,/>；

Through iterative calculation of 3 matrixes, a globally optimal solution of the HMM model can be obtained, namely, the HMM gesture training model corresponding to each gesture is trained through the gesture training sample set.

After the HMM gesture training model is successfully constructed, calculating the probability of generating observable new gesture sequence data by the model by the HMM gesture training models corresponding to 11 gestures, and selecting the gesture category corresponding to the maximum probability, namely successfully realizing gesture recognition. The process recursively calculates each probability value by adopting a forward algorithm in the HMM theory. After the three-dimensional modeling intention of the user expressed by the gesture is obtained, executing corresponding three-dimensional modeling operation, and realizing the operation of three-dimensional modeling through the gesture.

The embodiment also provides a cloud native CAD software gesture interaction geometric modeling system, which comprises the following modules:

the data acquisition and identification module is used for acquiring gestures for simulating mouse to control the online three-dimensional modeling software and identifying corresponding geometric model data;

the data preprocessing module is used for preprocessing gesture data;

The embodiment also provides a cloud-native CAD software gesture interaction geometric modeling device, which comprises:

One or more processors;

The embodiment also provides a cloud-native CAD software gesture interaction geometric modeling device, the overall architecture of which is shown in fig. 5, comprising: a software interaction unit; an acquisition unit; a real-time communication unit; a data processing unit; a storage unit; a calculation unit; and an execution unit.

And the user is connected with the Leap Motion sensor at the PC end and accesses the cloud primary CAD software deployed on the cloud server through the browser. The acquisition unit acquires gesture information of a user and the gesture information in real time and sends the gesture information to the real-time communication unit; the real-time communication unit encapsulates the acquired information into a lightweight data storage file-JSON file so as to adapt to the requirement of light weight of data transmitted by a network environment, and performs data real-time interaction with the cloud server through a 6437 port of the client by means of a WebSocket protocol, namely, the data captured by the Leap Motion sensor is uploaded to the cloud server in real time; the software interaction unit deploys cloud native CAD software, configures a Leap. Js library file, simulates click operation of a right button of a mouse by means of the library file and a vertical virtual light curtain of a Leap Motion sensor, and facilitates a user to control the cloud native CAD software directly through gestures; the data processing unit analyzes the JSON file, and normalizes all data into a gesture training sample set, so that the requirement of an HMM model on normalization of training data is met; the storage unit stores the processed gesture training sample set by adopting a relational database; the computing unit performs scientific computation based on the HMM gesture training model, and the cloud native CAD software can effectively recognize 11 geometric modeling gestures by receiving gesture data in the gesture training sample set and repeatedly training the gesture data; the execution unit responds to the specific geometric modeling gesture and executes the corresponding geometric modeling operation.

Further, a specific explanation is made with respect to the simulated mouse right click operation in the software interaction unit, as shown in fig. 6. The vertical virtual light curtain is provided with the attribute of the Leap Motion sensor and is positioned on an XOY plane of a three-dimensional coordinate system constructed by the Leap Motion sensor. The navigation auxiliary point is the mapping of X, Y coordinates of the finger tip of the right index finger of the user on the PC screen, and is used for assisting the user in observing the position of the finger tip of the right index finger, and the navigation auxiliary point has three color states of purple, yellow and red and corresponds to three events of right key loosening, right key hovering and right key clicking respectively. When the Z coordinate of the finger tip of the right index finger is larger than 10, the right key is released, the navigation auxiliary point is changed to be purple, and the finger can freely move to any button which is desired to be clicked. When the Z coordinate of the fingertip of the index finger of the right hand is between-1 and-1, the navigation auxiliary point is in a right key hovering state and turns yellow, and at the moment, the navigation auxiliary point cannot move, which means that a user is inquired whether to confirm clicking the positioned button. When the Z coordinate of the fingertip of the right index finger is smaller than 10, in the right click state, the navigation auxiliary point turns red, and the system executes a button clicking event, which is equivalent to clicking the button through a mouse. The function can simulate a mouse click button event, and the right index finger is used for clicking a button in the cloud native CAD software to operate the software.

The invention is further illustrated by the following specific experiments.

In the experiment, the traditional mouse and keyboard mode is adopted for geometric modeling man-machine interaction operation, and the Leap Motion is adopted for gesture interaction geometric modeling. The Leap Motion is adopted to carry out the three-dimensional modeling of the somatosensory, and the three-dimensional model is only needed to be created through gestures, so that a mouse and a keyboard do not need to be operated, and the man-machine interaction process is more natural.

In this experiment, as shown in fig. 7, the gesture data collected by the Leap Motion is uploaded to the cloud server in real time by using the conventional AJAX polling technology and WebSocket protocol, so as to perform gesture three-dimensional modeling, and the online three-dimensional modeling software renders the frame rate (FPS) variation trend in real time, the test time is 2 minutes, and the FPS value is collected every 1 s. As can be seen from FIG. 7, the FPS value of the communication using the AJAX polling technique fluctuates between about 45 and 60, and the FPS value of the communication using the WebSocket protocol fluctuates between about 65 and 85, which proves that the transmission efficiency is higher, the gesture recognition process is faster, and the geometric modeling process is smoother.

The experiment adopts a gesture track similarity matching method and an HMM model recognition method to recognize and test 11 predefined three-dimensional modeling gestures. A total of 20 user 11 gesture samples were collected, 10 for men, 10 for women, and different heights from 165 mm to 180 mm for men and from 155 cm to 170 cm for women, to test the effect of different sizes of hand patterns on gesture recognition. Fig. 8 (a) shows a gesture recognition confusion matrix based on a gesture track similarity matching method, and fig. 8 (b) shows a gesture recognition confusion matrix based on an HMM model. As can be seen from the figure, the average accuracy of gesture recognition based on the gesture track similarity matching method is only 79.1%, the accuracy of gesture recognition based on the HMM model is up to 90%, and the number of times that any gesture is misrecognized as other gestures is 1 at most, so that the HMM model has certain self-learning capability and the gesture recognition effect is relatively good.

The invention creates 6 basic geometries by defining different types of gestures, and performs operations of translation, rotation and scaling on the created geometries. A large number of samples are collected for different gestures, a gesture training sample set is constructed, and training is carried out by means of a Hidden Markov Model (HMM), so that gesture recognition three-dimensional modeling is realized. In addition, the invention also relates to a gesture interaction geometric modeling device of the cloud native CAD software, and the on-line three-dimensional modeling software for somatosensory control is realized by simulating a mouse control computer mode by using a right index finger by means of a vertical virtual light curtain of a Leap Motion sensor.

The cloud native CAD software gesture interaction geometric modeling scheme is mainly based on the technical principles of virtual reality, man-machine interaction, computer graphics and the like to assist a user in carrying out online geometric modeling operation based on a Web browser end in a gesture interaction mode. It should be noted that, this scheme is applicable to deploying cloud native CAD software on a cloud server, and the user can use this software through a personal computer and by means of a Leap Motion sensor.

It should be understood that the embodiments described above are some, but not all, embodiments of the invention. In addition, the technical features of each embodiment or the single embodiment provided by the invention can be combined with each other at will to form a feasible technical scheme, and the combination is not limited by the sequence of steps and/or the structural composition mode, but is necessarily based on the fact that a person of ordinary skill in the art can realize the combination, and when the technical scheme is contradictory or can not realize, the combination of the technical scheme is not considered to exist and is not within the protection scope of the invention claimed.

It should be understood that the foregoing description of the preferred embodiments is not intended to limit the scope of the invention, but rather to limit the scope of the claims, and that those skilled in the art can make substitutions or modifications without departing from the scope of the invention as set forth in the appended claims.

Claims

1. The gesture interaction geometric modeling method of the cloud native CAD software is characterized by comprising the following steps of:

step 1: predefining a plurality of gestures, collecting gestures for simulating a mouse to control cloud native CAD software, and identifying corresponding geometric modeling operation;

Step 2: uploading the captured gesture data to a cloud server for deploying cloud native CAD software in real time based on a WebSocket protocol, and preprocessing the gesture data;

Step 3: inputting the preprocessed gesture data into a hidden Markov HMM model, and identifying a gesture sequence corresponding to geometric modeling operation;

2. The cloud-native CAD software gesture interaction geometric modeling method of claim 1, wherein: in step 1, the geometric modeling operation includes creating a cube model, creating a cuboid model, creating a cylinder model, creating a cone model, creating a sphere model, creating a torus model, translating a model, rotating the model around an X-axis, rotating the model around a Y-axis, rotating the model around a Z-axis, and scaling the model.

3. The cloud-native CAD software gesture interaction geometric modeling method of claim 1, wherein: in step 2, the start and end states of the gesture are identified, and the palm P _Ri rate of the right hand H _Ri in the ith frame data is recorded asWhen the rate is smaller than a set threshold epsilon, the right hand is considered to be in a static state, the frame data is invalid, and when the rate is larger than the threshold epsilon, the right hand is considered to be in a motion state, and the frame data is valid;

Normalizing three-dimensional coordinate values of tracks in all effective frame data, and recording the space coordinate of palm P _Ri of right hand H _Ri as For each gesture track, find out the maximum of the gesture tracks on the X axis, the Y axis and the Z axisAnd minimum/>And calculates the compression ratio/>, of the track on three coordinate axesTransforming the trajectory into a space of 10 ³;

4. The cloud-native CAD software gesture interaction geometric modeling method of claim 1, wherein: in step 2, the i-th frame gesture H _i after preprocessing is recorded as:

；

wherein, Representing left hand gesture data,/>Representing right hand gesture data; /(I)Data representing left and right palmar heart respectively,/>Finger tip data of the thumb, the index finger, the middle finger, the ring finger and the little finger of the left hand are sequentially represented; sequentially representing fingertip data of a thumb, an index finger, a middle finger, a ring finger and a little finger;

；

wherein, Representing the finger tip coordinates of the left hand thumb,/>Representing a left thumb fingertip direction vector; and obtaining the product in the same way.

5. The cloud-native CAD software gesture interaction geometric modeling method of claim 1, wherein: in step 4, based on the recognized gesture sequence, selecting a gesture category corresponding to the maximum probability as a recognized gesture; after the three-dimensional modeling intention of the user expressed by the gesture is obtained, executing corresponding three-dimensional modeling operation to realize three-dimensional modeling through the gesture.

6. The cloud native CAD software gesture interaction geometry modeling method according to any one of claims 1-5, wherein: the HMM model in the step 3 is a trained model;

the specific training comprises the following substeps:

Step 3.1: constructing a gesture training sample set;

Step 3.2: training a model;

；

wherein, The posterior probability is the probability that the state is i at the moment t when a gesture sample sequence and an HMM model are given; As a forward variable, the probability that the state at time t is q _i and the state at time t+1 is q _j is expressed; n represents the number of states and M represents the number of gesture observations.

7. The cloud-native CAD software gesture interaction geometric modeling method of claim 6, wherein: each iterative calculationThen, adopting a simulated annealing algorithm to mix matrix/>The correction is carried out, specifically comprising the following substeps:

(2) Setting a cooling function ；

(4) For a pair ofNormalization processing is carried out,/>；

8. The gesture interaction geometric modeling system of the cloud native CAD software is characterized by comprising the following modules:

The data acquisition and recognition module is used for predefining a plurality of gestures, acquiring the gestures for simulating the mouse to control the online three-dimensional modeling software and recognizing the corresponding geometric model data;

the data preprocessing module is used for uploading the captured gesture data to a cloud server for deploying cloud native CAD software in real time based on a WebSocket protocol, and preprocessing the gesture data;

9. The utility model provides a cloud native CAD software gesture interactive geometry modeling device which is characterized in that includes: the system comprises a software interaction unit, an acquisition unit, a real-time communication unit, a data processing unit, a storage unit, a calculation unit and an execution unit;

10. A cloud-native CAD software gesture interaction geometry modeling apparatus, comprising:

One or more processors;

storage means for storing one or more programs that, when executed by the one or more processors, cause the one or more processors to implement the cloud native CAD software gesture interaction geometry modeling method of any of claims 1-7.