CN115798054A - Gesture recognition method based on AR/MR technology and electronic device - Google Patents
Gesture recognition method based on AR/MR technology and electronic device Download PDFInfo
- Publication number
- CN115798054A CN115798054A CN202310091517.5A CN202310091517A CN115798054A CN 115798054 A CN115798054 A CN 115798054A CN 202310091517 A CN202310091517 A CN 202310091517A CN 115798054 A CN115798054 A CN 115798054A
- Authority
- CN
- China
- Prior art keywords
- gesture recognition
- setting
- thumb
- gesture
- time
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 82
- 238000005516 engineering process Methods 0.000 title claims abstract description 42
- 210000003813 thumb Anatomy 0.000 claims abstract description 79
- 210000003811 finger Anatomy 0.000 claims abstract description 67
- 230000008569 process Effects 0.000 claims abstract description 40
- 238000007667 floating Methods 0.000 claims description 59
- 230000008859 change Effects 0.000 claims description 18
- 238000001816 cooling Methods 0.000 claims description 3
- 238000004590 computer program Methods 0.000 claims description 2
- 238000007405 data analysis Methods 0.000 abstract description 2
- 238000012549 training Methods 0.000 description 6
- 230000006870 function Effects 0.000 description 5
- 210000004247 hand Anatomy 0.000 description 5
- 238000013473 artificial intelligence Methods 0.000 description 3
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 230000009471 action Effects 0.000 description 2
- 230000003190 augmentative effect Effects 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 238000013499 data model Methods 0.000 description 2
- 230000004424 eye movement Effects 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 230000002452 interceptive effect Effects 0.000 description 2
- 238000007726 management method Methods 0.000 description 2
- 238000012544 monitoring process Methods 0.000 description 2
- 239000004278 EU approved seasoning Substances 0.000 description 1
- 230000004913 activation Effects 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 230000015556 catabolic process Effects 0.000 description 1
- 238000010411 cooking Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 235000013312 flour Nutrition 0.000 description 1
- 235000011194 food seasoning agent Nutrition 0.000 description 1
- 230000010365 information processing Effects 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 235000012054 meals Nutrition 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 208000013860 rhabdoid tumor of the kidney Diseases 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Landscapes
- User Interface Of Digital Computer (AREA)
Abstract
The invention provides a gesture recognition method and electronic equipment based on AR/MR technology, which belongs to the technical field of spatial data analysis and recognition and is used for setting and storing relevant variables in the gesture recognition process; setting constant information as a gesture recognition judgment basis, executing a gesture recognition engine, and performing recognition judgment on each frame: judging whether the SDK tracks the thumb fingertips and the ring finger fingertips, and if any one of the fingertips is not tracked, ending the frame identification process; if the finger tips of the thumb and the finger tips of the ring finger are identified at the same time, recording the position of the finger tips of the thumb of the current frame, namely the spatial three-dimensional coordinate; the gesture recognition method and the gesture recognition system can collect the related variable and constant information of gesture recognition and the gesture recognition engine, are convenient for a user to refer to and use, and are convenient for the system to execute a corresponding gesture recognition process according to the related variable and constant information and the gesture recognition engine. The accuracy and precision of gesture recognition are effectively improved.
Description
Technical Field
The invention relates to the technical field of spatial data analysis and recognition, in particular to a gesture recognition method based on an AR/MR technology and electronic equipment.
Background
At present, the application prospect of the AR/MR technology is very wide, wherein AR (augmented reality) is a technology for calculating the position and angle of the camera image in real time and adding corresponding image, video and 3D models, and MR is a mixed reality technology. Generally speaking, AR superimposes a virtual image on an actual scene, and MR superimposes a virtual image on an actual scene, and then can interact with the virtual image. The most prominent characteristic of AR/MR is that the hands of the user can be liberated through non-contact operation of gesture, voice or eye movement fixation without using an operation mode of occupying the hands such as a keyboard, a mouse, a handle or a touch screen, and data can be acquired and operated more conveniently and rapidly.
Compared with the traditional interactive platform, the most prominent characteristic of AR/MR is that the hands of a user can be liberated through non-contact operation of gesture, voice or eye movement gazing without using an operation mode of occupying the hands such as a keyboard, a mouse, a handle or a touch screen, and data can be acquired and operated more conveniently and rapidly. At present, through the characteristic of liberation both hands, when for example overhauing equipment in the industry, the user can look up the data, seek long-range assistance or put the drawing and carry out real-time comparison beside the equipment when handheld tool overhauls, needn't put down the instrument of hand and pick up the manual or the flat board repeatedly and look up. When the user learns the cooking art in life, the user does not need to worry about that the mobile phone is polluted by a hand sliding screen stuck with seasonings or flour and the meal is reversely polluted when the user looks up the menu.
For another example, in production and life, it may be inconvenient to touch a screen, take paper materials or operate equipment due to problems such as site limitation, handheld tools, stain influence and the like, and thus a gesture recognition mode is required to call up a resource list.
The current gesture recognition mainly adopts an artificial intelligence image training mode, and the mode faces three problems that the training period is long, an image data set needs to be trained, a training model meeting requirements is matched, and the accuracy of gesture recognition is affected if the training model is not matched or the model accuracy is low. Secondly, the data model quantity is large, a large amount of system resources are consumed, the system operation is blocked, and the identification requirement cannot be met. And the lack of three-bit data alignment accuracy has a certain effect only by plane image judgment.
Disclosure of Invention
The invention provides a gesture recognition method based on an AR/MR (augmented reality/magnetic resonance) technology, which comprehensively analyzes the accuracy of gestures through 2D (two-dimensional) and 3D (three-dimensional) data of skeleton points.
The gesture recognition method based on the AR/MR technology comprises the following steps:
s101, setting and storing related variables in a gesture recognition process;
s102, setting constant information used as a gesture recognition judgment basis, wherein the constant information is a value defined according to AR/MR equipment and a use environment;
s103, executing a gesture recognition engine, and performing the following recognition judgment on each frame:
judging whether the SDK tracks the thumb tip and the ring finger tip, if any one of the finger tips is not tracked, setting IfTrack as false, and setting GesturState as the following value: the frame identification process is finished when the identification is not carried out;
if the thumb fingertip and the ring finger fingertip are identified at the same time, recording the position of the current frame thumb fingertip, namely the space three-dimensional coordinate by using thumb Tip, and recording the position of the current frame ring finger fingertip, namely the space three-dimensional coordinate by using RingTip;
if the state value of GesturEST is successfully identified, recording the time point-EndTm of the current frame by using IntervalTm, namely the time from the last successful judgment to the current frame;
if IntervalTm is greater than ReAcvTm, the state value of GesturState is changed to be unrecognized; setting OriTm as a time point of a current frame; ringTipOri is set to the position of the current ring finger tip; the ThumbTipOri is set as the position of the current thumb tip, and the frame identification process is directly finished.
It should be further noted that the relevant variables set and stored in step S101 include:
setting a three-dimensional floating point number variable RingTip, and recording and tracking the position information of the ring finger tip in real time;
setting a three-dimensional floating point number variable RingTipOri, recording a three-dimensional space position coordinate of the ring finger tip at a certain time point, and comparing the position change of the ring finger tip;
setting a three-dimensional floating point number variable thumb tip, recording and tracking the fingertip position of the thumb in real time, and updating the three-dimensional space position coordinate of the thumb acquired by the current frame SDK;
and setting a three-dimensional floating point number variable ThumbTipOri, and recording the three-dimensional space position coordinates of the thumb and the fingertip at a certain time point for comparing the position change of the thumb and the fingertip.
It should be further noted that the relevant variables set and stored in step S101 further include:
setting an enumeration type variable GestureState, and identifying the state value of the enumeration type variable, wherein the state value comprises the following steps: 1. not identifying, 2, identifying, 3, successfully identifying, and 4, forbidding;
the default state of the enumerated variable GestureState is: no identification is performed;
setting a Boolean type variable IfTrack, and recording whether the first time is to simultaneously recognize the finger tips of the thumb and the ring finger;
setting a floating point type variable oriTm and recording a time point of starting gesture recognition, wherein the time point is the number of seconds from the time when a program starts to run to the time when the program is recorded;
setting a floating point variable EndTM, and recording a time point when gesture recognition is successful, wherein the time point is the number of seconds from the time when a program starts to run to the time when the gesture recognition is recorded;
setting a floating point type variable IntervalTm;
the floating point type variable quantity DisThumbTipToRingTip is set, and the distance information from the thumb tip to the ring finger tip is recorded.
Further, step S102 further includes: setting a floating point type constant StartDis as a distance condition for judging whether to start gesture recognition; when the distance between the RingTip and the thumb is smaller than StartDis, starting to judge whether the gesture condition is met;
setting a floating point constant EndDis as a distance which needs to be reached by the gesture, and when the distance between the thumb tip and the Ringtip exceeds the EndDis, indicating that the gesture position reaches a completion standard;
setting a floating point constant GusturmeminTm as a limiting condition of a lower limit of gesture judgment time, namely determining the gesture recognition state as an effective gesture recognition state if the elapsed recognition time is more than GusturmeminTm after gesture recognition is started and if the gesture position change condition is met;
setting a limitation condition GesturEMaxTm of the upper limit of the judgment time of the floating point type constant gesture; after gesture recognition is started, the gesture position change condition is met within the GesturmemaxTm time.
Further, step S102 further includes: setting a floating point constant GesturreleaseTm as time for stopping gesture recognition; after gesture recognition is started, if the distance between the thumb tip and the Ringtip is not changed within the GesturreleaseTm time, the gesture recognition is not performed any more;
setting a floating point constant GesturreleaseDis to judge the distance for stopping gesture recognition, namely judging whether the distance change of thumb and RingTip exceeds a preset distance or not during gesture recognition, such as in GesturreleaseTm time, and stopping current gesture recognition if the distance change of thumb and RingTip does not exceed the preset distance;
and setting a constant value of the floating point number, namely, the Retvtm, as the cooling time of the gesture.
It should be further noted that, in step S103, the condition for ending the frame identification process further includes:
if the GestureState is not recognized, judging whether the ThumbTip is consistent with the ThumbTIpOri or whether the RingTip is consistent with the RingTipOri, and if so, finishing the frame recognition process;
if the distance between the thumb tip and the ring finger tip is larger than StartDis, the frame identification process is finished;
after the end, the state value of getturestate is set to the recognition execution state.
It should be further noted that after setting the state value of getturestate to the recognition state, setting IntervalTm to the current time minus OriTm;
setting DisThumbTipToRingTip to be the distance from the current thumb fingertip to the ring finger fingertip;
if IntervalTm is greater than ReleaseTm or DisThumbTipToRingTip is greater than ReleaseDis, the GestureState state value is set as recognition-not-started, oriTm is set as the current time point, ringTipOri is set as the current position of the ring finger tip, thumbTipOri is set as the current position of the thumb tip, and the frame recognition process is ended;
if IntervalTm is less than GesturEMaxTm, and IntervalTm is greater than GesturEMinTm, dishumbTipToRingTip is greater than GestureDis, then GestureState status is set to: and if the identification is successful, endTm is set as the current time point, and the frame identification process is ended.
It should be further noted that if invaltm is greater than geturetm, which indicates that the recognition is timed out, the geturestate value is set to: not recognized, oriTm is set as the current time point, thumbTipOri is set as the position of the thumb tip at this time, and RingTipOri is set as the position of the ring finger tip at this time.
It should be further noted that if IfTrack is false, and the thumb tip and the ring finger tip have not been recognized before the frame, the positions of the joints of the thumb tip and the ring finger tip are obtained and recorded in ThumbTipOri and RingTipOri, respectively, and the current time point is recorded in OriTm.
The invention also provides an electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the steps of the AR/MR technology based gesture recognition method when executing the program.
According to the technical scheme, the invention has the following advantages:
the gesture recognition system and the gesture recognition method can summarize related variables, constant information and a gesture recognition engine of gesture recognition, are convenient for a user to use for reference, and are convenient for the system to execute a corresponding gesture recognition process according to the related variables, the constant information and the gesture recognition engine. The accuracy and precision of gesture recognition are effectively improved. The gesture recognition information can be efficiently collected, stored and processed, process monitoring can be achieved based on the gesture recognition state and the states of the thumb finger tip and the ring finger tip, and the whole gesture recognition process is described by using a multi-dimensional space. The method and the device improve the quality and efficiency of gesture recognition, timely find the abnormity in the gesture recognition process, and correct the abnormity so as to improve the execution accuracy of the AR/MR technology, meet the use requirements, and realize the timeliness and scientificity of the gesture recognition overall process supervision, management and control based on the AR/MR technology.
The invention provides a gesture recognition method based on AR/MR technology, and the method can also record the tracked position of a ring finger tip in real time by setting a three-dimensional floating point variable RingTip, recording the three-dimensional space position coordinate of the ring finger tip at a certain time point by the three-dimensional floating point variable RingTiori, recording the tracked position of the thumb tip in real time by the three-dimensional floating point variable ThumbTip, recording the three-dimensional space position coordinate of the thumb tip at a certain time point by the three-dimensional floating point variable ThumbTipOri, recording the recognition state of an enumerated variable GesturteState, recording the time point of the start of gesture recognition by the floating point variable OriTm, recording the time point when the gesture recognition succeeds by the floating point variable EndTm, recording the time of each state by the floating point variable IntervalTm, recording a plurality of parameter information such as the distance from the thumb tip to the ring finger tip and the like, and can judge and recognize the fused gesture scene of the thumb tip and the ring finger tip by combining the judgment.
The three-dimensional data variable is set, so that each frame of the three-dimensional data variable in an engine can be recognized and judged, the gesture is comprehensively analyzed by utilizing the three-dimensional data, the accuracy of recognizing the unfolding gesture is improved, and the problems that the artificial intelligent image training is adopted, judgment can be only carried out through a plane image, three-dimensional data is lacked, certain influence is caused on the accuracy, the training period is long, the data model quantity is large, and the judgment condition and the judgment precision of the gesture are not favorably and rapidly adjusted are solved.
The invention can dynamically register and cancel the identification of the skeletal points. And comprehensively analyzing the accuracy of the gesture through the 2D and 3D data of the skeleton points. The gesture recognition method has the advantages that the gesture judgment condition and the gesture judgment precision can be quickly adjusted, the positions of fingertips are confirmed, the gesture recognition function is more stable, the use requirements of users are met, and the reliability of gesture recognition in the AR/MR technology is guaranteed.
Drawings
In order to more clearly illustrate the technical solution of the present invention, the drawings required to be used in the description will be briefly introduced below, and it is apparent that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art that other drawings can be obtained based on these drawings without creative efforts.
FIG. 1 is a flow chart of a gesture recognition method based on AR/MR technology.
Detailed Description
The gesture recognition method based on the AR/MR technology can acquire and process the associated data based on the artificial intelligence technology. The gesture recognition method based on the AR/MR technology simulates, extends and expands human intelligence by using a digital computer or a machine controlled by the digital computer, senses the environment, acquires knowledge and obtains the theory, method, technology and application device of the best result by using the knowledge.
The gesture recognition method based on the AR/MR technology has the hardware level technology and the software level technology. The basic technologies of the gesture recognition method generally include technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation/interaction systems, electromechanical integration and the like. The software technology of the gesture recognition method mainly comprises a computer visual angle technology, machine learning/deep learning and a programming language. The programming languages include, but are not limited to, object oriented programming languages such as Java, smalltalk, C + +, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the AR/MR device, partly on the user computer, partly on a remote computer or entirely on the remote computer, as a stand-alone software package. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
For purposes of the present invention, the gesture recognition engine, or 3D engine, is the 3D engine used to develop AR/MR software, such as Unity, UE, etc.
The time point is a time point referred to in the present invention, and refers to a time point in the 3D engine, specifically to a number of seconds elapsed from when the program starts to run as a time starting point to each time the time point is acquired.
And (3) SDK: the development tool provided by the equipment manufacturer has basic equipment function call, for example, the microsoft SDK of the Hololens manufacturer is MRTK, and can track the spatial position information of the required hand skeleton point in real time.
Frame: the unit of calculating frequency in the engine is one frame every time the picture in the engine is updated, and within the time of each frame, the unit of calculating frequency in the engine is one calculation period in the engine.
The gesture recognition method utilizes AR/MR technology, sets relevant variables and relevant constant information of gesture recognition, executes a gesture recognition engine, and carries out recognition judgment on each frame as follows, so that non-contact operation based on gesture actions is realized, and a user can acquire and operate data more conveniently and rapidly. The method can be effectively applied to remote control equipment or multiple fields of data lookup and the like. The problems that the traditional gesture recognition is inaccurate, gesture information cannot be effectively acquired, the AR/MR technology cannot be effectively operated, and user experience is influenced are further effectively solved.
The gesture recognition method may be applied to one or more electronic devices, which are devices capable of automatically performing numerical calculation and/or information processing according to preset or stored instructions, and the hardware thereof includes, but is not limited to, a microprocessor, an Application Specific Integrated Circuit (ASIC), a Programmable gate array (FPGA), a Digital Signal Processor (DSP), an embedded device, and the like.
The electronic device may be any electronic product capable of performing human-computer interaction with a user, for example, a Personal computer, a tablet computer, a smart phone, a Personal Digital Assistant (PDA), an interactive web Television (IPTV), an AR/MR smart wearable device, and the like.
The Network where the electronic device is located includes, but is not limited to, the internet, a wide area Network, a metropolitan area Network, a local area Network, a Virtual Private Network (VPN), and the like.
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Referring to fig. 1, which is a flowchart illustrating a gesture recognition method based on AR/MR technology according to an embodiment, S101 sets and stores related variables in a gesture recognition process;
specifically, 1.1, setting a three-dimensional floating point number variable RingTip to record the tracked fingertip position of the ring finger in real time, namely updating the three-dimensional space position coordinates of the ring finger acquired by the current frame SDK.
And setting a three-dimensional floating point number variable RingTipOri to record the three-dimensional space position coordinates of the ring finger tip at a certain time point, and comparing the position change of the ring finger tip at a specific stage.
1.2, setting a three-dimensional floating point number variable thumb tip to record the tracked thumb fingertip position in real time, namely updating the three-dimensional space position coordinate of the thumb acquired by the current frame SDK.
And setting a three-dimensional floating point number variable ThumbTipOri to record the three-dimensional space position coordinates of the thumb and the fingertip at a certain time point, and comparing the position change of the thumb and the fingertip at a specific stage.
1.3 setting enumeration type variable GestureState recognition state, the possible state values are four kinds respectively: 1. no identification is made, 2 identification is in progress, 3 identification is successful, and 4 is disabled. The default state of GestureState is: no identification is made.
1.4 set Boolean type variable IfTrack record whether the thumb and ring finger tips are recognized for the first time simultaneously. The initial value is false.
1.5 set the floating point type variable OriTm records the time point of the start of gesture recognition, which is the number of seconds elapsed from when the program started running to when it was recorded.
1.6 sets the time point when the floating point type variable EndTM records the gesture recognition successfully, which is the number of seconds elapsed from the start of the program to the recording.
1.7 a floating point type variable IntervalTm is set to record the time elapsed for each state, for example, the time elapsed from the last successful gesture recognition to the recording after the successful recognition is recorded, or the time elapsed from the beginning of the recognition to the recording is recorded.
1.8 set the floating point variable DisthumbTipToRingTip, record the distance from the thumb tip to the ring finger tip.
S102, setting constant information used as a gesture recognition judgment basis, wherein the constant information is a value defined according to AR/MR equipment and a use environment;
2.1 a floating point constant StartDis is set as a distance condition for judging whether to start gesture recognition, that is, when the distance between RingTip and ThumbTip is smaller than the StartDis, whether the gesture condition is met is judged, and the value of StartDis set here is 0.035.
2.2 setting a floating point constant EndDis as a distance which needs to be reached by the gesture, namely when the distance between the thumb and RingTip exceeds EndDis, indicating that the gesture position reaches a completion standard, and setting the value of EndDis to be 0.5.
2.3 The floating point constant GustureMinTm is set as a limiting condition of a lower limit of gesture judgment time, namely, after gesture recognition is started, the recognition time which passes when the gesture position change condition is met is greater than GustureMinTm, and the value is set to be 0.13.
2.4, the limitation condition geturemaxtm of the upper limit of the judgment time of the floating point type constant gesture is set, that is, after gesture recognition is started, all conditions of gesture position change are met within the duration of geturemaxtm, and the value of the limitation condition geturemaxtm is set to be 0.5.
2.5 The floating point constant GesturreleaseTm is set as the time for stopping gesture recognition, namely after gesture recognition is started, if the thumb tip and the Ringtip do not have enough distance change within the GesturreleaseTm time, the user is not considered to be ready to perform the gesture, and gesture recognition is not performed any more.
2.6 And setting a floating point constant GesturreleaseDis to judge the distance for stopping gesture recognition, namely after gesture recognition is started, judging whether the thumb tip and the Ringtip have enough distance change within GesturreleaseTm according to a judgment standard, and setting the value to be 0.038.
2.7 The cooling time of the floating point number type constant RecavTm gesture is set, namely after the gesture is recognized to be completed, the gesture is not recognized within several seconds, and the functions are two:
1) Because the gesture will not be continuously triggered in a short time under normal conditions, if the situation that the judgment standard is continuously met occurs, the probability is that misoperation causes most.
2) The approximate probability of the joints as recognition criteria after the activation of the action still meets the criteria for achieving the gesture, which results in the gesture being successfully recognized with high frequency continuously and leads to program breakdown, and the value is set to be 1.5.
S103, executing a gesture recognition engine, and performing recognition judgment on each frame, wherein the specific judgment steps are as follows:
3.1 judging whether the SDK tracks the finger tips of the thumb and the ring finger, if any finger tip is not tracked, setting IfTrack to false, and setting GesturState to be: no identification is made. The frame identification process is directly ended.
3.2 If the thumb tip and the ring finger tip are identified at the same time, the position of the current frame thumb tip, namely the space three-dimensional coordinate, is recorded by using the thumb tip, and the position of the current frame ring finger tip, namely the space three-dimensional coordinate, is recorded by using the Ringtip.
3.3 if ifTrack is false, after the program starts running or the last gesture recognition is successful, the positions of the joints of the thumb tip and the ring finger tip are obtained and recorded in ThumbTipOri and RingTipOri respectively if the positions of the joints of the thumb tip and the ring finger tip are not recognized simultaneously before the frame. The current time point is recorded in OriTm. The value of ifTrack is changed to true.
3.4 if the state value of GesturStatate is recognized successfully, recording the time point-EndTm of the current frame by using IntervalTm, namely the time from the last successful judgment to the current frame.
If IntervalTm is greater than ReAcvTm, the state value of GesturState is set to unrecognized. OriTm is set as the time point of the current frame. RingTipOri is set to the position of the current ring finger tip. The ThumbTipOri is set to the current thumb tip position. The frame identification process ends directly.
3.4 If GesturESTate is not recognized, then:
3.4.1 if ThumbTip is consistent with ThumbTIpOri, or RingTip is consistent with RingTipOri, the present frame identification process ends directly.
3.4.2 If the distance between the tip of the thumb and the tip of the ring finger is greater than StartDis, the frame identification process is directly ended.
3.4.3 The state value of geturestate is set to recognition in progress.
3.5 If the state value of GestureState is recognizing in progress, then
3.5.1 if RingTip is consistent with RingTipOri or ThumbTip is consistent with ThumbTipOri, the present frame identification process ends directly.
3.5.2 IntervalTm is set to the current time point minus OriTm.
3.5.3 Distumbtiptorringtip is set to the current thumb-fingertip distance to the ring finger-fingertip distance.
3.5.4 If IntervalTm is greater than ReleaseTm or distumbtiptorngtip is greater than ReleaseDis, the getturestate value is set to be not started to recognize, oriTm is set to be the current time point, ringTipOri is set to be the position of the current ring finger tip, thumbTipOri is set to be the position of the current thumb tip, and the frame recognition process is directly ended.
3.5.5 if IntervalTm is less than GesturEMaxTm, and IntervalTm is greater than GesturEMINTm, and DisthenbTipToRingTip is greater than GesturDis, then GestureState status is set to: having identified success, endTm is set to the current point in time. The gesture of blooming is recognized, and the frame recognition process is directly finished.
3.6 if IntvalTm is greater than GesturreTm, indicating a recognition timeout, gesturreState status value is set to: recognition is not started, oriTm is set as the current time point, thumbTipOri is set as the position of the thumb tip at this time, and RingTipOri is set as the position of the ring finger tip at this time.
It should be understood that, the sequence numbers of the steps in the foregoing embodiments do not imply an execution sequence, and the execution sequence of each process should be determined by its function and inherent logic, and should not constitute any limitation to the implementation process of the embodiments of the present invention.
Therefore, the gesture recognition method and the gesture recognition system can collect the relevant variable, the constant information and the gesture recognition engine of gesture recognition, are convenient for a user to refer to and use, and are convenient for the system to execute a corresponding gesture recognition process according to the relevant variable, the constant information and the gesture recognition engine. The accuracy and precision of gesture recognition are effectively improved. The gesture recognition information can be efficiently collected, stored and processed, process monitoring can be achieved based on the gesture recognition state and the states of the thumb fingertips and the ring finger fingertips, and the whole gesture recognition process is described by using a multidimensional space. The method and the device improve the quality and efficiency of gesture recognition, timely find the abnormity in the gesture recognition process, and correct the abnormity so as to improve the execution accuracy of the AR/MR technology, meet the use requirements, and realize the timeliness and scientificity of the gesture recognition overall process supervision, management and control based on the AR/MR technology.
The invention provides a gesture recognition method based on AR/MR technology, and the method can also record the tracked position of a ring finger tip in real time by setting a three-dimensional floating point variable RingTip, recording the three-dimensional space position coordinate of the ring finger tip at a certain time point by the three-dimensional floating point variable RingTiori, recording the tracked position of the thumb tip in real time by the three-dimensional floating point variable ThumbTip, recording the three-dimensional space position coordinate of the thumb tip at a certain time point by the three-dimensional floating point variable ThumbTipOri, recording the recognition state of an enumerated variable GesturteState, recording the time point of the start of gesture recognition by the floating point variable OriTm, recording the time point when the gesture recognition succeeds by the floating point variable EndTm, recording the time of each state by the floating point variable IntervalTm, recording a plurality of parameter information such as the distance from the thumb tip to the ring finger tip and the like, and can judge and recognize the fused gesture scene of the thumb tip and the ring finger tip by combining the judgment. The invention can dynamically register and cancel the identification of the skeleton points. And comprehensively analyzing the accuracy of the gesture through the 2D and 3D data of the skeleton points. The gesture recognition method has the advantages that the gesture judgment condition and the gesture judgment precision can be rapidly adjusted, the positions of fingertips are confirmed, the gesture recognition function is more stable, the use requirements of users are met, and the reliability of gesture recognition in the AR/MR technology is guaranteed.
The elements and algorithm steps of the examples described in the embodiments of the AR/MR technology based gesture recognition method according to the present invention can be implemented in electronic hardware, computer software, or a combination of both, and in the above description the components and steps of the examples have been generally described in terms of functionality in order to clearly illustrate the interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
In the several embodiments provided in the present application, it should be understood that the disclosed system, apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the units is only one logical division, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may also be an electric, mechanical or other form of connection.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (10)
1. A gesture recognition method based on AR/MR technology is characterized by comprising the following steps:
s101, setting and storing related variables in a gesture recognition process;
s102, setting constant information used as a gesture recognition judgment basis, wherein the constant information is a value defined according to AR/MR equipment and a use environment;
s103, executing a gesture recognition engine, and performing the following gesture recognition judgment on each frame:
judging whether the SDK tracks the thumb tip and the ring finger tip, if any one of the finger tips is not tracked, setting IfTrack as false, and setting GesturState as the following value: if the frame is not identified, the frame identification process is finished;
if the thumb fingertip and the ring finger fingertip are identified at the same time, recording the position of the current frame thumb fingertip, namely the space three-dimensional coordinate by using thumb Tip, and recording the position of the current frame ring finger fingertip, namely the space three-dimensional coordinate by using RingTip;
if the state value of GesturEST is successfully identified, recording the time point-EndTm of the current frame by using IntervalTm, namely the time from the last successful judgment to the current frame;
if IntervalTm is greater than ReAcvTm, the state value of GesturState is changed to unrecognized; setting OriTm as a time point of a current frame; ringTipOri is set to the position of the current ring finger tip; the thumb TipOri is set as the current position of the thumb tip, and the frame identification process is directly ended.
2. The method for recognizing gestures based on AR/MR technology according to claim 1, wherein the relevant variables set and stored in step S101 include:
setting a three-dimensional floating point number variable RingTip, and recording and tracking the position information of the ring finger tip in real time;
setting a three-dimensional floating point number variable RingTipOri, recording a three-dimensional space position coordinate of the fingertip of the ring finger at a certain time point, and comparing the position change of the fingertip of the ring finger;
setting a three-dimensional floating point number variable ThumbTip, recording and tracking the position of a fingertip of the thumb in real time, and updating the three-dimensional space position coordinate of the thumb obtained by the current frame SDK;
and setting a three-dimensional floating point number variable ThumbTipOri, and recording the three-dimensional space position coordinates of the thumb and the fingertip at a certain time point for comparing the position change of the thumb and the fingertip.
3. The method for recognizing gestures based on AR/MR technology according to claim 1, wherein the related variables set and stored in step S101 further comprise:
setting an enumeration type variable GestureState, and identifying the state value of the enumeration type variable, wherein the state value comprises the following steps: 1. not identifying, 2, identifying, 3, successfully identifying, and 4, forbidding;
the default state of the enumerated variable GestureState is: no identification is performed;
setting a Boolean type variable IfTrack, and recording whether the fingertips of the thumb and the ring finger are recognized for the first time;
setting a floating point type variable oriTm and recording a time point of starting gesture recognition, wherein the time point is the number of seconds from the time when a program starts to run to the time when the program is recorded;
setting a floating point type variable EndTM, and recording a time point when gesture recognition is successful, wherein the time point is the number of seconds from the start of running of a program to the recording;
setting a floating point type variable IntervalTm;
the floating point type variable quantity DisThumbTipToRingTip is set, and the distance information from the thumb tip to the ring finger tip is recorded.
4. The method for gesture recognition based on AR/MR technology according to claim 1, wherein step S102 further comprises: setting a floating point type constant StartDis as a distance condition for judging whether to start gesture recognition; when the distance between the RingTip and the thumb is smaller than StartDis, starting to judge whether the gesture condition is met;
setting a floating point constant EndDis as a distance which needs to be reached by the gesture, and when the distance between thumb tip and Ringtip exceeds EndDis, indicating that the gesture position reaches a completion standard;
setting a floating point constant GusturmeminTm as a limiting condition of a lower limit of gesture judgment time, namely determining the gesture recognition state as an effective gesture recognition state if the elapsed recognition time is more than GusturmeminTm after gesture recognition is started and if the gesture position change condition is met;
setting a limitation condition GesturmemaxTm of a floating point type constant gesture judgment time upper limit; after gesture recognition is started, the gesture position change condition is satisfied within the time of GesturmeMaxTm.
5. The method for gesture recognition based on AR/MR technology according to claim 1, wherein step S102 further comprises: setting a floating point constant GesturreleaseTm as time for stopping gesture recognition; after gesture recognition is started, if the distance between the thumb tip and the Ringtip is not changed within the GesturreleaseTm time, the gesture recognition is not performed any more;
setting a floating point constant GestureleaseDis to judge the distance for stopping gesture recognition, namely judging whether the distance change of thumb tip and Ringtip exceeds a preset distance or not during gesture recognition within GestureleaseTm time, and stopping current gesture recognition if the distance change of thumb tip and Ringtip does not exceed the preset distance;
and setting a constant value of the floating point number, namely, the Retvtm, as the cooling time of the gesture.
6. The method for recognizing gesture based on AR/MR technology according to claim 1, wherein the ending condition of the frame recognition process in step S103 further includes:
if the GestureState is not recognized, judging whether the ThumbTip is consistent with the ThumbTIpOri or whether the RingTip is consistent with the RingTipOri, and if so, finishing the frame recognition process;
if the distance between the finger tip of the thumb and the finger tip of the ring finger is greater than StartDis, the frame identification process is ended;
after the end, the state value of getturestate is set to the recognition execution state.
7. The AR/MR technology-based gesture recognition method according to claim 6, wherein after setting the state value of getcurestate to recognition state, setting IntervalTm to the current time point minus OriTm;
setting DisThumbTipToRingTip to be the distance from the current thumb fingertip to the ring finger fingertip;
if IntervalTm is greater than ReleaseTm, or distumbtiptorngtip is greater than ReleaseDis, the gettrestate state value is set to be recognition-unfitted, oriTm is set to be the current time point, ringTipOri is set to be the current position of the ring finger tip, thumbTipOri is set to be the current position of the thumb tip, and the frame recognition process is ended;
if IntervalTm is less than GesturEMaxTm, and IntervalTm is greater than GesturEMINTm, disthenbTipToRingTip is greater than GestureDis, gestureState status is set to: and if the identification is successful, endTm is set as the current time point, and the frame identification process is ended.
8. The method of claim 7, wherein if IntvalTm is greater than GestureTm indicating a recognition timeout, the GestureState state value is set to: unrecognized, oriTm is set as the current time point, thumbTipOri is set as the position of the thumb tip at that time, and RingTipOri is set as the position of the ring finger tip at that time.
9. The AR/MR technique-based gesture recognition method according to claim 8,
if ifTrack is false, the thumb tip and the ring finger tip are not recognized before the frame, the positions of the bony prominences of the thumb tip and the ring finger tip are obtained and recorded in ThumbTipOri and RingTipOri respectively, and the current time point is recorded in OriTm.
10. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor when executing the program carries out the steps of the method for gesture recognition based on AR/MR technology according to any of claims 1 to 9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310091517.5A CN115798054B (en) | 2023-02-10 | 2023-02-10 | Gesture recognition method based on AR/MR technology and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310091517.5A CN115798054B (en) | 2023-02-10 | 2023-02-10 | Gesture recognition method based on AR/MR technology and electronic equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115798054A true CN115798054A (en) | 2023-03-14 |
| CN115798054B CN115798054B (en) | 2023-11-10 |
Family
ID=85430701
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310091517.5A Active CN115798054B (en) | 2023-02-10 | 2023-02-10 | Gesture recognition method based on AR/MR technology and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115798054B (en) |
Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106909871A (en) * | 2015-12-22 | 2017-06-30 | 江苏达科智能科技有限公司 | Gesture instruction recognition methods |
| CN106970701A (en) * | 2016-01-14 | 2017-07-21 | 芋头科技（杭州）有限公司 | A kind of gesture changes recognition methods |
| WO2018076523A1 (en) * | 2016-10-25 | 2018-05-03 | 科世达（上海）管理有限公司 | Gesture recognition method and apparatus, and in-vehicle system |
| US20190179417A1 (en) * | 2017-12-11 | 2019-06-13 | Shenzhen Starfield Information Technologies Co., Ltd. | 3D Interaction Method, Device, Computer Equipment and Storage Medium |
| WO2019178114A1 (en) * | 2018-03-13 | 2019-09-19 | Magic Leap, Inc. | Gesture recognition system and method of using same |
| CN111273778A (en) * | 2020-02-14 | 2020-06-12 | 北京百度网讯科技有限公司 | Method and device for controlling electronic equipment based on gestures |
| US20200225758A1 (en) * | 2019-01-11 | 2020-07-16 | Microsoft Technology Licensing, Llc | Augmented two-stage hand gesture input |
| CN112926423A (en) * | 2021-02-07 | 2021-06-08 | 青岛小鸟看看科技有限公司 | Kneading gesture detection and recognition method, device and system |
| CN113238650A (en) * | 2021-04-15 | 2021-08-10 | 青岛小鸟看看科技有限公司 | Gesture recognition and control method and device and virtual reality equipment |
| CN114973414A (en) * | 2022-06-01 | 2022-08-30 | 斑马网络技术有限公司 | Dynamic gesture recognition method and intelligent vehicle-mounted equipment |
| CN115601781A (en) * | 2021-06-25 | 2023-01-13 | 青岛海尔多媒体有限公司（Cn） | Method and device for dynamic gesture recognition and electronic equipment |
-
2023
- 2023-02-10 CN CN202310091517.5A patent/CN115798054B/en active Active
Patent Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106909871A (en) * | 2015-12-22 | 2017-06-30 | 江苏达科智能科技有限公司 | Gesture instruction recognition methods |
| CN106970701A (en) * | 2016-01-14 | 2017-07-21 | 芋头科技（杭州）有限公司 | A kind of gesture changes recognition methods |
| WO2018076523A1 (en) * | 2016-10-25 | 2018-05-03 | 科世达（上海）管理有限公司 | Gesture recognition method and apparatus, and in-vehicle system |
| US20190179417A1 (en) * | 2017-12-11 | 2019-06-13 | Shenzhen Starfield Information Technologies Co., Ltd. | 3D Interaction Method, Device, Computer Equipment and Storage Medium |
| WO2019178114A1 (en) * | 2018-03-13 | 2019-09-19 | Magic Leap, Inc. | Gesture recognition system and method of using same |
| US20200225758A1 (en) * | 2019-01-11 | 2020-07-16 | Microsoft Technology Licensing, Llc | Augmented two-stage hand gesture input |
| CN111273778A (en) * | 2020-02-14 | 2020-06-12 | 北京百度网讯科技有限公司 | Method and device for controlling electronic equipment based on gestures |
| CN112926423A (en) * | 2021-02-07 | 2021-06-08 | 青岛小鸟看看科技有限公司 | Kneading gesture detection and recognition method, device and system |
| CN113238650A (en) * | 2021-04-15 | 2021-08-10 | 青岛小鸟看看科技有限公司 | Gesture recognition and control method and device and virtual reality equipment |
| CN115601781A (en) * | 2021-06-25 | 2023-01-13 | 青岛海尔多媒体有限公司（Cn） | Method and device for dynamic gesture recognition and electronic equipment |
| CN114973414A (en) * | 2022-06-01 | 2022-08-30 | 斑马网络技术有限公司 | Dynamic gesture recognition method and intelligent vehicle-mounted equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115798054B (en) | 2023-11-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20180024643A1 (en) | Gesture Based Interface System and Method | |
| CN104216642B (en) | A kind of terminal control method | |
| US8199115B2 (en) | System and method for inputing user commands to a processor | |
| KR101956325B1 (en) | System for finger recognition and tracking | |
| Rautaray et al. | Real time multiple hand gesture recognition system for human computer interaction | |
| CN104216516B (en) | A kind of terminal | |
| AU2012268589A1 (en) | System for finger recognition and tracking | |
| EP2605844A2 (en) | Method circuit and system for human to machine interfacing by hand gestures | |
| CN105589553A (en) | Gesture control method and system for intelligent equipment | |
| CN114138121B (en) | User gesture recognition method, device and system, storage medium and computing equipment | |
| CN111783600B (en) | Face recognition model training method, device, equipment and medium | |
| US20130229348A1 (en) | Driving method of virtual mouse | |
| CN106601217A (en) | Interactive-type musical instrument performing method and device | |
| CN109725722B (en) | Gesture control method and device for screen equipment | |
| KR101525011B1 (en) | tangible virtual reality display control device based on NUI, and method thereof | |
| CN115798054A (en) | Gesture recognition method based on AR/MR technology and electronic device | |
| Lakshmi et al. | A novel air writing recognition system using Raspberry Pi for the control and interaction of digital systems | |
| CN114756162B (en) | Touch system and method, electronic device and computer readable storage medium | |
| CN103809846A (en) | Function calling method and electronic equipment | |
| Feng et al. | FM: Flexible mapping from one gesture to multiple semantics | |
| CN104679400B (en) | A kind of method and terminal of contactless input information | |
| Dohare et al. | Virtual-mouse using artificial intelligence | |
| Rautaray et al. | Manipulating objects through hand gesture recognition in virtual environment | |
| Prabhakar et al. | AI And Hand Gesture Recognition Based Virtual Mouse | |
| CN114415929B (en) | Control method and device of electronic equipment, electronic equipment and readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |