CN115798054A - Gesture recognition method based on AR/MR technology and electronic device - Google Patents

Gesture recognition method based on AR/MR technology and electronic device Download PDF

Info

Publication number
CN115798054A
CN115798054A CN202310091517.5A CN202310091517A CN115798054A CN 115798054 A CN115798054 A CN 115798054A CN 202310091517 A CN202310091517 A CN 202310091517A CN 115798054 A CN115798054 A CN 115798054A
Authority
CN
China
Prior art keywords
gesture recognition
setting
thumb
gesture
time
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202310091517.5A
Other languages
Chinese (zh)
Other versions
CN115798054B (en
Inventor
张瑞程
邢振华
褚烨
徐淼
王书源
王天翔
孙英洲
王帅
代佳佳
韩平
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
TaiAn Power Supply Co of State Grid Shandong Electric Power Co Ltd
Original Assignee
TaiAn Power Supply Co of State Grid Shandong Electric Power Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by TaiAn Power Supply Co of State Grid Shandong Electric Power Co Ltd filed Critical TaiAn Power Supply Co of State Grid Shandong Electric Power Co Ltd
Priority to CN202310091517.5A priority Critical patent/CN115798054B/en
Publication of CN115798054A publication Critical patent/CN115798054A/en
Application granted granted Critical
Publication of CN115798054B publication Critical patent/CN115798054B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Landscapes

  • User Interface Of Digital Computer (AREA)

Abstract

The invention provides a gesture recognition method and electronic equipment based on AR/MR technology, which belongs to the technical field of spatial data analysis and recognition and is used for setting and storing relevant variables in the gesture recognition process; setting constant information as a gesture recognition judgment basis, executing a gesture recognition engine, and performing recognition judgment on each frame: judging whether the SDK tracks the thumb fingertips and the ring finger fingertips, and if any one of the fingertips is not tracked, ending the frame identification process; if the finger tips of the thumb and the finger tips of the ring finger are identified at the same time, recording the position of the finger tips of the thumb of the current frame, namely the spatial three-dimensional coordinate; the gesture recognition method and the gesture recognition system can collect the related variable and constant information of gesture recognition and the gesture recognition engine, are convenient for a user to refer to and use, and are convenient for the system to execute a corresponding gesture recognition process according to the related variable and constant information and the gesture recognition engine. The accuracy and precision of gesture recognition are effectively improved.

Description

Gesture recognition method based on AR/MR technology and electronic device
Technical Field
The invention relates to the technical field of spatial data analysis and recognition, in particular to a gesture recognition method based on an AR/MR technology and electronic equipment.
Background
At present, the application prospect of the AR/MR technology is very wide, wherein AR (augmented reality) is a technology for calculating the position and angle of the camera image in real time and adding corresponding image, video and 3D models, and MR is a mixed reality technology. Generally speaking, AR superimposes a virtual image on an actual scene, and MR superimposes a virtual image on an actual scene, and then can interact with the virtual image. The most prominent characteristic of AR/MR is that the hands of the user can be liberated through non-contact operation of gesture, voice or eye movement fixation without using an operation mode of occupying the hands such as a keyboard, a mouse, a handle or a touch screen, and data can be acquired and operated more conveniently and rapidly.
Compared with the traditional interactive platform, the most prominent characteristic of AR/MR is that the hands of a user can be liberated through non-contact operation of gesture, voice or eye movement gazing without using an operation mode of occupying the hands such as a keyboard, a mouse, a handle or a touch screen, and data can be acquired and operated more conveniently and rapidly. At present, through the characteristic of liberation both hands, when for example overhauing equipment in the industry, the user can look up the data, seek long-range assistance or put the drawing and carry out real-time comparison beside the equipment when handheld tool overhauls, needn't put down the instrument of hand and pick up the manual or the flat board repeatedly and look up. When the user learns the cooking art in life, the user does not need to worry about that the mobile phone is polluted by a hand sliding screen stuck with seasonings or flour and the meal is reversely polluted when the user looks up the menu.
For another example, in production and life, it may be inconvenient to touch a screen, take paper materials or operate equipment due to problems such as site limitation, handheld tools, stain influence and the like, and thus a gesture recognition mode is required to call up a resource list.
The current gesture recognition mainly adopts an artificial intelligence image training mode, and the mode faces three problems that the training period is long, an image data set needs to be trained, a training model meeting requirements is matched, and the accuracy of gesture recognition is affected if the training model is not matched or the model accuracy is low. Secondly, the data model quantity is large, a large amount of system resources are consumed, the system operation is blocked, and the identification requirement cannot be met. And the lack of three-bit data alignment accuracy has a certain effect only by plane image judgment.
Disclosure of Invention
The invention provides a gesture recognition method based on an AR/MR (augmented reality/magnetic resonance) technology, which comprehensively analyzes the accuracy of gestures through 2D (two-dimensional) and 3D (three-dimensional) data of skeleton points.
The gesture recognition method based on the AR/MR technology comprises the following steps:
s101, setting and storing related variables in a gesture recognition process;
s102, setting constant information used as a gesture recognition judgment basis, wherein the constant information is a value defined according to AR/MR equipment and a use environment;
s103, executing a gesture recognition engine, and performing the following recognition judgment on each frame:
judging whether the SDK tracks the thumb tip and the ring finger tip, if any one of the finger tips is not tracked, setting IfTrack as false, and setting GesturState as the following value: the frame identification process is finished when the identification is not carried out;
if the thumb fingertip and the ring finger fingertip are identified at the same time, recording the position of the current frame thumb fingertip, namely the space three-dimensional coordinate by using thumb Tip, and recording the position of the current frame ring finger fingertip, namely the space three-dimensional coordinate by using RingTip;
if the state value of GesturEST is successfully identified, recording the time point-EndTm of the current frame by using IntervalTm, namely the time from the last successful judgment to the current frame;
if IntervalTm is greater than ReAcvTm, the state value of GesturState is changed to be unrecognized; setting OriTm as a time point of a current frame; ringTipOri is set to the position of the current ring finger tip; the ThumbTipOri is set as the position of the current thumb tip, and the frame identification process is directly finished.
It should be further noted that the relevant variables set and stored in step S101 include:
setting a three-dimensional floating point number variable RingTip, and recording and tracking the position information of the ring finger tip in real time;
setting a three-dimensional floating point number variable RingTipOri, recording a three-dimensional space position coordinate of the ring finger tip at a certain time point, and comparing the position change of the ring finger tip;
setting a three-dimensional floating point number variable thumb tip, recording and tracking the fingertip position of the thumb in real time, and updating the three-dimensional space position coordinate of the thumb acquired by the current frame SDK;
and setting a three-dimensional floating point number variable ThumbTipOri, and recording the three-dimensional space position coordinates of the thumb and the fingertip at a certain time point for comparing the position change of the thumb and the fingertip.
It should be further noted that the relevant variables set and stored in step S101 further include:
setting an enumeration type variable GestureState, and identifying the state value of the enumeration type variable, wherein the state value comprises the following steps: 1. not identifying, 2, identifying, 3, successfully identifying, and 4, forbidding;
the default state of the enumerated variable GestureState is: no identification is performed;
setting a Boolean type variable IfTrack, and recording whether the first time is to simultaneously recognize the finger tips of the thumb and the ring finger;
setting a floating point type variable oriTm and recording a time point of starting gesture recognition, wherein the time point is the number of seconds from the time when a program starts to run to the time when the program is recorded;
setting a floating point variable EndTM, and recording a time point when gesture recognition is successful, wherein the time point is the number of seconds from the time when a program starts to run to the time when the gesture recognition is recorded;
setting a floating point type variable IntervalTm;
the floating point type variable quantity DisThumbTipToRingTip is set, and the distance information from the thumb tip to the ring finger tip is recorded.
Further, step S102 further includes: setting a floating point type constant StartDis as a distance condition for judging whether to start gesture recognition; when the distance between the RingTip and the thumb is smaller than StartDis, starting to judge whether the gesture condition is met;
setting a floating point constant EndDis as a distance which needs to be reached by the gesture, and when the distance between the thumb tip and the Ringtip exceeds the EndDis, indicating that the gesture position reaches a completion standard;
setting a floating point constant GusturmeminTm as a limiting condition of a lower limit of gesture judgment time, namely determining the gesture recognition state as an effective gesture recognition state if the elapsed recognition time is more than GusturmeminTm after gesture recognition is started and if the gesture position change condition is met;
setting a limitation condition GesturEMaxTm of the upper limit of the judgment time of the floating point type constant gesture; after gesture recognition is started, the gesture position change condition is met within the GesturmemaxTm time.
Further, step S102 further includes: setting a floating point constant GesturreleaseTm as time for stopping gesture recognition; after gesture recognition is started, if the distance between the thumb tip and the Ringtip is not changed within the GesturreleaseTm time, the gesture recognition is not performed any more;
setting a floating point constant GesturreleaseDis to judge the distance for stopping gesture recognition, namely judging whether the distance change of thumb and RingTip exceeds a preset distance or not during gesture recognition, such as in GesturreleaseTm time, and stopping current gesture recognition if the distance change of thumb and RingTip does not exceed the preset distance;
and setting a constant value of the floating point number, namely, the Retvtm, as the cooling time of the gesture.
It should be further noted that, in step S103, the condition for ending the frame identification process further includes:
if the GestureState is not recognized, judging whether the ThumbTip is consistent with the ThumbTIpOri or whether the RingTip is consistent with the RingTipOri, and if so, finishing the frame recognition process;
if the distance between the thumb tip and the ring finger tip is larger than StartDis, the frame identification process is finished;
after the end, the state value of getturestate is set to the recognition execution state.
It should be further noted that after setting the state value of getturestate to the recognition state, setting IntervalTm to the current time minus OriTm;
setting DisThumbTipToRingTip to be the distance from the current thumb fingertip to the ring finger fingertip;
if IntervalTm is greater than ReleaseTm or DisThumbTipToRingTip is greater than ReleaseDis, the GestureState state value is set as recognition-not-started, oriTm is set as the current time point, ringTipOri is set as the current position of the ring finger tip, thumbTipOri is set as the current position of the thumb tip, and the frame recognition process is ended;
if IntervalTm is less than GesturEMaxTm, and IntervalTm is greater than GesturEMinTm, dishumbTipToRingTip is greater than GestureDis, then GestureState status is set to: and if the identification is successful, endTm is set as the current time point, and the frame identification process is ended.
It should be further noted that if invaltm is greater than geturetm, which indicates that the recognition is timed out, the geturestate value is set to: not recognized, oriTm is set as the current time point, thumbTipOri is set as the position of the thumb tip at this time, and RingTipOri is set as the position of the ring finger tip at this time.
It should be further noted that if IfTrack is false, and the thumb tip and the ring finger tip have not been recognized before the frame, the positions of the joints of the thumb tip and the ring finger tip are obtained and recorded in ThumbTipOri and RingTipOri, respectively, and the current time point is recorded in OriTm.
The invention also provides an electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the steps of the AR/MR technology based gesture recognition method when executing the program.
According to the technical scheme, the invention has the following advantages:
the gesture recognition system and the gesture recognition method can summarize related variables, constant information and a gesture recognition engine of gesture recognition, are convenient for a user to use for reference, and are convenient for the system to execute a corresponding gesture recognition process according to the related variables, the constant information and the gesture recognition engine. The accuracy and precision of gesture recognition are effectively improved. The gesture recognition information can be efficiently collected, stored and processed, process monitoring can be achieved based on the gesture recognition state and the states of the thumb finger tip and the ring finger tip, and the whole gesture recognition process is described by using a multi-dimensional space. The method and the device improve the quality and efficiency of gesture recognition, timely find the abnormity in the gesture recognition process, and correct the abnormity so as to improve the execution accuracy of the AR/MR technology, meet the use requirements, and realize the timeliness and scientificity of the gesture recognition overall process supervision, management and control based on the AR/MR technology.
The invention provides a gesture recognition method based on AR/MR technology, and the method can also record the tracked position of a ring finger tip in real time by setting a three-dimensional floating point variable RingTip, recording the three-dimensional space position coordinate of the ring finger tip at a certain time point by the three-dimensional floating point variable RingTiori, recording the tracked position of the thumb tip in real time by the three-dimensional floating point variable ThumbTip, recording the three-dimensional space position coordinate of the thumb tip at a certain time point by the three-dimensional floating point variable ThumbTipOri, recording the recognition state of an enumerated variable GesturteState, recording the time point of the start of gesture recognition by the floating point variable OriTm, recording the time point when the gesture recognition succeeds by the floating point variable EndTm, recording the time of each state by the floating point variable IntervalTm, recording a plurality of parameter information such as the distance from the thumb tip to the ring finger tip and the like, and can judge and recognize the fused gesture scene of the thumb tip and the ring finger tip by combining the judgment.
The three-dimensional data variable is set, so that each frame of the three-dimensional data variable in an engine can be recognized and judged, the gesture is comprehensively analyzed by utilizing the three-dimensional data, the accuracy of recognizing the unfolding gesture is improved, and the problems that the artificial intelligent image training is adopted, judgment can be only carried out through a plane image, three-dimensional data is lacked, certain influence is caused on the accuracy, the training period is long, the data model quantity is large, and the judgment condition and the judgment precision of the gesture are not favorably and rapidly adjusted are solved.
The invention can dynamically register and cancel the identification of the skeletal points. And comprehensively analyzing the accuracy of the gesture through the 2D and 3D data of the skeleton points. The gesture recognition method has the advantages that the gesture judgment condition and the gesture judgment precision can be quickly adjusted, the positions of fingertips are confirmed, the gesture recognition function is more stable, the use requirements of users are met, and the reliability of gesture recognition in the AR/MR technology is guaranteed.
Drawings
In order to more clearly illustrate the technical solution of the present invention, the drawings required to be used in the description will be briefly introduced below, and it is apparent that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art that other drawings can be obtained based on these drawings without creative efforts.
FIG. 1 is a flow chart of a gesture recognition method based on AR/MR technology.
Detailed Description
The gesture recognition method based on the AR/MR technology can acquire and process the associated data based on the artificial intelligence technology. The gesture recognition method based on the AR/MR technology simulates, extends and expands human intelligence by using a digital computer or a machine controlled by the digital computer, senses the environment, acquires knowledge and obtains the theory, method, technology and application device of the best result by using the knowledge.
The gesture recognition method based on the AR/MR technology has the hardware level technology and the software level technology. The basic technologies of the gesture recognition method generally include technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation/interaction systems, electromechanical integration and the like. The software technology of the gesture recognition method mainly comprises a computer visual angle technology, machine learning/deep learning and a programming language. The programming languages include, but are not limited to, object oriented programming languages such as Java, smalltalk, C + +, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the AR/MR device, partly on the user computer, partly on a remote computer or entirely on the remote computer, as a stand-alone software package. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
For purposes of the present invention, the gesture recognition engine, or 3D engine, is the 3D engine used to develop AR/MR software, such as Unity, UE, etc.
The time point is a time point referred to in the present invention, and refers to a time point in the 3D engine, specifically to a number of seconds elapsed from when the program starts to run as a time starting point to each time the time point is acquired.
And (3) SDK: the development tool provided by the equipment manufacturer has basic equipment function call, for example, the microsoft SDK of the Hololens manufacturer is MRTK, and can track the spatial position information of the required hand skeleton point in real time.
Frame: the unit of calculating frequency in the engine is one frame every time the picture in the engine is updated, and within the time of each frame, the unit of calculating frequency in the engine is one calculation period in the engine.
The gesture recognition method utilizes AR/MR technology, sets relevant variables and relevant constant information of gesture recognition, executes a gesture recognition engine, and carries out recognition judgment on each frame as follows, so that non-contact operation based on gesture actions is realized, and a user can acquire and operate data more conveniently and rapidly. The method can be effectively applied to remote control equipment or multiple fields of data lookup and the like. The problems that the traditional gesture recognition is inaccurate, gesture information cannot be effectively acquired, the AR/MR technology cannot be effectively operated, and user experience is influenced are further effectively solved.
The gesture recognition method may be applied to one or more electronic devices, which are devices capable of automatically performing numerical calculation and/or information processing according to preset or stored instructions, and the hardware thereof includes, but is not limited to, a microprocessor, an Application Specific Integrated Circuit (ASIC), a Programmable gate array (FPGA), a Digital Signal Processor (DSP), an embedded device, and the like.
The electronic device may be any electronic product capable of performing human-computer interaction with a user, for example, a Personal computer, a tablet computer, a smart phone, a Personal Digital Assistant (PDA), an interactive web Television (IPTV), an AR/MR smart wearable device, and the like.
The Network where the electronic device is located includes, but is not limited to, the internet, a wide area Network, a metropolitan area Network, a local area Network, a Virtual Private Network (VPN), and the like.
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Referring to fig. 1, which is a flowchart illustrating a gesture recognition method based on AR/MR technology according to an embodiment, S101 sets and stores related variables in a gesture recognition process;
specifically, 1.1, setting a three-dimensional floating point number variable RingTip to record the tracked fingertip position of the ring finger in real time, namely updating the three-dimensional space position coordinates of the ring finger acquired by the current frame SDK.
And setting a three-dimensional floating point number variable RingTipOri to record the three-dimensional space position coordinates of the ring finger tip at a certain time point, and comparing the position change of the ring finger tip at a specific stage.
1.2, setting a three-dimensional floating point number variable thumb tip to record the tracked thumb fingertip position in real time, namely updating the three-dimensional space position coordinate of the thumb acquired by the current frame SDK.
And setting a three-dimensional floating point number variable ThumbTipOri to record the three-dimensional space position coordinates of the thumb and the fingertip at a certain time point, and comparing the position change of the thumb and the fingertip at a specific stage.
1.3 setting enumeration type variable GestureState recognition state, the possible state values are four kinds respectively: 1. no identification is made, 2 identification is in progress, 3 identification is successful, and 4 is disabled. The default state of GestureState is: no identification is made.
1.4 set Boolean type variable IfTrack record whether the thumb and ring finger tips are recognized for the first time simultaneously. The initial value is false.
1.5 set the floating point type variable OriTm records the time point of the start of gesture recognition, which is the number of seconds elapsed from when the program started running to when it was recorded.
1.6 sets the time point when the floating point type variable EndTM records the gesture recognition successfully, which is the number of seconds elapsed from the start of the program to the recording.
1.7 a floating point type variable IntervalTm is set to record the time elapsed for each state, for example, the time elapsed from the last successful gesture recognition to the recording after the successful recognition is recorded, or the time elapsed from the beginning of the recognition to the recording is recorded.
1.8 set the floating point variable DisthumbTipToRingTip, record the distance from the thumb tip to the ring finger tip.
S102, setting constant information used as a gesture recognition judgment basis, wherein the constant information is a value defined according to AR/MR equipment and a use environment;
2.1 a floating point constant StartDis is set as a distance condition for judging whether to start gesture recognition, that is, when the distance between RingTip and ThumbTip is smaller than the StartDis, whether the gesture condition is met is judged, and the value of StartDis set here is 0.035.
2.2 setting a floating point constant EndDis as a distance which needs to be reached by the gesture, namely when the distance between the thumb and RingTip exceeds EndDis, indicating that the gesture position reaches a completion standard, and setting the value of EndDis to be 0.5.
2.3 The floating point constant GustureMinTm is set as a limiting condition of a lower limit of gesture judgment time, namely, after gesture recognition is started, the recognition time which passes when the gesture position change condition is met is greater than GustureMinTm, and the value is set to be 0.13.
2.4, the limitation condition geturemaxtm of the upper limit of the judgment time of the floating point type constant gesture is set, that is, after gesture recognition is started, all conditions of gesture position change are met within the duration of geturemaxtm, and the value of the limitation condition geturemaxtm is set to be 0.5.
2.5 The floating point constant GesturreleaseTm is set as the time for stopping gesture recognition, namely after gesture recognition is started, if the thumb tip and the Ringtip do not have enough distance change within the GesturreleaseTm time, the user is not considered to be ready to perform the gesture, and gesture recognition is not performed any more.
2.6 And setting a floating point constant GesturreleaseDis to judge the distance for stopping gesture recognition, namely after gesture recognition is started, judging whether the thumb tip and the Ringtip have enough distance change within GesturreleaseTm according to a judgment standard, and setting the value to be 0.038.
2.7 The cooling time of the floating point number type constant RecavTm gesture is set, namely after the gesture is recognized to be completed, the gesture is not recognized within several seconds, and the functions are two:
1) Because the gesture will not be continuously triggered in a short time under normal conditions, if the situation that the judgment standard is continuously met occurs, the probability is that misoperation causes most.
2) The approximate probability of the joints as recognition criteria after the activation of the action still meets the criteria for achieving the gesture, which results in the gesture being successfully recognized with high frequency continuously and leads to program breakdown, and the value is set to be 1.5.
S103, executing a gesture recognition engine, and performing recognition judgment on each frame, wherein the specific judgment steps are as follows:
3.1 judging whether the SDK tracks the finger tips of the thumb and the ring finger, if any finger tip is not tracked, setting IfTrack to false, and setting GesturState to be: no identification is made. The frame identification process is directly ended.
3.2 If the thumb tip and the ring finger tip are identified at the same time, the position of the current frame thumb tip, namely the space three-dimensional coordinate, is recorded by using the thumb tip, and the position of the current frame ring finger tip, namely the space three-dimensional coordinate, is recorded by using the Ringtip.
3.3 if ifTrack is false, after the program starts running or the last gesture recognition is successful, the positions of the joints of the thumb tip and the ring finger tip are obtained and recorded in ThumbTipOri and RingTipOri respectively if the positions of the joints of the thumb tip and the ring finger tip are not recognized simultaneously before the frame. The current time point is recorded in OriTm. The value of ifTrack is changed to true.
3.4 if the state value of GesturStatate is recognized successfully, recording the time point-EndTm of the current frame by using IntervalTm, namely the time from the last successful judgment to the current frame.
If IntervalTm is greater than ReAcvTm, the state value of GesturState is set to unrecognized. OriTm is set as the time point of the current frame. RingTipOri is set to the position of the current ring finger tip. The ThumbTipOri is set to the current thumb tip position. The frame identification process ends directly.
3.4 If GesturESTate is not recognized, then:
3.4.1 if ThumbTip is consistent with ThumbTIpOri, or RingTip is consistent with RingTipOri, the present frame identification process ends directly.
3.4.2 If the distance between the tip of the thumb and the tip of the ring finger is greater than StartDis, the frame identification process is directly ended.
3.4.3 The state value of geturestate is set to recognition in progress.
3.5 If the state value of GestureState is recognizing in progress, then
3.5.1 if RingTip is consistent with RingTipOri or ThumbTip is consistent with ThumbTipOri, the present frame identification process ends directly.
3.5.2 IntervalTm is set to the current time point minus OriTm.
3.5.3 Distumbtiptorringtip is set to the current thumb-fingertip distance to the ring finger-fingertip distance.
3.5.4 If IntervalTm is greater than ReleaseTm or distumbtiptorngtip is greater than ReleaseDis, the getturestate value is set to be not started to recognize, oriTm is set to be the current time point, ringTipOri is set to be the position of the current ring finger tip, thumbTipOri is set to be the position of the current thumb tip, and the frame recognition process is directly ended.
3.5.5 if IntervalTm is less than GesturEMaxTm, and IntervalTm is greater than GesturEMINTm, and DisthenbTipToRingTip is greater than GesturDis, then GestureState status is set to: having identified success, endTm is set to the current point in time. The gesture of blooming is recognized, and the frame recognition process is directly finished.
3.6 if IntvalTm is greater than GesturreTm, indicating a recognition timeout, gesturreState status value is set to: recognition is not started, oriTm is set as the current time point, thumbTipOri is set as the position of the thumb tip at this time, and RingTipOri is set as the position of the ring finger tip at this time.
It should be understood that, the sequence numbers of the steps in the foregoing embodiments do not imply an execution sequence, and the execution sequence of each process should be determined by its function and inherent logic, and should not constitute any limitation to the implementation process of the embodiments of the present invention.
Therefore, the gesture recognition method and the gesture recognition system can collect the relevant variable, the constant information and the gesture recognition engine of gesture recognition, are convenient for a user to refer to and use, and are convenient for the system to execute a corresponding gesture recognition process according to the relevant variable, the constant information and the gesture recognition engine. The accuracy and precision of gesture recognition are effectively improved. The gesture recognition information can be efficiently collected, stored and processed, process monitoring can be achieved based on the gesture recognition state and the states of the thumb fingertips and the ring finger fingertips, and the whole gesture recognition process is described by using a multidimensional space. The method and the device improve the quality and efficiency of gesture recognition, timely find the abnormity in the gesture recognition process, and correct the abnormity so as to improve the execution accuracy of the AR/MR technology, meet the use requirements, and realize the timeliness and scientificity of the gesture recognition overall process supervision, management and control based on the AR/MR technology.
The invention provides a gesture recognition method based on AR/MR technology, and the method can also record the tracked position of a ring finger tip in real time by setting a three-dimensional floating point variable RingTip, recording the three-dimensional space position coordinate of the ring finger tip at a certain time point by the three-dimensional floating point variable RingTiori, recording the tracked position of the thumb tip in real time by the three-dimensional floating point variable ThumbTip, recording the three-dimensional space position coordinate of the thumb tip at a certain time point by the three-dimensional floating point variable ThumbTipOri, recording the recognition state of an enumerated variable GesturteState, recording the time point of the start of gesture recognition by the floating point variable OriTm, recording the time point when the gesture recognition succeeds by the floating point variable EndTm, recording the time of each state by the floating point variable IntervalTm, recording a plurality of parameter information such as the distance from the thumb tip to the ring finger tip and the like, and can judge and recognize the fused gesture scene of the thumb tip and the ring finger tip by combining the judgment. The invention can dynamically register and cancel the identification of the skeleton points. And comprehensively analyzing the accuracy of the gesture through the 2D and 3D data of the skeleton points. The gesture recognition method has the advantages that the gesture judgment condition and the gesture judgment precision can be rapidly adjusted, the positions of fingertips are confirmed, the gesture recognition function is more stable, the use requirements of users are met, and the reliability of gesture recognition in the AR/MR technology is guaranteed.
The elements and algorithm steps of the examples described in the embodiments of the AR/MR technology based gesture recognition method according to the present invention can be implemented in electronic hardware, computer software, or a combination of both, and in the above description the components and steps of the examples have been generally described in terms of functionality in order to clearly illustrate the interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
In the several embodiments provided in the present application, it should be understood that the disclosed system, apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the units is only one logical division, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may also be an electric, mechanical or other form of connection.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims (10)

1. A gesture recognition method based on AR/MR technology is characterized by comprising the following steps:
s101, setting and storing related variables in a gesture recognition process;
s102, setting constant information used as a gesture recognition judgment basis, wherein the constant information is a value defined according to AR/MR equipment and a use environment;
s103, executing a gesture recognition engine, and performing the following gesture recognition judgment on each frame:
judging whether the SDK tracks the thumb tip and the ring finger tip, if any one of the finger tips is not tracked, setting IfTrack as false, and setting GesturState as the following value: if the frame is not identified, the frame identification process is finished;
if the thumb fingertip and the ring finger fingertip are identified at the same time, recording the position of the current frame thumb fingertip, namely the space three-dimensional coordinate by using thumb Tip, and recording the position of the current frame ring finger fingertip, namely the space three-dimensional coordinate by using RingTip;
if the state value of GesturEST is successfully identified, recording the time point-EndTm of the current frame by using IntervalTm, namely the time from the last successful judgment to the current frame;
if IntervalTm is greater than ReAcvTm, the state value of GesturState is changed to unrecognized; setting OriTm as a time point of a current frame; ringTipOri is set to the position of the current ring finger tip; the thumb TipOri is set as the current position of the thumb tip, and the frame identification process is directly ended.
2. The method for recognizing gestures based on AR/MR technology according to claim 1, wherein the relevant variables set and stored in step S101 include:
setting a three-dimensional floating point number variable RingTip, and recording and tracking the position information of the ring finger tip in real time;
setting a three-dimensional floating point number variable RingTipOri, recording a three-dimensional space position coordinate of the fingertip of the ring finger at a certain time point, and comparing the position change of the fingertip of the ring finger;
setting a three-dimensional floating point number variable ThumbTip, recording and tracking the position of a fingertip of the thumb in real time, and updating the three-dimensional space position coordinate of the thumb obtained by the current frame SDK;
and setting a three-dimensional floating point number variable ThumbTipOri, and recording the three-dimensional space position coordinates of the thumb and the fingertip at a certain time point for comparing the position change of the thumb and the fingertip.
3. The method for recognizing gestures based on AR/MR technology according to claim 1, wherein the related variables set and stored in step S101 further comprise:
setting an enumeration type variable GestureState, and identifying the state value of the enumeration type variable, wherein the state value comprises the following steps: 1. not identifying, 2, identifying, 3, successfully identifying, and 4, forbidding;
the default state of the enumerated variable GestureState is: no identification is performed;
setting a Boolean type variable IfTrack, and recording whether the fingertips of the thumb and the ring finger are recognized for the first time;
setting a floating point type variable oriTm and recording a time point of starting gesture recognition, wherein the time point is the number of seconds from the time when a program starts to run to the time when the program is recorded;
setting a floating point type variable EndTM, and recording a time point when gesture recognition is successful, wherein the time point is the number of seconds from the start of running of a program to the recording;
setting a floating point type variable IntervalTm;
the floating point type variable quantity DisThumbTipToRingTip is set, and the distance information from the thumb tip to the ring finger tip is recorded.
4. The method for gesture recognition based on AR/MR technology according to claim 1, wherein step S102 further comprises: setting a floating point type constant StartDis as a distance condition for judging whether to start gesture recognition; when the distance between the RingTip and the thumb is smaller than StartDis, starting to judge whether the gesture condition is met;
setting a floating point constant EndDis as a distance which needs to be reached by the gesture, and when the distance between thumb tip and Ringtip exceeds EndDis, indicating that the gesture position reaches a completion standard;
setting a floating point constant GusturmeminTm as a limiting condition of a lower limit of gesture judgment time, namely determining the gesture recognition state as an effective gesture recognition state if the elapsed recognition time is more than GusturmeminTm after gesture recognition is started and if the gesture position change condition is met;
setting a limitation condition GesturmemaxTm of a floating point type constant gesture judgment time upper limit; after gesture recognition is started, the gesture position change condition is satisfied within the time of GesturmeMaxTm.
5. The method for gesture recognition based on AR/MR technology according to claim 1, wherein step S102 further comprises: setting a floating point constant GesturreleaseTm as time for stopping gesture recognition; after gesture recognition is started, if the distance between the thumb tip and the Ringtip is not changed within the GesturreleaseTm time, the gesture recognition is not performed any more;
setting a floating point constant GestureleaseDis to judge the distance for stopping gesture recognition, namely judging whether the distance change of thumb tip and Ringtip exceeds a preset distance or not during gesture recognition within GestureleaseTm time, and stopping current gesture recognition if the distance change of thumb tip and Ringtip does not exceed the preset distance;
and setting a constant value of the floating point number, namely, the Retvtm, as the cooling time of the gesture.
6. The method for recognizing gesture based on AR/MR technology according to claim 1, wherein the ending condition of the frame recognition process in step S103 further includes:
if the GestureState is not recognized, judging whether the ThumbTip is consistent with the ThumbTIpOri or whether the RingTip is consistent with the RingTipOri, and if so, finishing the frame recognition process;
if the distance between the finger tip of the thumb and the finger tip of the ring finger is greater than StartDis, the frame identification process is ended;
after the end, the state value of getturestate is set to the recognition execution state.
7. The AR/MR technology-based gesture recognition method according to claim 6, wherein after setting the state value of getcurestate to recognition state, setting IntervalTm to the current time point minus OriTm;
setting DisThumbTipToRingTip to be the distance from the current thumb fingertip to the ring finger fingertip;
if IntervalTm is greater than ReleaseTm, or distumbtiptorngtip is greater than ReleaseDis, the gettrestate state value is set to be recognition-unfitted, oriTm is set to be the current time point, ringTipOri is set to be the current position of the ring finger tip, thumbTipOri is set to be the current position of the thumb tip, and the frame recognition process is ended;
if IntervalTm is less than GesturEMaxTm, and IntervalTm is greater than GesturEMINTm, disthenbTipToRingTip is greater than GestureDis, gestureState status is set to: and if the identification is successful, endTm is set as the current time point, and the frame identification process is ended.
8. The method of claim 7, wherein if IntvalTm is greater than GestureTm indicating a recognition timeout, the GestureState state value is set to: unrecognized, oriTm is set as the current time point, thumbTipOri is set as the position of the thumb tip at that time, and RingTipOri is set as the position of the ring finger tip at that time.
9. The AR/MR technique-based gesture recognition method according to claim 8,
if ifTrack is false, the thumb tip and the ring finger tip are not recognized before the frame, the positions of the bony prominences of the thumb tip and the ring finger tip are obtained and recorded in ThumbTipOri and RingTipOri respectively, and the current time point is recorded in OriTm.
10. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor when executing the program carries out the steps of the method for gesture recognition based on AR/MR technology according to any of claims 1 to 9.
CN202310091517.5A 2023-02-10 2023-02-10 Gesture recognition method based on AR/MR technology and electronic equipment Active CN115798054B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202310091517.5A CN115798054B (en) 2023-02-10 2023-02-10 Gesture recognition method based on AR/MR technology and electronic equipment

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202310091517.5A CN115798054B (en) 2023-02-10 2023-02-10 Gesture recognition method based on AR/MR technology and electronic equipment

Publications (2)

Publication Number Publication Date
CN115798054A true CN115798054A (en) 2023-03-14
CN115798054B CN115798054B (en) 2023-11-10

Family

ID=85430701

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202310091517.5A Active CN115798054B (en) 2023-02-10 2023-02-10 Gesture recognition method based on AR/MR technology and electronic equipment

Country Status (1)

Country Link
CN (1) CN115798054B (en)

Citations (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106909871A (en) * 2015-12-22 2017-06-30 江苏达科智能科技有限公司 Gesture instruction recognition methods
CN106970701A (en) * 2016-01-14 2017-07-21 芋头科技(杭州)有限公司 A kind of gesture changes recognition methods
WO2018076523A1 (en) * 2016-10-25 2018-05-03 科世达(上海)管理有限公司 Gesture recognition method and apparatus, and in-vehicle system
US20190179417A1 (en) * 2017-12-11 2019-06-13 Shenzhen Starfield Information Technologies Co., Ltd. 3D Interaction Method, Device, Computer Equipment and Storage Medium
WO2019178114A1 (en) * 2018-03-13 2019-09-19 Magic Leap, Inc. Gesture recognition system and method of using same
CN111273778A (en) * 2020-02-14 2020-06-12 北京百度网讯科技有限公司 Method and device for controlling electronic equipment based on gestures
US20200225758A1 (en) * 2019-01-11 2020-07-16 Microsoft Technology Licensing, Llc Augmented two-stage hand gesture input
CN112926423A (en) * 2021-02-07 2021-06-08 青岛小鸟看看科技有限公司 Kneading gesture detection and recognition method, device and system
CN113238650A (en) * 2021-04-15 2021-08-10 青岛小鸟看看科技有限公司 Gesture recognition and control method and device and virtual reality equipment
CN114973414A (en) * 2022-06-01 2022-08-30 斑马网络技术有限公司 Dynamic gesture recognition method and intelligent vehicle-mounted equipment
CN115601781A (en) * 2021-06-25 2023-01-13 青岛海尔多媒体有限公司(Cn) Method and device for dynamic gesture recognition and electronic equipment

Patent Citations (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106909871A (en) * 2015-12-22 2017-06-30 江苏达科智能科技有限公司 Gesture instruction recognition methods
CN106970701A (en) * 2016-01-14 2017-07-21 芋头科技(杭州)有限公司 A kind of gesture changes recognition methods
WO2018076523A1 (en) * 2016-10-25 2018-05-03 科世达(上海)管理有限公司 Gesture recognition method and apparatus, and in-vehicle system
US20190179417A1 (en) * 2017-12-11 2019-06-13 Shenzhen Starfield Information Technologies Co., Ltd. 3D Interaction Method, Device, Computer Equipment and Storage Medium
WO2019178114A1 (en) * 2018-03-13 2019-09-19 Magic Leap, Inc. Gesture recognition system and method of using same
US20200225758A1 (en) * 2019-01-11 2020-07-16 Microsoft Technology Licensing, Llc Augmented two-stage hand gesture input
CN111273778A (en) * 2020-02-14 2020-06-12 北京百度网讯科技有限公司 Method and device for controlling electronic equipment based on gestures
CN112926423A (en) * 2021-02-07 2021-06-08 青岛小鸟看看科技有限公司 Kneading gesture detection and recognition method, device and system
CN113238650A (en) * 2021-04-15 2021-08-10 青岛小鸟看看科技有限公司 Gesture recognition and control method and device and virtual reality equipment
CN115601781A (en) * 2021-06-25 2023-01-13 青岛海尔多媒体有限公司(Cn) Method and device for dynamic gesture recognition and electronic equipment
CN114973414A (en) * 2022-06-01 2022-08-30 斑马网络技术有限公司 Dynamic gesture recognition method and intelligent vehicle-mounted equipment

Also Published As

Publication number Publication date
CN115798054B (en) 2023-11-10

Similar Documents

Publication Publication Date Title
US20180024643A1 (en) Gesture Based Interface System and Method
CN104216642B (en) A kind of terminal control method
US8199115B2 (en) System and method for inputing user commands to a processor
KR101956325B1 (en) System for finger recognition and tracking
Rautaray et al. Real time multiple hand gesture recognition system for human computer interaction
CN104216516B (en) A kind of terminal
AU2012268589A1 (en) System for finger recognition and tracking
EP2605844A2 (en) Method circuit and system for human to machine interfacing by hand gestures
CN105589553A (en) Gesture control method and system for intelligent equipment
CN114138121B (en) User gesture recognition method, device and system, storage medium and computing equipment
CN111783600B (en) Face recognition model training method, device, equipment and medium
US20130229348A1 (en) Driving method of virtual mouse
CN106601217A (en) Interactive-type musical instrument performing method and device
CN109725722B (en) Gesture control method and device for screen equipment
KR101525011B1 (en) tangible virtual reality display control device based on NUI, and method thereof
CN115798054A (en) Gesture recognition method based on AR/MR technology and electronic device
Lakshmi et al. A novel air writing recognition system using Raspberry Pi for the control and interaction of digital systems
CN114756162B (en) Touch system and method, electronic device and computer readable storage medium
CN103809846A (en) Function calling method and electronic equipment
Feng et al. FM: Flexible mapping from one gesture to multiple semantics
CN104679400B (en) A kind of method and terminal of contactless input information
Dohare et al. Virtual-mouse using artificial intelligence
Rautaray et al. Manipulating objects through hand gesture recognition in virtual environment
Prabhakar et al. AI And Hand Gesture Recognition Based Virtual Mouse
CN114415929B (en) Control method and device of electronic equipment, electronic equipment and readable storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant