CN115798054A - A gesture recognition method and electronic equipment based on AR/MR technology - Google Patents

Abstract

The invention provides a gesture recognition method based on AR/MR technology and electronic equipment, in the technical field of spatial data analysis and recognition, setting and storing related variables in the gesture recognition process; setting constant information as the basis for gesture recognition judgment, and performing gesture recognition Engine, to identify and judge each frame: judge whether the SDK has tracked the tip of the thumb and the fingertip of the ring finger. If any fingertip is not tracked, the recognition process of this frame ends; if both the fingertip of the thumb and the ring finger are recognized point, then record the position of the thumbtip in the current frame, that is, the three-dimensional coordinates in space; the present invention can summarize the relevant variables, constant information and gesture recognition engine of gesture recognition, which is convenient for users to refer to and use, and is convenient for the system to use according to relevant variables, constant information and The gesture recognition engine executes the corresponding gesture recognition process. Effectively improve the accuracy and precision of gesture recognition.

Description

A gesture recognition method and electronic equipment based on AR/MR technology

technical field

The invention relates to the technical field of spatial data analysis and recognition, in particular to a gesture recognition method and electronic equipment based on AR/MR technology.

Background technique

At present, the application prospect of AR/MR technology is very broad. Among them, AR (augmented reality) is a technology that calculates the position and angle of camera images in real time and adds corresponding images, videos, and 3D models. MR is a mixed reality technology. Generally speaking, AR is to superimpose virtual images on the actual scene, and MR is to interact with virtual images after superimposing virtual images on the actual scene. The most prominent feature of AR/MR is that it does not use keyboard, mouse, handle or touch screen and other hands-occupied operation methods, but uses gestures, voice or eye movement to perform non-contact operations, freeing users' hands and making it more convenient and fast. Get and manipulate data.

Compared with traditional interactive platforms, the most prominent feature of AR/MR is that it can use non-contact operations such as gestures, voice or eye movement to liberate users without using keyboards, mice, handles or touch screens. With both hands, it is more convenient and quick to obtain and operate data. At present, through the feature of freeing hands, in the industry, for example, when overhauling equipment, users can consult information, seek remote assistance, or place drawings next to the equipment for real-time comparison while overhauling the hand-held tool, without having to put down the tool at hand. from the manual or tablet. In life, for example, when learning cooking skills, you don’t have to worry about swiping the screen with your hands with seasonings or flour to dirty your phone and reversely contaminate your meals when you look up recipes.

For another example, in production and life, it may be inconvenient to touch the screen, take paper materials or operate equipment due to space restrictions, hand-held tools, and stains. This requires gesture recognition to call the resource list.

The current gesture recognition mainly adopts the method of artificial intelligence image training. This method faces three problems. First, the training period is long, and the image data set needs to be trained to match the training model that meets the requirements. If the training model does not match Or the accuracy of the model is low, which affects the accuracy of gesture recognition. Second, the amount of data models is large, which consumes a lot of system resources, and also causes the system to run stuck, which cannot meet the recognition requirements. Moreover, the lack of three-digit data has a certain impact on the accuracy only through the plane image judgment.

Contents of the invention

The present invention provides a gesture recognition method based on AR/MR technology, which comprehensively analyzes the accuracy of gestures through 2D and 3D data of skeleton points.

Gesture recognition methods based on AR/MR technology include:

S101, setting and storing relevant variables in the gesture recognition process;

S102. Set constant information as a basis for gesture recognition judgment, where the constant information is a value defined according to the AR/MR device and the use environment;

S103. Execute the gesture recognition engine, and perform the following recognition judgment on each frame:

Determine whether the SDK has tracked the tip of the thumb and the fingertip of the ring finger. If any fingertip is not tracked, set IfTrack to false, and set the value of GestureState to: no recognition, the recognition process of this frame ends;

If the fingertip of the thumb and the fingertip of the ring finger are recognized at the same time, then use ThumbTip to record the position of the thumb tip of the current frame, that is, the three-dimensional coordinates in space, and use RingTip to record the position of the fingertip of the ring finger in the current frame, that is, the three-dimensional coordinates in space;

If the status value of GestureState has been recognized successfully, use IntervalTm to record the current frame time point-EndTm, which is the time from the last successful judgment to the current frame;

If IntervalTm is greater than ReAcvTm, the state value of GestureState is changed to not recognized; OriTm is set to the time point of the current frame; RingTipOri is set to the current position of the fingertip of the ring finger; ThumbTipOri is set to the current position of the fingertip of the thumb, and the recognition process of this frame Straight to the end.

It should be further noted that the relevant variables set and stored in step S101 include:

Set the three-dimensional floating-point variable RingTip, and record and track the position information of the ring finger fingertip in real time;

Set the three-dimensional floating-point number variable RingTipOri, and record the three-dimensional space position coordinates of the fingertip of the ring finger at a certain point in time, for comparing the position change of the fingertip of the ring finger;

Set the 3D floating point variable ThumbTip, record and track the position of the thumb tip in real time, and update the 3D space coordinates of the thumb acquired by the SDK in the current frame;

Set the three-dimensional floating-point variable ThumbTipOri to record the three-dimensional space position coordinates of the thumb tip at a certain time point, which is used to compare the position change of the thumb tip.

It should be further noted that the relevant variables set and stored in step S101 also include:

Set the enumeration variable GestureState to identify the state value of the enumeration variable. The state value includes: 1. Not recognized, 2. Recognized in progress, 3. Recognized successfully, 4. Disabled;

The default state of the enumeration variable GestureState is: not recognized;

Set the Boolean variable IfTrack, and record whether it is the first time that the thumb tip and the ring finger tip are recognized at the same time;

Set the floating-point variable OriTm, and record the time point when the gesture recognition starts, and the time point is the number of seconds elapsed from the time the program starts running to the recording time;

Set the floating-point variable EndTm, and record the time point when the gesture recognition is successful, the time point is the number of seconds elapsed from the time the program starts running to the time of recording;

Set the floating-point variable IntervalTm;

Set the floating point variable DisThumbTipToRingTip, and record the distance information from the tip of the thumb to the tip of the ring finger.

It should be further noted that step S102 also includes: setting a floating-point constant StartDis as a distance condition for judging whether to start gesture recognition; when the distance between RingTip and ThumbTip is smaller than StartDis, start judging whether the gesture condition is met;

Set the floating-point constant EndDis as the distance that the gesture needs to reach. When the distance between ThumbTip and RingTip exceeds EndDis, it means that the gesture position has reached the completion standard;

Set the floating-point constant GustureMinTm as the limit condition of the lower limit of the gesture judgment time, that is, when the gesture recognition is started, such as when the position of the gesture changes, the elapsed recognition time is greater than GustureMinTm, then it is determined to be a valid gesture recognition state;

Set the limit condition GestureMaxTm of the upper limit of the gesture judgment time of the floating-point constant; when the gesture recognition starts, the gesture position change condition is satisfied within the GestureMaxTm time.

It should be further noted that step S102 also includes: setting a floating-point constant GestureReleaseTm as the time to stop gesture recognition; that is, after starting gesture recognition, if there is no distance change between ThumbTip and RingTip within the GestureReleaseTm time, then no further Gesture Recognition;

Set the floating-point constant GestureReleaseDis to judge the distance to stop gesture recognition, that is, during gesture recognition, if within the GestureReleaseTm time, judge whether the distance between ThumbTip and RingTip exceeds the preset distance, and if it does not exceed the preset distance, stop the current gesture recognition ;

Set the floating point type constant ReAcvTm, and the floating point type constant ReAcvTm is the cooling time of the gesture.

It should be further noted that the end conditions of the frame recognition process in step S103 also include:

If the state of GestureState is not identified, then judge whether the ThumbTip is consistent with the ThumbTIpOri, or whether the RingTip is consistent with the RingTipOri, if they are consistent, the identification process of this frame ends;

If the distance between the tip of the thumb and the tip of the ring finger is greater than StartDis, the recognition process of this frame ends;

After the end, set the state value of GestureState to the recognition state.

It should be further explained that after setting the state value of GestureState to the recognition state, set IntervalTm to the current time point minus OriTm;

Set DisThumbTipToRingTip to the distance from the current thumb tip to the ring finger tip;

If IntervalTm is greater than ReleaseTm, or DisThumbTipToRingTip is greater than ReleaseDis, the GestureState state value is set to not start recognition, OriTm is set to the current time point, RingTipOri is set to the current position of the fingertip of the ring finger, ThumbTipOri is set to the current position of the fingertip of the thumb, and this frame is recognized end of process;

If IntervalTm is less than GestureMaxTm, and IntervalTm is greater than GestureMinTm, and DisThumbTipToRingTip is greater than GestureDis, the GestureState state is set to: recognized successfully, EndTm is set to the current time point, and the frame recognition process ends.

It should be further explained that if IntvalTm is greater than GestureTm, it means that the recognition has timed out, then set the GestureState state value to: Unrecognized, OriTm is set to the current time point, ThumbTipOri is set to the position of the thumb tip at this time, RingTipOri is set to the ring finger at this time pointed position.

It should be further explained that if IfTrack is false and the thumbtip and ring fingertip have not been recognized before this frame, then obtain the joint positions of the thumbtip and ring fingertip, record them in ThumbTipOri and RingTipOri respectively, and put the current The time points were recorded in OriTm.

The present invention also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and operable on the processor. When the processor executes the program, the AR/MR-based Steps of the technology gesture recognition method.

As can be seen from the above technical solutions, the present invention has the following advantages:

The present invention can summarize relevant variables, constant information and gesture recognition engines of gesture recognition, which is convenient for users to refer to and use, and facilitates the system to execute corresponding gesture recognition processes according to relevant variables, constant information and gesture recognition engines. Effectively improve the accuracy and precision of gesture recognition. It can also efficiently collect, store, and process gesture recognition information. Based on the state of gesture recognition and the state of the thumb and ring fingertips, process monitoring can be realized, and the entire gesture recognition process can be described using multi-dimensional space. Improve the quality and efficiency of gesture recognition, discover abnormalities in the gesture recognition process in time, and make corrections to improve the execution accuracy of AR/MR technology and meet the requirements of use. It also realizes the supervision of the whole process of gesture recognition based on AR/MR technology , Timeliness and scientificity of management and control.

The present invention provides a gesture recognition method based on AR/MR technology. It also sets the three-dimensional floating-point number variable RingTip to record and track the fingertip position of the ring finger in real time, and the three-dimensional floating-point number variable RingTipOri records the three-dimensional space position of the ring finger fingertip at a certain point in time. Coordinates, the 3D floating-point variable ThumbTip records the tracked thumbtip position in real time, the 3D floating-point variable ThumbTipOri records the coordinates of the 3D space position of the thumbtip at a certain point in time, the enumerated variable GestureState recognizes the state, and the floating-point variable OriTm Record the start time of gesture recognition, the floating point variable EndTm records the time point when the gesture recognition is successful, the floating point variable IntervalTm records the time elapsed in each state, and the floating point variable DisThumbTipToRingTip records the thumb tip to the ring finger tip It can judge whether the SDK has tracked the tip of the thumb and the fingertip of the ring finger. It can also judge and recognize the gesture state in combination with the scene, and integrate the recognition and verification of the tip of the thumb and the fingertip of the ring finger.

By setting the 3D data variable, it can be recognized and judged in every frame in the engine, and the 3D data is used to comprehensively analyze gestures to improve the accuracy of recognition of unfolding gestures, and to overcome the artificial intelligence image training that can only be judged by plane images , the lack of three-dimensional data has a certain impact on accuracy, and the long training period and large amount of data models are not conducive to quickly adjusting the judgment conditions and judgment accuracy of gestures.

The invention can dynamically register and cancel the identification of skeleton points. The accuracy of gestures is comprehensively analyzed through 2D and 3D data of skeleton points. It can quickly adjust the judgment conditions and judgment accuracy of gestures, confirm the position of fingertips, make the gesture recognition function more stable, meet the needs of users, and ensure the reliability of gesture recognition in AR/MR technology.

Description of drawings

In order to illustrate the technical solution of the present invention more clearly, the accompanying drawings that need to be used in the description will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. As far as people are concerned, other drawings can also be obtained based on these drawings on the premise of not paying creative work.

Fig. 1 is a flowchart of a gesture recognition method based on AR/MR technology.

Detailed ways

The gesture recognition method based on AR/MR technology provided by the present invention can acquire and process associated data based on artificial intelligence technology. Among them, the gesture recognition method based on AR/MR technology uses a digital computer or a machine controlled by a digital computer to simulate, extend and expand human intelligence, the theory, method, technology and application device of perceiving the environment, acquiring knowledge and using knowledge to obtain the best results .

The gesture recognition method based on AR/MR technology includes both hardware-level technology and software-level technology. The basic technologies of gesture recognition methods generally include technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation/interaction systems, and mechatronics. The software technology of the gesture recognition method mainly includes computer perspective technology, machine learning/deep learning and programming language. Wherein, programming languages include but are not limited to object-oriented programming languages—such as Java, Smalltalk, C++, and conventional procedural programming languages—such as “C” language or similar programming languages. The program code can be completely executed on the AR/MR device, can be executed as an independent software package, or partially executed on the user computer and partially executed on the remote computer, or completely executed on the remote computer. In cases involving a remote computer, the remote computer can be connected to the user computer via any kind of network, including a local area network (LAN) or a wide area network (WAN), or, alternatively, can be connected to an external computer (using, for example, an Internet service Provider via Internet connection).

For the present invention, the gesture recognition engine, or 3D engine, or the engine is the 3D engine used to develop AR/MR software, such as Unity, UE, etc.

The time point is the time point mentioned in the present invention, which refers to the time point in the 3D engine, and specifically refers to the number of seconds elapsed from the start of the program as the time starting point to each acquisition of the time point.

SDK: The development tools provided by device manufacturers have basic device function calls. For example, Hololens manufacturer Microsoft's SDK is MRTK, which can track the required spatial position information of hand bone points in real time.

Frame: The unit of calculation frequency in the engine. Every update of the picture in the engine is a frame, and within the time of each frame, it is a calculation cycle in the engine.

The gesture recognition method involved in the present invention utilizes AR/MR technology, by setting relevant variables and constant information of gesture recognition, and executing the gesture recognition engine, performing the following recognition and judgment on each frame, so as to realize non-contact operation based on gesture actions, Make it easier for users to obtain and operate data. It can be effectively applied to multiple fields such as remote control equipment, or access to information. It further effectively solves the problem that the traditional gesture recognition is inaccurate and the gesture information cannot be effectively obtained, which leads to the ineffective operation of AR/MR technology and affects the user experience.

The gesture recognition method can be applied to one or more electronic devices. The electronic device is a device that can automatically perform numerical calculation and/or information processing according to preset or stored instructions. Its hardware includes but is not limited to micro Processor, application specific integrated circuit (Application Specific Integrated Circuit, ASIC), programmable gate array (Field-Programmable GateArray, FPGA), digital processor (Digital Signal Processor, DSP), embedded equipment, etc.

An electronic device can be any electronic product that can interact with a user, such as a personal computer, a tablet computer, a smart phone, a personal digital assistant (Personal Digital Assistant, PDA), an interactive network TV (Internet Protocol Television, IPTV), etc. ), AR/MR smart wearable devices, etc.

The network where the electronic device is located includes but is not limited to the Internet, a wide area network, a metropolitan area network, a local area network, a virtual private network (Virtual Private Network, VPN) and the like.

The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some, not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

Please refer to FIG. 1 which is a flowchart of a gesture recognition method based on AR/MR technology in a specific embodiment, S101, setting and storing relevant variables in the gesture recognition process;

Specifically, 1.1 set the three-dimensional floating-point variable RingTip to record the tracked fingertip position of the ring finger in real time, that is, update the three-dimensional space position coordinates of the ring finger acquired by the SDK in the current frame.

Set the three-dimensional floating point variable RingTipOri to record the three-dimensional position coordinates of the fingertip of the ring finger at a certain time point, which is used to compare the position changes of the fingertip of the ring finger at a specific stage.

1.2 Set the 3D floating-point variable ThumbTip to record the tracked thumbtip position in real time, that is, to update the 3D space position coordinates of the thumb acquired by the SDK for the current frame.

Set the three-dimensional floating-point variable ThumbTipOri to record the three-dimensional position coordinates of the thumbtip at a certain time point, which is used to compare the position changes of the thumbtip at a specific stage.

1.3 Set the enumerated variable GestureState recognition state, and its possible state values are four kinds: 1. Not recognized, 2. Recognized in progress, 3. Recognized successfully, 4. Disabled. The default state of GestureState is: not recognized.

1.4 Set the Boolean variable IfTrack to record whether the fingertip of the thumb and the fingertip of the ring finger are recognized at the same time for the first time. The initial value is false.

1.5 Set the floating-point variable OriTm to record the start time of gesture recognition, which is the number of seconds elapsed from the start of the program to the time of recording.

1.6 Set the floating-point variable EndTm to record the time point when the gesture recognition is successful. This time point is the number of seconds elapsed from the start of the program to the recording time.

1.7 Set the floating-point variable IntervalTm to record the elapsed time of each state, such as recording the time elapsed from the last successful gesture recognition to recording after successful recognition, or recording the elapsed time from the start of recognition to recording.

1.8 Set the floating-point variable DisThumbTipToRingTip to record the distance from the tip of the thumb to the tip of the ring finger.

S102. Set constant information as a basis for gesture recognition judgment, where the constant information is a value defined according to the AR/MR device and the use environment;

2.1 Set the floating-point constant StartDis as the distance condition for judging whether to start gesture recognition, that is, when the distance between RingTip and ThumbTip is smaller than StartDis, judge whether the gesture condition is met, and the value of StartDis set here is 0.035.

2.2 Set the floating-point constant EndDis as the distance that the gesture needs to reach, that is, when the distance between ThumbTip and RingTip exceeds EndDis, it means that the gesture position has reached the completion standard, and the value of EndDis is set to 0.5.

2.3 Set the floating-point constant GustureMinTm as the limit condition of the lower limit of the gesture judgment time, that is, after the gesture recognition starts, the recognition time elapsed when the position change condition of the gesture is satisfied is greater than GustureMinTm, and its value is set to 0.13.

2.4 Set the limit condition GestureMaxTm of the upper limit of the floating-point constant gesture judgment time, that is, when the gesture recognition starts, all the conditions for the gesture position change are satisfied within the GestureMaxTm time, and its value is set to 0.5.

2.5 Set the floating-point constant GestureReleaseTm as the time to stop gesture recognition, that is, after starting gesture recognition, if the distance between ThumbTip and RingTip does not change enough within the time of GestureReleaseTm, it is not considered that the user is ready to perform this gesture, so it will not be performed again Gesture Recognition.

2.6 Set the floating-point constant GestureReleaseDis to judge the distance to stop gesture recognition, that is, the criterion for judging whether there is a sufficient distance change between ThumbTip and RingTip within GestureReleaseTm after gesture recognition starts, and set its value to 0.038.

2.7 Set the cooling time of the floating-point number constant ReAcvTm gesture, that is, after the gesture is recognized, how many seconds will not be recognized for the gesture. There are two functions:

1). Under normal circumstances, the gesture does not need to be triggered continuously in a short period of time. If there is a continuous meeting of the judgment criteria, it is likely to be caused by misuse.

2). After the action is activated, the joints used as the recognition standard have a high probability of still meeting the standard of reaching the gesture, which will cause the gesture to be successfully recognized by high frequency and continue to cause the program to crash. Set its value to 1.5.

S103. Execute the gesture recognition engine to perform recognition and judgment on each frame, and the specific judgment steps are as follows:

3.1 Determine whether the SDK has tracked the tip of the thumb and the fingertip of the ring finger. If any fingertip is not tracked, set IfTrack to false, and set the value of GestureState to: no recognition. The frame recognition process ends directly.

3.2 If the thumbtip and the ring fingertip are recognized at the same time, use ThumbTip to record the position of the thumb tip in the current frame, that is, the three-dimensional space coordinates, and use RingTip to record the position of the ring fingertip in the current frame, that is, the three-dimensional space coordinates.

3.3 If IfTrack is false, it means that the thumbtip and the ring fingertip have not been recognized at the same time before this frame after the start of the program or after the last gesture recognition is successful, then obtain the joint positions of the thumbtip and the ring fingertip, respectively Documented in ThumbTipOri and RingTipOri. Record the current time point in OriTm. Change the value of IfTrack to true.

3.4 If the state value of GestureState has been recognized successfully, use IntervalTm to record the current frame time point-EndTm, that is, the time from the last successful judgment to the current frame.

If IntervalTm is greater than ReAcvTm, change the state value of GestureState to not recognized. Set OriTm to the time point of the current frame. RingTipOri is set to the current position of the fingertip of the ring finger. ThumbTipOri is set to the current position of the thumb tip. The frame recognition process ends directly.

3.4 If the status of GestureState is not identified, then:

3.4.1 If the ThumbTip is consistent with the ThumbTIpOri, or the RingTip is consistent with the RingTipOri, the frame identification process ends directly.

3.4.2 If the distance between the tip of the thumb and the tip of the ring finger is greater than StartDis, the recognition process of this frame ends directly.

3.4.3 Set the status value of GestureState to Recognition in progress.

3.5 If the status value of GestureState is identifying, then

3.5.1 If the RingTip is consistent with the RingTipOri or the ThumbTip is consistent with the ThumbTipOri, the frame identification process ends directly.

3.5.2 Set IntervalTm to the current time point minus OriTm.

3.5.3 Set DisThumbTipToRingTip to the distance from the current thumb tip to the ring finger tip.

3.5.4 If IntervalTm is greater than ReleaseTm or DisThumbTipToRingTip is greater than ReleaseDis, the GestureState state value is set to not start recognition, OriTm is set to the current time point, RingTipOri is set to the current position of the fingertip of the ring finger, ThumbTipOri is set to the current position of the fingertip of the thumb, this frame The recognition process ends directly.

3.5.5 If IntervalTm is less than GestureMaxTm, and IntervalTm is greater than GestureMinTm, and DisThumbTipToRingTip is greater than GestureDis, the GestureState state is set to: recognized successfully, and EndTm is set to the current time point. The blooming gesture is recognized, and the recognition process of this frame ends directly.

3.6 If IntvalTm is greater than GestureTm, it means that the recognition has timed out, then set the GestureState state value to: recognition has not started, OriTm is set to the current time point, ThumbTipOri is set to the position of the thumb tip at this time, and RingTipOri is set to the position of the ring finger tip at this time.

It should be understood that the sequence numbers of the steps in the above embodiments do not mean the order of execution, and the execution order of each process should be determined by its functions and internal logic, and should not constitute any limitation to the implementation process of the embodiment of the present invention.

In this way, the present invention can summarize related variables, constant information and gesture recognition engines of gesture recognition, which is convenient for users to refer to and use, and facilitates the system to execute corresponding gesture recognition processes according to relevant variables, constant information and gesture recognition engines. Effectively improve the accuracy and precision of gesture recognition. It can also efficiently collect, store, and process gesture recognition information. Based on the state of gesture recognition and the state of the thumb and ring fingertips, process monitoring can be realized, and the entire gesture recognition process can be described using multi-dimensional space. Improve the quality and efficiency of gesture recognition, discover abnormalities in the gesture recognition process in time, and make corrections to improve the execution accuracy of AR/MR technology and meet the requirements of use. It also realizes the supervision of the whole process of gesture recognition based on AR/MR technology , Timeliness and scientificity of management and control.

The present invention provides a gesture recognition method based on AR/MR technology. It also sets the three-dimensional floating-point number variable RingTip to record and track the fingertip position of the ring finger in real time, and the three-dimensional floating-point number variable RingTipOri records the three-dimensional space position of the ring finger fingertip at a certain point in time. Coordinates, the 3D floating point variable ThumbTip records the tracked thumbtip position in real time, the 3D floating point variable ThumbTipOri records the coordinates of the 3D space position of the thumbtip at a certain point in time, the enumerated variable GestureState recognizes the state, and the floating point variable OriTm Record the time point when the gesture recognition starts, the floating point variable EndTm records the time point when the gesture recognition is successful, the floating point variable IntervalTm records the time elapsed in each state, and the floating point variable DisThumbTipToRingTip records the thumb tip to the ring finger tip It can judge whether the SDK has tracked the tip of the thumb and the fingertip of the ring finger. It can also judge and recognize the gesture state in combination with the scene, and integrate the identification and verification of the fingertip of the thumb and the ring finger. The invention can dynamically register and cancel the identification of skeleton points. The accuracy of gestures is comprehensively analyzed through 2D and 3D data of skeleton points. It can quickly adjust the judgment conditions and judgment accuracy of gestures, confirm the position of fingertips, make the gesture recognition function more stable, meet the needs of users, and ensure the reliability of gesture recognition in AR/MR technology.

The units and algorithm steps of each example described in the embodiments disclosed in the gesture recognition method based on AR/MR technology involved in the present invention can be realized by electronic hardware, computer software or a combination of the two. In order to clearly illustrate the hardware and Interchangeability of software. In the above description, the components and steps of each example have been generally described according to their functions. Whether these functions are executed by hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be regarded as exceeding the scope of the present invention.

In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods may be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or May be integrated into another system, or some features may be ignored, or not implemented. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices or units, and may also be electrical, mechanical or other forms of connection.

The above description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention will not be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims (10)

1. A gesture recognition method based on AR/MR technology is characterized by comprising the following steps:

s101, setting and storing related variables in a gesture recognition process;

s102, setting constant information used as a gesture recognition judgment basis, wherein the constant information is a value defined according to AR/MR equipment and a use environment;

s103, executing a gesture recognition engine, and performing the following gesture recognition judgment on each frame:

judging whether the SDK tracks the thumb tip and the ring finger tip, if any one of the finger tips is not tracked, setting IfTrack as false, and setting GesturState as the following value: if the frame is not identified, the frame identification process is finished;

if the thumb fingertip and the ring finger fingertip are identified at the same time, recording the position of the current frame thumb fingertip, namely the space three-dimensional coordinate by using thumb Tip, and recording the position of the current frame ring finger fingertip, namely the space three-dimensional coordinate by using RingTip;

if the state value of GesturEST is successfully identified, recording the time point-EndTm of the current frame by using IntervalTm, namely the time from the last successful judgment to the current frame;

if IntervalTm is greater than ReAcvTm, the state value of GesturState is changed to unrecognized; setting OriTm as a time point of a current frame; ringTipOri is set to the position of the current ring finger tip; the thumb TipOri is set as the current position of the thumb tip, and the frame identification process is directly ended.

2. The method for recognizing gestures based on AR/MR technology according to claim 1, wherein the relevant variables set and stored in step S101 include:

setting a three-dimensional floating point number variable RingTip, and recording and tracking the position information of the ring finger tip in real time;

setting a three-dimensional floating point number variable RingTipOri, recording a three-dimensional space position coordinate of the fingertip of the ring finger at a certain time point, and comparing the position change of the fingertip of the ring finger;

setting a three-dimensional floating point number variable ThumbTip, recording and tracking the position of a fingertip of the thumb in real time, and updating the three-dimensional space position coordinate of the thumb obtained by the current frame SDK;

and setting a three-dimensional floating point number variable ThumbTipOri, and recording the three-dimensional space position coordinates of the thumb and the fingertip at a certain time point for comparing the position change of the thumb and the fingertip.

3. The method for recognizing gestures based on AR/MR technology according to claim 1, wherein the related variables set and stored in step S101 further comprise:

setting an enumeration type variable GestureState, and identifying the state value of the enumeration type variable, wherein the state value comprises the following steps: 1. not identifying, 2, identifying, 3, successfully identifying, and 4, forbidding;

the default state of the enumerated variable GestureState is: no identification is performed;

setting a Boolean type variable IfTrack, and recording whether the fingertips of the thumb and the ring finger are recognized for the first time;

setting a floating point type variable oriTm and recording a time point of starting gesture recognition, wherein the time point is the number of seconds from the time when a program starts to run to the time when the program is recorded;

setting a floating point type variable EndTM, and recording a time point when gesture recognition is successful, wherein the time point is the number of seconds from the start of running of a program to the recording;

setting a floating point type variable IntervalTm;

the floating point type variable quantity DisThumbTipToRingTip is set, and the distance information from the thumb tip to the ring finger tip is recorded.

4. The method for gesture recognition based on AR/MR technology according to claim 1, wherein step S102 further comprises: setting a floating point type constant StartDis as a distance condition for judging whether to start gesture recognition; when the distance between the RingTip and the thumb is smaller than StartDis, starting to judge whether the gesture condition is met;

setting a floating point constant EndDis as a distance which needs to be reached by the gesture, and when the distance between thumb tip and Ringtip exceeds EndDis, indicating that the gesture position reaches a completion standard;

setting a floating point constant GusturmeminTm as a limiting condition of a lower limit of gesture judgment time, namely determining the gesture recognition state as an effective gesture recognition state if the elapsed recognition time is more than GusturmeminTm after gesture recognition is started and if the gesture position change condition is met;

setting a limitation condition GesturmemaxTm of a floating point type constant gesture judgment time upper limit; after gesture recognition is started, the gesture position change condition is satisfied within the time of GesturmeMaxTm.

5. The method for gesture recognition based on AR/MR technology according to claim 1, wherein step S102 further comprises: setting a floating point constant GesturreleaseTm as time for stopping gesture recognition; after gesture recognition is started, if the distance between the thumb tip and the Ringtip is not changed within the GesturreleaseTm time, the gesture recognition is not performed any more;

setting a floating point constant GestureleaseDis to judge the distance for stopping gesture recognition, namely judging whether the distance change of thumb tip and Ringtip exceeds a preset distance or not during gesture recognition within GestureleaseTm time, and stopping current gesture recognition if the distance change of thumb tip and Ringtip does not exceed the preset distance;

and setting a constant value of the floating point number, namely, the Retvtm, as the cooling time of the gesture.

6. The method for recognizing gesture based on AR/MR technology according to claim 1, wherein the ending condition of the frame recognition process in step S103 further includes:

if the GestureState is not recognized, judging whether the ThumbTip is consistent with the ThumbTIpOri or whether the RingTip is consistent with the RingTipOri, and if so, finishing the frame recognition process;

if the distance between the finger tip of the thumb and the finger tip of the ring finger is greater than StartDis, the frame identification process is ended;

after the end, the state value of getturestate is set to the recognition execution state.

7. The AR/MR technology-based gesture recognition method according to claim 6, wherein after setting the state value of getcurestate to recognition state, setting IntervalTm to the current time point minus OriTm;

setting DisThumbTipToRingTip to be the distance from the current thumb fingertip to the ring finger fingertip;

if IntervalTm is greater than ReleaseTm, or distumbtiptorngtip is greater than ReleaseDis, the gettrestate state value is set to be recognition-unfitted, oriTm is set to be the current time point, ringTipOri is set to be the current position of the ring finger tip, thumbTipOri is set to be the current position of the thumb tip, and the frame recognition process is ended;

if IntervalTm is less than GesturEMaxTm, and IntervalTm is greater than GesturEMINTm, disthenbTipToRingTip is greater than GestureDis, gestureState status is set to: and if the identification is successful, endTm is set as the current time point, and the frame identification process is ended.

8. The method of claim 7, wherein if IntvalTm is greater than GestureTm indicating a recognition timeout, the GestureState state value is set to: unrecognized, oriTm is set as the current time point, thumbTipOri is set as the position of the thumb tip at that time, and RingTipOri is set as the position of the ring finger tip at that time.

9. The AR/MR technique-based gesture recognition method according to claim 8,

if ifTrack is false, the thumb tip and the ring finger tip are not recognized before the frame, the positions of the bony prominences of the thumb tip and the ring finger tip are obtained and recorded in ThumbTipOri and RingTipOri respectively, and the current time point is recorded in OriTm.

10. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor when executing the program carries out the steps of the method for gesture recognition based on AR/MR technology according to any of claims 1 to 9.

Images