WO2019105376A1

WO2019105376A1 - Gesture recognition method, terminal and storage medium

Info

Publication number: WO2019105376A1
Application number: PCT/CN2018/117864
Authority: WO
Inventors: 刘永霞; 史波; 易科; 李�瑞; 刘杉
Original assignee: 腾讯科技（深圳）有限公司
Priority date: 2017-11-30
Filing date: 2018-11-28
Publication date: 2019-06-06
Also published as: CN109857245B; CN109857245A

Abstract

The embodiments of the present application provide a gesture recognition method. The method is applied to a terminal, which terminal is provided with a speaker and a microphone and has an application program installed thereon. The method comprises: determining, according to an input control instruction of the application program, whether the terminal starts a gesture recognition mode; when it is determined that the terminal starts the gesture recognition mode, playing a first ultrasonic wave by means of the speaker; receiving a second ultrasonic wave by means of the microphone, the second ultrasonic wave being waveform data obtained upon acquisition of the first ultrasonic wave; performing gesture recognition according to the received second ultrasonic wave, to obtain a gesture recognition result; and executing a corresponding gesture instruction on the application program according to the gesture recognition result.

Description

Gesture recognition method, terminal and storage medium

The present application claims priority to Chinese Patent Application No. PCT Application No. No. No. No. No. No. No. No. No. No. No. No. No. No. No. No.

Technical field

The present application relates to the field of computer technologies, and in particular, to a gesture recognition method, a terminal, and a storage medium.

background

Users can use a mobile browser to access network resources, such as accessing web pages. The input mode of the user on the mobile phone browser is mainly contact input, that is, the user needs to manually input on the touch screen or physical button of the mobile phone terminal.

Users can also use non-contact input methods for input, such as gesture recognition, which is convenient for users. Among them, the gesture refers to various actions made by the human hand under the control of the human consciousness, such as finger bending, stretching and hand movement in space, etc., which may be performing a certain task or communicating with people to express a certain Meaning or intent. Based on three-dimensional interactive input technology of gesture recognition, there are commonly used data glove-based and vision-based (such as camera) gesture recognition.

Technical content

An example of the present application provides a gesture recognition method, where the method is applied to a terminal, where a speaker and a microphone are disposed, and an application is installed on the terminal, and the method includes:

Determining whether the terminal turns on the gesture recognition mode according to an input control indication of the application;

When the terminal determines to turn on the gesture recognition mode, playing the first ultrasonic wave through the speaker;

Receiving, by the microphone, a second ultrasonic wave, wherein the second ultrasonic wave is waveform data obtained after collecting the first ultrasonic wave;

Performing gesture recognition according to the received second ultrasonic wave to obtain a gesture recognition result;

Corresponding gesture instructions are executed in the application according to the gesture recognition result.

The application example further provides a terminal, where the terminal is configured with a speaker and a microphone, and the terminal is installed with an application program, and the terminal includes:

a mode determining module, configured to determine, according to an input control indication of the application, whether the terminal turns on a gesture recognition mode;

An ultrasonic transmitting module, configured to: when the terminal determines to enable the gesture recognition mode, play the first ultrasonic wave through the speaker;

An ultrasonic acquisition module, configured to receive a second ultrasonic wave by using the microphone, where the second ultrasonic wave is waveform data obtained after collecting the first ultrasonic wave;

a gesture recognition module, configured to perform gesture recognition according to the received second ultrasonic wave, to obtain a gesture recognition result;

And an instruction execution module, configured to execute a corresponding gesture instruction in the application according to the gesture recognition result.

The application example further provides a terminal, where the terminal includes: a speaker, a microphone, a processor, and a memory;

The processor and the speaker, the microphone, and the memory communicate with each other;

The memory is for storing instructions;

The speaker for playing a first ultrasonic wave under the control of the processor;

The microphone for receiving a second ultrasonic wave under the control of the processor;

The processor is operative to execute the instructions in the memory, performing the method of any of the preceding aspects.

The present application examples provide a computer readable storage medium having instructions stored therein that, when executed on a computer, cause the computer to perform the methods described in the various aspects above.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the technical solutions in the examples of the present application, the drawings used in the description of the examples will be briefly described below. Obviously, the drawings in the following description are only some examples of the present application, For the skilled person, other figures can also be obtained from these figures.

1a is a schematic structural diagram of a system involved in an example of the present application;

1b is a schematic block diagram of a gesture recognition method provided by an example of the present application;

1c is a schematic structural diagram of a system involved in an example of the present application;

2 is a schematic flow chart of ultrasonic transmission provided by an example of the present application;

3 is a schematic flow chart of ultrasonic receiving provided by an example of the present application;

4 is a schematic flowchart of a one-dimensional gesture recognition process provided by an example of the present application;

FIG. 5 is a schematic flowchart of a two-dimensional gesture recognition process provided by an example of the present application;

6 is a schematic flowchart of gesture recognition action management provided by an example of the present application;

7-a is a schematic diagram of the action of gesture recognition provided by the example of the present application;

7-b is a schematic diagram of the end of the action of gesture recognition provided by the example of the present application;

FIG. 8 is a schematic diagram of a basic operation of gesture recognition provided by an example of the present application; FIG.

9-a is a schematic structural diagram of a terminal provided by an example of the present application;

9-b is a schematic structural diagram of a mode determining module provided by an example of the present application;

9-c is a schematic structural diagram of another terminal provided by an example of the present application;

9-d is a schematic structural diagram of another terminal provided by an example of the present application;

FIG. 10 is a schematic structural diagram of a gesture recognition method provided by an example of the present application applied to a terminal;

FIG. 11 is a schematic structural diagram of another terminal according to an example of the present application.

Implementation

The application example provides a gesture recognition method and a terminal for reducing the complexity of gesture recognition on an application of the terminal and reducing the computing performance requirement of the terminal.

In order to make the objects, features, and advantages of the present application more obvious and easy to understand, the technical solutions in the examples of the present application will be clearly and completely described in conjunction with the drawings in the examples of the present application. It is only a part of the examples of this application, not all examples. All other examples obtained by those skilled in the art based on the examples in the present application are within the scope of the present application.

The terms "including" and "comprising", and any variations thereof, are intended to cover a non-exclusive inclusion in order to include a series of units of processes, methods, systems, or products. The devices are not necessarily limited to those units, but may include other units not explicitly listed or inherent to such processes, methods, products, or devices.

The details are described below separately.

In the contact input mode, when it is inconvenient or impossible to make contact input in some occasions, for example, cooking in the kitchen, it is inconvenient to switch pages when switching recipes, and it is inconvenient to carry out the mobile phone when driving. Operation, the user can not input.

In the image recognition-based manner, the terminal can recognize the gesture of the user based on the image of the gesture made by the user. However, the image recognition algorithm is complex, computationally intensive, high in power consumption, and susceptible to light, and has high performance requirements on the terminal device. Therefore, this method cannot achieve accurate recognition on the terminal.

Based on the above deficiencies, the present application provides a gesture recognition method. In some examples, the method may be specifically applied to a gesture recognition scene of a terminal to a user. Referring to FIG. 1a, the gesture recognition method provided by the example of the present application is applied to the terminal 101. The terminal 101 is configured with a speaker 102 and a microphone 103. The terminal 101 also has an application 104 installed thereon. For example, the application program installed on the terminal includes : Browser APP, or office software APP, or game app, etc., is not limited here.

FIG. 1b is a schematic flow chart of a gesture recognition method according to an illustrative example of the present application.

As shown in FIG. 1b, the gesture recognition method includes the following steps:

S101. Determine, according to an input control indication of the application, whether the terminal turns on the gesture recognition mode.

In the example of the present application, an application is installed on the terminal, and the user can use the application, for example, the user can use the browser APP to query the webpage. The input control instruction may be specifically set in the application's setting menu, or may be set in the application's configuration file. For example, the input control indication may be set in the installation configuration file, and configured by the user when the application is installed on the terminal. This input control indication. For example, in the input control indication menu of the application, the open page of the gesture recognition mode can be set. If the user selects to enable the gesture recognition mode, it can be determined that the gesture recognition mode is enabled on the terminal, and if the user does not enable the gesture recognition mode, It is determined that the terminal has turned off the gesture recognition mode.

It should be noted that, in the example of the present application, when the terminal determines to enable the gesture recognition mode, the subsequent steps are triggered, otherwise the gesture recognition method in the example of the present application is ended. In the example of the present application, the input control indication of the application can control whether the terminal enables the gesture recognition mode.

In some examples of the present application, step S101 determines whether the terminal turns on the gesture recognition mode according to the input control indication of the application, including:

Determining whether the input control indication of the application is the default open gesture recognition mode according to the configuration file of the application;

Detect whether the application runs successfully on the terminal;

When the application runs successfully, if the terminal turns on the gesture recognition mode by default, it is determined that the gesture recognition mode is enabled in the terminal.

In the example of the present application, the input control indication of the application can be configured as the default open gesture recognition mode, and the gesture recognition mode can be automatically turned on after the application runs on the terminal, thereby eliminating the trouble of whether the user resets or not. To simplify the user's operation. Enables the user to automatically enable gesture recognition mode when running an application.

Determining, according to a profile of the application, an input control indication of the application to turn on a gesture recognition mode for the first two-dimensional gesture, or a second two-dimensional gesture to turn off the gesture recognition mode;

When the first two-dimensional gesture is detected, determining that the terminal has turned on the gesture recognition mode;

When the second two-dimensional gesture is detected, it is determined that the terminal has turned off the gesture recognition mode.

The opening and closing of the gesture recognition mode in the example of the present application may also be determined by some preset gesture actions. For example, some two-dimensional gestures may be preset, and the preset two-dimensional gesture is used by the user to control whether the terminal turns on the gesture. Identify the mode to meet the user's real-time control needs for the terminal. For example, determining that the input control indication of the application may open the gesture recognition mode for the first two-dimensional gesture according to the configuration file of the application, the terminal may transmit the ultrasonic wave through the speaker, and acquire the ultrasonic wave through the microphone, thereby identifying whether the user has made a preset. One of the gesture actions (for example, the first two-dimensional gesture) can be turned on only when the preset gesture motion is detected. For another example, according to the configuration file of the application, determining that the input control indication of the application can turn off the gesture recognition mode for the second two-dimensional gesture, the terminal can transmit the ultrasonic wave through the speaker, and acquire the ultrasonic wave through the microphone, thereby identifying whether the user has made a preset. One of the gesture actions (eg, the second two-dimensional gesture) can be turned off only when the preset gesture action is detected.

In some examples of the present application, in addition to performing the foregoing method steps, the gesture recognition method provided by the example of the present application may further include the following steps:

When the speaker plays the training ultrasonic wave, the waveform data of the specified gesture action is recorded through the microphone;

Establishing a gesture waveform template library according to waveform data of the specified gesture action;

The gesture recognition result corresponding to the specified gesture action is determined according to the gesture waveform template library.

In the example of the present application, when the user operates different types of terminals for gesture recognition, since different terminal models have different waveform data received by the user gestures, in order to improve the matching accuracy of the gesture actions, the gesture waveform is established in the example of the present application. Template library. For example, in order to solve the problem of insufficient computing power of the terminal, a cloud server-based gesture waveform template library can also be established. That is, in some examples, the gesture recognition method provided by the examples of the present application can also be applied to the system architecture as shown in FIG. 1c.

As shown in FIG. 1c, the system architecture includes a terminal 101 and a cloud server 105 that interact via one or more Internet 106.

The user records the waveform data of the specified gesture action through the terminal 101, and the cloud server 105 can establish a gesture waveform template library based on the above interaction, and adopts a machine learning method to perform gesture recognition, thereby improving the accuracy of gesture matching. In some examples, cloud server 105 can include two components: one is offline gesture waveform template library training and the other is online gesture recognition.

The terminal 101 may extract the received second ultrasonic wave to the ultrasonic data, and then send the ultrasonic data to the cloud server 105, and the cloud server 105 may acquire the gesture recognition result through the gesture waveform template library by using online cloud recognition, and then the cloud The server 105 sends the gesture recognition result to the terminal 101, so that different terminal types can obtain the gesture recognition result applicable to the terminal type. For example, the type information of the terminal and the configuration information of the terminal can be obtained through the configuration file of the terminal. For example, the system type of the terminal, the operating system version, the number of processors used by the terminal, and the size of the content space, thereby entering the gesture waveform template library set for the terminal according to the type information of the terminal and the configuration information of the terminal, thereby improving gesture matching. Accuracy.

S102. When the terminal determines to enable the gesture recognition mode, the first ultrasonic wave is played through the speaker.

In the example of the present application, the terminal has a built-in speaker, and the terminal can first generate an ultrasonic signal, and then play the ultrasonic wave through the speaker, and define the ultrasonic wave played by the speaker as the “first ultrasonic wave”. After the first ultrasonic wave is emitted from the position where the terminal is located, the first ultrasonic wave hits the target obstacle and reflects to the position where the terminal is located. The target obstacle is mainly a user's hand, such as a single or multiple fingers of the user, or one or two palms of the user.

In the example of the present application, the terminal can use the built-in ultrasonic generator to generate ultrasonic waves, and then transmit the ultrasonic waves through the built-in speakers of the terminal. The ultrasonic generator generates ultrasonic waves by mechanical vibration, and generally propagates in an elastic medium in a longitudinal wave manner, which is a form of energy propagation. The ultrasonic wave has a short wavelength and good directivity. The ultrasonic generator generates ultrasonic waves with a frequency higher than 20,000 Hz, good directionality, strong penetrating power, and easy to obtain concentrated sound energy.

In some examples of the present application, after the first ultrasonic wave is played through the speaker, the gesture recognition method provided by the example of the present application further includes:

Detecting whether there is an abnormality in a signal waveform of the first ultrasonic wave played by the speaker;

If there is an abnormality in the signal waveform of the first ultrasonic wave, re-execute the foregoing step S102: playing the first ultrasonic wave through the speaker;

If there is no abnormality in the signal waveform of the first ultrasonic wave, the trigger performs the following step S103: receiving the second ultrasonic wave through the microphone.

After the first ultrasonic wave is played by the speaker, abnormality detection may be performed on the playing data, for example, the signal waveform of the first ultrasonic wave is detected from multiple dimensions such as time domain and frequency domain. For example, the terminal can detect whether the played data has dropped frames, for example, whether the sampled frequency points are the same as the frequency points played by the speaker in one second. For another example, it is possible to detect whether there is a playback abnormality in the speaker built in the terminal. For another example, the terminal can check whether the energy of the signal is normal in the frequency domain, for example, the ultrasonic signal played is 20khz, and it is determined whether the received ultrasonic energy is 20khz. For example, when the terminal plays an ultrasonic wave, it can also detect whether there is a frame loss by playing music with fast rhythm. By the above-described abnormality determination of the signal waveform of the first ultrasonic wave, the success rate of the received ultrasonic wave can be improved.

S103. Receive a second ultrasonic wave through a microphone, and the second ultrasonic wave is waveform data obtained by collecting the first ultrasonic wave.

In the example of the present application, after the speaker broadcasts the first ultrasonic wave, the sound wave signal can be received through the microphone built in the terminal, and the sound wave signal received by the microphone is defined as the “second ultrasonic wave”. The microphone built in the terminal may be one or more, which is not limited herein. The second ultrasonic wave normally received from the surroundings of the terminal is a mixed signal mixed with ultrasonic waves reflected by the target obstacle, and ultrasonic waves emitted from the speaker, and ultrasonic waves emitted from the vicinity of the terminal.

In some examples of the present application, after the step S103 receives the second ultrasonic wave through the microphone, the gesture recognition method provided by the example of the present application further includes:

Detecting whether there is an abnormality in the signal waveform of the received second ultrasonic wave;

If the signal waveform of the second ultrasonic wave is abnormal, discarding the received second ultrasonic wave;

If there is no abnormality in the signal waveform of the second ultrasonic wave, the trigger performs the following steps: performing gesture recognition according to the received second ultrasonic wave.

After the terminal acquires the second ultrasonic wave, it can also determine whether there is an abnormality in the recorded signal waveform, for example, whether the received signal waveform is complete, that is, whether there is a frame loss. For example, the detection can be performed from multiple dimensions such as frequency domain and time domain, otherwise the subsequent gesture recognition result is affected. There are various methods of detecting, for example, determining whether the received signal waveform energy is equal to the played signal waveform energy. Another example is whether the frequency points recorded per second are the same as the sampling frequency. Another example is the energy of the time-frequency diagram of the signal. Through the aforementioned abnormal determination of the second ultrasonic wave, the gesture recognition is performed only when there is no abnormality in the received second ultrasonic wave, thereby improving the accuracy of the gesture recognition.

In some examples of the present application, step S103 receives the second ultrasonic wave through the microphone, including:

Controlling sound wave signals in the surrounding environment of the microphone recording terminal according to preset recording parameters;

The recorded sound wave signal is stored in the recording buffer of the terminal, wherein the sound wave signal stored in the recording buffer is the second ultrasonic wave.

Among them, the terminal preset recording parameters can be various, such as mono or multi-channel. The terminal can select mono, multi-channel, etc. according to the needs. For example, the recording parameter may further include a sampling frequency, which is the same as the transmission frequency of the ultrasonic wave. After the acoustic signal is acquired, the terminal can store the acoustic signal in the recording buffer. The size of the recording buffer can be set as follows. If the recording buffer is set too small, it may cause frame loss. Generally, the minimum buffer size is more than 2 times.

S104. Perform gesture recognition according to the received second ultrasonic wave to obtain a gesture recognition result.

In the example of the present application, after the second ultrasonic wave is collected by the foregoing step S103, the received second ultrasonic wave may be gesture-recognized. Since the first ultrasonic wave emitted by the speaker is blocked by the user's hand, the microphone collects the human hand. The blocked ultrasonic wave, so by analyzing the receiving position, waveform energy, and the like of the second ultrasonic wave received by the microphone, the gesture made by the user can be recognized, and the gesture recognition result is obtained.

In some examples of the present application, in addition to performing the foregoing method steps, the step S104 performs gesture recognition according to the received second ultrasonic wave, and after the gesture recognition result is obtained, the gesture recognition method provided by the example of the present application may further include the following steps:

The prompt information for successful gesture recognition is displayed through the display interface of the application.

The display interface of the application of the terminal in the example of the present application can also implement real-time interaction with the user, and the second ultrasonic wave received by the step S104 performs gesture recognition to obtain a gesture recognition result, indicating that the current gesture recognition is successful, in order to avoid The user repeatedly performs multiple identical actions, and the display interface of the application of the terminal can also display prompt information, for example, by using an animation displayed on the display interface to inform the user of the current gesture recognition program, or by prompting the user by text or sound, here is not Make a limit.

S105. Perform a corresponding gesture instruction in the application according to the gesture recognition result.

In the example of the present application, after the gesture recognition result is obtained, the terminal may respond to the gesture of the user, and may execute a corresponding gesture instruction according to the gesture recognition result, for example, operating the application of the terminal in response to the gesture instruction of the user. For example, the gesture recognition result is a circle, and the terminal executes a gesture instruction corresponding to the gesture recognition result on the application: opening the application or operating a certain menu of the application. The correspondence between the gesture recognition result and the gesture instruction may be determined according to a pre-configured list of the user on the terminal. For example, the terminal recognizes the gesture of the user to obtain a gesture recognition result, and the terminal may execute a gesture instruction of the user on the browser application, for example, inputting a text in a search box of the browser.

In some examples of the present application, step S102 plays the first ultrasonic wave through the speaker, including:

Playing a first ultrasonic wave of N frequencies through a speaker, N being a positive integer greater than or equal to 1;

In this implementation scenario, step S103 receives the second ultrasonic wave through the microphone, including:

A second ultrasonic wave of N frequencies is collected from the surroundings of the terminal through a microphone.

Wherein, the terminal can transmit ultrasonic waves of one or more frequencies, for example, transmitting a plurality of ultrasonic waves of different frequencies, and the ultrasonic waves can be separated by the same distance. The terminal can separately acquire ultrasonic waves of one or more frequencies played by the speaker through the microphone.

In some examples of the present application, step S104 performs gesture recognition according to the received second ultrasonic wave to obtain a gesture recognition result, including:

When the value of N is 1, the one-dimensional gesture recognition is performed according to the waveform energy of the second ultrasonic wave, and the first gesture recognition result is obtained. The first gesture recognition result includes: the gesture is close to the terminal, or is away from the terminal.

Wherein, the terminal can transmit an ultrasonic wave through the speaker, and the terminal can collect the ultrasonic wave through the microphone, and the terminal can perform coarse-grained one-dimensional gesture recognition on the received one ultrasonic wave, for example, the first gesture is obtained by Doppler waveform energy change. The result of the recognition, the first gesture recognition result includes: the gesture is close to the terminal, or is away from the terminal.

In other examples of the present application, step S104 performs gesture recognition according to the received second ultrasonic wave to obtain a gesture recognition result, including:

When the value of N is greater than or equal to 2, the moving distance of the gesture is calculated for each of the N second ultrasonic waves;

Excluding an abnormal moving distance from the moving distance of the N gestures, performing one-dimensional gesture recognition on the retained moving distance, and obtaining a first gesture recognition result, where the first gesture recognition result includes: the gesture is close to the terminal, or is away from the terminal, or Moving to the left relative to the terminal, or moving to the right relative to the terminal, or moving up relative to the terminal, or moving downward relative to the terminal.

The terminal can transmit a plurality of ultrasonic waves through the speaker, and the terminal can separately collect each ultrasonic wave through the microphone, and the terminal can perform fine-grained one-dimensional gesture recognition on the received plurality of ultrasonic waves. For example, using N-frequency ultrasonic phase changes to calculate the distance, the response is more sensitive. N ultrasonic waves in the middle of 17500HZ-23000HZ are emitted. The distance between adjacent ultrasonic waves is the same. The sampling frequency is 48000Hz. 512 points can be sampled to calculate the distance change in real time. The reaction time is 10.7ms. In the process of sound transmission, there will be some multipath effects, and some background static objects. In order to reduce the multipath effect and background interference, the dynamic signal can be subtracted from the background interference, and N distances can be calculated by N frequencies. The least squares method is used to calculate the deviation, and the frequency distance with large deviation is removed, and the deviation is small. By excluding the abnormal distance, the calculation accuracy of the distance can be improved.

The second ultrasonic wave is separately collected by the two microphones of the terminal.

In this implementation scenario, step S104 performs gesture recognition according to the received second ultrasonic wave to obtain a gesture recognition result, including:

Calculating a relative position and an initial position of the gesture according to the second ultrasonic waves respectively received by the two microphones;

The two-dimensional gesture recognition is performed according to the calculated relative position and the initial position, and the second gesture recognition result is obtained, and the second gesture recognition result includes: two-dimensional coordinates of the gesture change.

In order to improve the accuracy of the gesture recognition, at least two microphones may be disposed in the terminal to separately acquire the second ultrasonic waves. Then, for the second ultrasonic wave received by each microphone, the relative position and initial position of the gesture can be calculated. Wherein, the relative position of the gesture refers to the position of the gesture relative to the terminal based on the phase measurement of the second ultrasonic wave, and the initial position of the gesture refers to the position of the user gesture at the time of initial recognition. Through the foregoing calculated relative position and initial position of the gesture, two-dimensional gesture recognition can be performed, the precise coordinates of the gesture can be obtained, the accuracy of gesture recognition can be improved, and misoperation can be prevented.

Through the foregoing examples, the description of the present application shows that a speaker and a microphone are disposed in the terminal, and an application program is installed on the terminal. Firstly, according to the input control indication of the application, determining whether the terminal turns on the gesture recognition mode, when the terminal determines to turn on the gesture recognition mode, playing the first ultrasonic wave through the speaker, and then receiving the second ultrasonic wave through the microphone, and the second ultrasonic wave is for the first ultrasonic wave The waveform data obtained after the acquisition is then subjected to gesture recognition according to the received second ultrasonic wave to obtain a gesture recognition result, and finally a corresponding gesture instruction is executed in the application according to the gesture recognition result. In the example of the present application, the gesture recognition can be realized by using the built-in speaker and microphone of the terminal, that is, the motion of the user can be recognized through the transmission and detection of the ultrasonic wave, the complexity of the gesture recognition is reduced, and the calculation performance requirement of the terminal is reduced. And implements gesture control of the user on the application. Further, by calculating the second ultrasonic wave, the moving distance of the gesture or the two-dimensional change of the gesture is obtained, and the accuracy of the gesture recognition is improved.

To facilitate a better understanding and implementation of the above solution of the example of the present application, the following application scenarios are exemplified, for example, the application scenario shown in FIG. 1a is used for specific description.

The application example can be applied to a mobile phone browser and a game application (Application, APP). The application example can realize high-precision, low-delay, no-peripheral, non-contact one-dimensional and two-dimensional gesture recognition. The example of the present application performs ultrasonic gesture detection based on the speaker and microphone of the terminal, and does not require any external equipment. For example, in the example of the present application, the user does not need to manually operate the touch screen of the terminal, and does not need to input any command from the keyboard, and the user only needs to make a gesture action. The terminal in the example of the present application can detect by sending and receiving ultrasonic waves. The gesture action made by the user is performed to execute the gesture instruction corresponding to the gesture action on the application of the terminal, for example, the user operates the game application by making a gesture, and the character movement in the game application can be controlled, and the game application is opened. With off and so on.

Taking the gesture recognition in the browsing scenario of the browser APP as an example, the user can operate basic browsing scenarios such as upper, lower, left, and right.

Please refer to Figure 2 for an example of the ultrasonic transmission process.

201. Set the transmission frequency and sampling frequency of the ultrasonic wave. For example, an ultrasonic wave of N (N>=1) frequencies is transmitted, and the sampling frequency is twice or more of the transmission frequency.

202. Set the ultrasonic playback mode.

The terminal can adopt multiple playback modes such as static and streaming, and can be set according to user requirements.

Ultrasonic playback requirements are strict. If there is abnormal playback or frame loss, it will lead to calculation errors or errors. Therefore, the data acquisition process of the ultrasonic playback process needs to be prepared in advance. The playback process logic needs to ensure that the waveform data played is not dropped. That is, the waveform data that needs to be played is complete.

203. Detect a sound wave signal played by the speaker.

The terminal can detect whether the played data is abnormal, and can detect from multiple dimensions such as time domain and frequency domain. There are many ways to detect. For example, if the data being played is detected, there is no frame loss, and it is not the number of points for playing the sampling frequency in 1 second. Another example is to detect whether the playback module has reported an exception. Another example is to check whether the energy of the signal is normal in the frequency domain. For example, the frequency of the broadcast is 20khz, and the frequency of the received signal is 20khz. In addition, the playback process can also detect the loss of frames with fast-paced music playback.

As shown in Fig. 3, the ultrasonic reception is exemplified next.

301. Set the recording parameters of the microphone.

Among them, you can select mono, multi-channel, etc. according to your needs, and the sampling frequency is twice the transmission frequency of the ultrasonic wave. According to Nyquist's sampling law, the sampling frequency is compared with the playback frequency, and the sampling frequency can be twice the playback frequency. If the recording buffer is set too small, it may cause frame loss. Generally, the size of the minimum buffer is more than 2 times.

302. Start the recording thread.

Among them, reading data from the microphone, to prevent the recorded data recording thread from being completely recorded, resulting in frame loss. The logic of the recording thread is as light as possible, ensuring that the sampling frequency is received in one second.

303. Detect the recorded data of the microphone.

Among them, whether the recorded signal waveform is complete, whether there is no frame loss, need to detect from multiple dimensions such as frequency domain and time domain, otherwise it will affect the subsequent gesture recognition result. There are many ways to detect, such as whether the received signal energy is the energy of the frequency of playback. As another example, is the number of points recorded per second the same as the sampling frequency. As another example, there is no impurity in the energy of the time-frequency diagram of the signal.

In the example of the present application, the ultrasonic processing may be specifically a one-dimensional gesture recognition process or a two-dimensional gesture recognition process, which is respectively illustrated by way of example.

First, the one-dimensional gesture recognition processing will be described. Referring to FIG. 4, the one-dimensional gesture recognition processing can implement two schemes.

One of the coarse-grained gesture recognitions mainly includes:

401. Acquire waveform energy of ultrasonic waves. Using the Doppler principle, in the process of moving away from the gesture, a significant change in the waveform energy can be seen in the spectrum.

402. Modeling a multi-dimensional feature such as a velocity, an acceleration, and the like of the waveform to obtain a one-dimensional gesture change. Mainly according to the Doppler effect, in the process of the gesture approaching and far away, the waveform has different trends, and the calculation is based on the direction and acceleration of the waveform change. For example, as follows, an ultrasonic wave of 20,000 Hz is transmitted, and the sampling frequency is 44100 Hz, and 4096 points can be sampled each time for calculation.

Another fine-grained gesture recognition includes:

403. Calculating the distance by using the ultrasonic phase changes of N frequencies, the reaction is more sensitive. N ultrasonic waves in the middle of 17500HZ-23000HZ, the distance between adjacent two ultrasonic waves is the same, the sampling frequency is 48000Hz, and 512 points can be sampled to calculate the distance change in real time. The reaction time is 10.7ms, that is, in this example, the gesture The accuracy of the recognition can reach the ms level.

404. In order to calculate more accurately, the distance calculated for the phase changes of the N frequencies is eliminated by an algorithm to improve the calculation accuracy of the distance. In the process of sound transmission, there will be some multipath effects, and some background static objects will interfere. In order to reduce the multipath effect and background interference, the background interference will be subtracted, and N distances will be calculated by N frequencies. According to the least squares method, Calculate the deviation, remove the frequency distance with large deviation, and keep the distance with small deviation.

Next, the two-dimensional gesture recognition processing will be described. Referring to FIG. 5, the following processes are mainly included:

501. Acquire ultrasonic waves using two microphones respectively.

502. Determine whether the ultrasonic calculation of the two microphones is completed.

Among them, in the process of gesture movement, the influence of the two microphones on the waveform is different, so that the change of the two directions can be obtained.

The two-dimensional gesture recognition processing mainly solves the recognition of the two-dimensional coordinates of the gesture action, thereby realizing the recognition of the two-dimensional graphics (drawing circles, drawing squares, drawing triangles, etc.) or writing Chinese characters. With two speakers, two channels of data can be sampled.

Steps

503 and 504 are performed for each microphone channel.

503. Calculate a relative distance of the gesture movement based on the one-dimensional gesture recognition.

The phase based on the one-dimensional gesture recognition can calculate the relative distance of the gesture movement.

504. Calculate an initial position of the gesture based on the signal.

Wherein, based on the signal received by the microphone, the coarse-grained initial position of the gesture can be calculated. The initial position is the initial position of the gesture detected when the ultrasonic wave is received.

505. Combine the coordinates in two directions to obtain two-dimensional gesture recognition.

After the initial position and relative position calculation of the two microphone channels are completed, the two-dimensional gesture recognition can be combined with the coordinates in the two directions of x and y, and the precise coordinates (x, y) of the gesture recognition can be obtained through the initial The position and relative position changes can get the real-time gesture position.

As shown in FIG. 6, the management of gesture recognition is exemplified next. Through the action management of gesture recognition, avoid misuse and unnecessary interruption. mainly includes:

601. Determine, according to the two-dimensional gesture of the user, turning on and off the function of the gesture recognition.

Users can set their favorite gestures according to their needs. It is recommended to use complex two-dimensional gestures, such as drawing circles, drawing squares, drawing triangles and the like. As shown in FIG. 7-a, the motion of the gesture recognition may be a circular shape, as shown in FIG. 7-b, the motion recognition of the gesture recognition may be a square.

602. Perform an operation function of gesture recognition according to a one-dimensional gesture of the user.

Among them, the user can set the gesture recognition action to meet the basic operational needs. As shown in FIG. 8, the basic operation of the gesture recognition may include basic actions such as up, down, left, and right.

It should be noted that, in the example of the present application, in order to solve the computational burden too much, the playback and recording of the ultrasonic wave may be affected, thereby affecting the accuracy of the gesture recognition. In the example of the present application, an efficient programming language may be used, such as calculation logic. Language implementation, such as the use of pointer operations, optimize the processor's high-cost logic operations, minimize memory copy, input and output operations, and reduce computation time. Therefore, the time consumption of one-dimensional gesture recognition calculation in the example of the present application can be reduced to the ms level.

In the gesture recognition scene, the accuracy of the gesture recognition and the fastest response time are the most important. In the example of the present application, the user's misoperation can be prevented, and the opening and ending of the gesture recognition mode can be recognized by the two-dimensional gesture. In order to recognize an action more quickly and accurately, such as approaching, moving away, leftward, and rightward, the present application example proposes a one-dimensional gesture recognition manner, wherein the ultrasonic phase-based approach can achieve millisecond (mm) level accuracy. In the case where the hardware of the terminal such as the mobile phone is limited, the example of the present application can improve the calculation performance as much as possible and save the calculation time.

It should be noted that, for the foregoing method examples, for the sake of simple description, they are all expressed as a series of action combinations, but those skilled in the art should understand that the present application is not limited by the described action sequence, because In accordance with the present application, certain steps may be performed in other sequences or concurrently. Secondly, those skilled in the art should also understand that the examples described in the specification are all preferred examples, and the actions and modules involved are not necessarily required by the present application.

In order to facilitate the better implementation of the above described solution of the examples of the present application, related devices for implementing the above solutions are also provided below.

Referring to FIG. 9-a, a terminal 900 is provided. The terminal is configured with a speaker and a microphone. The terminal is installed with an application program, and the terminal may include: a mode determining module 901, and an ultrasound. a sending module 902, an ultrasonic acquiring module 903, a gesture recognition module 904, and an instruction execution module 905, wherein

The mode determining module 901 is configured to determine, according to an input control indication of the application, whether the terminal turns on the gesture recognition mode;

The ultrasonic sending module 902 is configured to: when the terminal determines to enable the gesture recognition mode, play the first ultrasonic wave through the speaker;

The ultrasonic acquisition module 903 is configured to receive a second ultrasonic wave by using the microphone, where the second ultrasonic wave is waveform data obtained after collecting the first ultrasonic wave;

a gesture recognition module 904, configured to perform gesture recognition according to the received second ultrasonic wave, to obtain a gesture recognition result;

The instruction execution module 905 is configured to execute a corresponding gesture instruction in the application according to the gesture recognition result.

In some examples of the present application, the ultrasonic transmitting module 902 is specifically configured to play a first ultrasonic wave of N frequencies through the speaker, where N is a positive integer greater than or equal to 1;

The ultrasonic acquisition module is configured to collect, by the microphone, a second ultrasonic wave of N frequencies from a surrounding environment of the terminal.

Further, in some examples of the present application, the gesture recognition module 904 is specifically configured to perform one-dimensional gesture recognition according to the waveform energy of the second ultrasonic wave when the value of the N is 1, to obtain a first A gesture recognition result, the first gesture recognition result includes: the gesture is close to the terminal, or is away from the terminal.

In some examples of the present application, referring to FIG. 9-b, the mode determining module 901 includes:

The control indication parsing module 9011 is configured to determine, according to the configuration file of the application, whether the input control indication of the application is a default open gesture recognition mode;

The program detection module 9012 is configured to detect whether the application runs successfully on the terminal;

The mode-opening module 9013 is configured to determine, when the application runs successfully, that the terminal turns on the gesture recognition mode by default, and determines that the terminal has enabled the gesture recognition mode.

In some examples of the present application, the mode determining module 901 is specifically configured to determine, according to the configuration file of the application, that the input control indication of the application is a first two-dimensional gesture open gesture recognition mode, or a second two-dimensional gesture Turning off the gesture recognition mode; when detecting the first two-dimensional gesture, determining that the terminal has turned on the gesture recognition mode; and when detecting the second two-dimensional gesture, determining that the terminal turns off the gesture recognition mode.

In some examples of the present application, as shown in FIG. 9-c, the terminal 900 further includes: a template library establishing module 906 and a gesture action matching module 907, where

The ultrasonic acquisition module 903 is configured to record waveform data of the specified gesture action through the microphone when the speaker plays the training ultrasonic wave;

a template library establishing module 906, configured to establish a gesture waveform template library according to the waveform data of the specified gesture action;

The gesture action matching module 907 is configured to determine a gesture recognition result corresponding to the specified gesture action according to the gesture waveform template library.

In some examples of the present application, as shown in FIG. 9-d, the terminal 900 further includes:

The display module 908 is configured to perform gesture recognition according to the received second ultrasonic wave to obtain gesture recognition result, and then display prompt information for successful gesture recognition through the display interface of the application.

In some examples of the present application, the gesture recognition module 904 includes:

a distance calculating module, configured to calculate a moving distance of the gesture for each of the N second ultrasonic waves when the value of the N is greater than or equal to 2;

a one-dimensional identification module, configured to exclude an abnormal moving distance from the moving distances of the N gestures, perform one-dimensional gesture recognition on the retained moving distance, and obtain a first gesture recognition result, where the first gesture recognition result is The method includes: the gesture is close to the terminal, or is away from the terminal, or moves to the left relative to the terminal, or moves to the right relative to the terminal, or moves upward relative to the terminal, or is opposite to the terminal Move down.

In some examples of the present application, the ultrasonic acquisition module 903 is specifically configured to separately collect a second ultrasonic wave through two microphones of the terminal;

The gesture recognition module 904 includes:

a position calculation module, configured to calculate a relative position and an initial position of the gesture according to the second ultrasonic waves respectively received by the two microphones;

The two-dimensional recognition module is configured to perform two-dimensional gesture recognition according to the calculated relative position and the initial position to obtain a second gesture recognition result, where the second gesture recognition result includes: two-dimensional coordinates of the gesture change.

In some examples of the present application, the ultrasonic acquisition module 903 includes:

a sound wave recording module, configured to control the microphone to record an acoustic signal in a surrounding environment of the terminal according to preset recording parameters;

And a signal storage module, configured to store the recorded acoustic wave signal into a recording buffer of the terminal, wherein the sound wave signal stored in the recording buffer is the second ultrasonic wave.

As can be seen from the above description of the examples of the present application, a speaker and a microphone are disposed in the terminal, and an application is installed on the terminal. Firstly, according to the input control indication of the application, determining whether the terminal turns on the gesture recognition mode, when the terminal determines to turn on the gesture recognition mode, playing the first ultrasonic wave through the speaker, and then receiving the second ultrasonic wave through the microphone, and the second ultrasonic wave is for the first ultrasonic wave The waveform data obtained after the acquisition is then subjected to gesture recognition according to the received second ultrasonic wave to obtain a gesture recognition result, and finally a corresponding gesture instruction is executed in the application according to the gesture recognition result. In the example of the present application, the gesture recognition can be realized by using the built-in speaker and microphone of the terminal, that is, the motion of the user can be recognized through the transmission and detection of the ultrasonic wave, the complexity of the gesture recognition is reduced, and the calculation performance requirement of the terminal is reduced. And implements gesture control of the user on the application. Further, by calculating the second ultrasonic wave, the moving distance of the gesture or the two-dimensional change of the gesture is obtained, and the accuracy of the gesture recognition is improved.

The present application also provides a terminal, as shown in FIG. 10, for the convenience of description, only the parts related to the examples of the present application are shown. For the specific technical details not disclosed, please refer to the example method part of the present application. The terminal may be any terminal device including a mobile phone, a tablet computer, a PDA (Personal Digital Assistant), a POS (Point of Sales), an in-vehicle computer, and the terminal is a mobile phone as an example:

FIG. 10 is a block diagram showing a partial structure of a mobile phone associated with a terminal provided by an example of the present application. Referring to FIG. 10, the mobile phone includes: a radio frequency (RF) circuit 1010, a memory 1020, an input unit 1030, a display unit 1040, a sensor 1050, an audio circuit 1060, a wireless fidelity (WiFi) module 1070, and a processor 1080. And power supply 1090 and other components. It will be understood by those skilled in the art that the structure of the handset shown in FIG. 10 does not constitute a limitation to the handset, and may include more or less components than those illustrated, or some components may be combined, or different component arrangements.

The following describes the components of the mobile phone in detail with reference to FIG. 10:

The RF circuit 1010 can be used for receiving and transmitting signals during the transmission or reception of information or during a call. In particular, after receiving the downlink information of the base station, it is processed by the processor 1080. In addition, the uplink data is designed to be sent to the base station. Generally, RF circuit 1010 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a Low Noise Amplifier (LNA), a duplexer, and the like. In addition, RF circuit 1010 can also communicate with the network and other devices via wireless communication. The above wireless communication may use any communication standard or protocol, including but not limited to Global System of Mobile communication (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (Code Division). Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), E-mail, Short Messaging Service (SMS), and the like.

The memory 1020 can be used to store software programs and modules, and the processor 1080 executes various functional applications and data processing of the mobile phone by running software programs and modules stored in the memory 1020. The memory 1020 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application required for at least one function (such as a sound playing function, an image playing function, etc.), and the like; the storage data area may be stored according to Data created by the use of the mobile phone (such as audio data, phone book, etc.). Moreover, memory 1020 can include high speed random access memory, and can also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid state storage device.

Input unit 1030 can be used to receive input numeric or character information and to generate key signal inputs related to user settings and function control of the handset. Specifically, the input unit 1030 may include a touch panel 1031 and other input devices 1032. The touch panel 1031, also referred to as a touch screen, can collect touch operations on or near the user (such as the user using a finger, a stylus, or the like on the touch panel 1031 or near the touch panel 1031. Operation), and drive the corresponding connecting device according to a preset program. In some examples, the touch panel 1031 can include two portions of a touch detection device and a touch controller. Wherein, the touch detection device detects the touch orientation of the user, and detects a signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts the touch information into contact coordinates, and sends the touch information. The processor 1080 is provided and can receive commands from the processor 1080 and execute them. In addition, the touch panel 1031 can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic waves. In addition to the touch panel 1031, the input unit 1030 may also include other input devices 1032. Specifically, other input devices 1032 may include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control buttons, switch buttons, etc.), trackballs, mice, joysticks, and the like.

The display unit 1040 can be used to display information input by the user or information provided to the user as well as various menus of the mobile phone. The display unit 1040 may include a display panel 1041. In some examples, the display panel 1041 may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like. Further, the touch panel 1031 may cover the display panel 1041, and when the touch panel 1031 detects a touch operation thereon or nearby, the touch panel 1031 transmits to the processor 1080 to determine the type of the touch event, and then the processor 1080 according to the touch event. The type provides a corresponding visual output on display panel 1041. Although in FIG. 10, the touch panel 1031 and the display panel 1041 are two independent components to implement the input and input functions of the mobile phone, in some examples, the touch panel 1031 and the display panel 1041 may be integrated. The input and output functions of the phone.

The handset can also include at least one type of sensor 1050, such as a light sensor, motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 1041 according to the brightness of the ambient light, and the proximity sensor may close the display panel 1041 and/or when the mobile phone moves to the ear. Or backlight. As a kind of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in all directions (usually three axes). When it is stationary, it can detect the magnitude and direction of gravity. It can be used to identify the gesture of the mobile phone (such as horizontal and vertical screen switching, related Game, magnetometer attitude calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for the mobile phone can also be configured with gyroscopes, barometers, hygrometers, thermometers, infrared sensors and other sensors, no longer Narration.

Audio circuit 1060, speaker 1061, microphone 1062, microphone 1063 can provide an audio interface between the user and the handset. The audio circuit 1060 can transmit the converted electrical data of the received audio data to the speaker 1061, and convert it into a sound signal output by the speaker 1061; on the other hand, the microphone 1062 converts the collected sound signal into an electrical signal, by the audio circuit 1060. After receiving, it is converted into audio data, and then processed by the audio data output processor 1080, sent to the other mobile phone via the RF circuit 1010, or outputted to the memory 1020 for further processing. The speaker 1061 can generate an ultrasonic signal, and then the microphone 1063 can acquire an ultrasonic signal from around the mobile phone.

WiFi is a short-range wireless transmission technology. The mobile phone through the WiFi module 1070 can help users to send and receive e-mail, browse the web and access streaming media, etc. It provides users with wireless broadband Internet access. Although FIG. 10 shows the WiFi module 1070, it can be understood that it does not belong to the essential configuration of the mobile phone, and can be omitted as needed within the scope of not changing the essence of the application.

The processor 1080 is the control center of the handset, which connects various portions of the entire handset using various interfaces and lines, by executing or executing software programs and/or modules stored in the memory 1020, and invoking data stored in the memory 1020, The phone's various functions and processing data, so that the overall monitoring of the phone. In some examples, processor 1080 can include one or more processing units; preferably, processor 1080 can integrate an application processor and a modem processor, wherein the application processor primarily processes an operating system, a user interface, and an application Etc. The modem processor primarily handles wireless communications. It will be appreciated that the above described modem processor may also not be integrated into the processor 1080.

The mobile phone also includes a power source 1090 (such as a battery) that supplies power to various components. Preferably, the power source can be logically coupled to the processor 1080 through a power management system to manage functions such as charging, discharging, and power management through the power management system.

Although not shown, the mobile phone may further include a camera, a Bluetooth module, and the like, and details are not described herein again.

In the example of the present application, the processor 1080 included in the terminal further has a flow of controlling a gesture recognition method performed by the terminal. For example, the process of identifying the ultrasonic waves by the processor 1080 can be referred to the description in the foregoing examples.

Next, another terminal provided by the example of the present application is introduced. Referring to FIG. 11, the terminal 1100 includes:

The speaker 1101, the microphone 1102, the processor 1103, and the memory 1104 (wherein the number of the processors 1103 in the terminal 1100 may be one or more, and one processor in FIG. 11 is taken as an example). In some examples of the present application, the speaker 1101, the microphone 1102, the processor 1103, and the memory 1104 may be connected by a bus or other means, wherein FIG. 11 is exemplified by a bus connection.

Memory 1104 can include read only memory and random access memory and provides instructions and data to processor 1103. A portion of the memory 1104 may also include a non-volatile random access memory (English name: Non-Volatile Random Access Memory, English abbreviation: NVRAM). The memory 1104 stores operating systems and operational instructions, executable modules or data structures, or a subset thereof, or an extended set thereof, wherein the operational instructions can include various operational instructions for implementing various operations. The operating system can include a variety of system programs for implementing various basic services and handling hardware-based tasks.

The processor 1103 controls the operation of the terminal, and the processor 1103 can also be referred to as a central processing unit (English full name: Central Processing Unit, English abbreviation: CPU). In a specific application, the components of the terminal are coupled together by a bus system. The bus system may include a power bus, a control bus, and a status signal bus in addition to the data bus. However, for the sake of clarity, the various buses are referred to as bus systems in the figures.

The method disclosed in the above examples of the present application may be applied to the processor 1103 or implemented by the processor 1103. The processor 1103 can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method may be completed by an integrated logic circuit of hardware in the processor 1103 or an instruction in a form of software. The processor 1103 may be a general-purpose processor, a digital signal processor (English full name: digital signal processing, English abbreviation: DSP), an application specific integrated circuit (English name: Application Specific Integrated Circuit, English abbreviation: ASIC), field programmable Gate array (English name: Field-Programmable Gate Array, English abbreviation: FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The methods, steps, and logical block diagrams disclosed in the examples of the present application can be implemented or executed. The general purpose processor may be a microprocessor or the processor or any conventional processor or the like. The steps of the method disclosed in connection with the examples of the present application may be directly embodied by the execution of the hardware decoding processor or by a combination of hardware and software modules in the decoding processor. The software module can be located in a conventional storage medium such as random access memory, flash memory, read only memory, programmable read only memory or electrically erasable programmable memory, registers, and the like. The storage medium is located in the memory 1104, and the processor 1103 reads the information in the memory 1104 and performs the steps of the above method in combination with its hardware.

The speaker 1101 is configured to play a first ultrasonic wave under the control of the processor;

The microphone 1102 is configured to receive a second ultrasonic wave under the control of the processor;

In the example of the present application, the processor 1103 is configured to execute the instructions in the memory, and perform the method in the foregoing example.

It should be further noted that the device examples described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical. Units can be located in one place or distributed to multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present example. In addition, in the example of the device provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, and specifically may be implemented as one or more communication buses or signal lines. Those of ordinary skill in the art can understand and implement without any creative effort.

Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, dedicated hardware, dedicated CPU, dedicated memory, dedicated memory, Special components and so on. In general, functions performed by computer programs can be easily implemented with the corresponding hardware, and the specific hardware structure used to implement the same function can be various, such as analog circuits, digital circuits, or dedicated circuits. Circuits, etc. However, software program implementation is a better implementation for more applications in this application. Based on such understanding, the technical solution of the present application, which is essential or contributes to the prior art, may be embodied in the form of a software product stored in a readable storage medium, such as a floppy disk of a computer. , U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., including a number of instructions to make a computer device (may be A personal computer, server, or network device, etc.) performs the methods described in various examples of the present application.

In summary, the above examples are only used to illustrate the technical solutions of the present application, and are not limited thereto; although the present application is described in detail with reference to the above examples, those skilled in the art should understand that they can still The technical solutions described in the examples are modified, or the equivalents of some of the technical features are replaced. The modifications and substitutions do not detract from the spirit and scope of the technical solutions of the examples of the present application.

Claims

A gesture recognition method is applied to a terminal, wherein the terminal is configured with a speaker and a microphone, and an application is installed on the terminal, and the method includes:

Determining whether the terminal turns on the gesture recognition mode according to an input control indication of the application;

When the terminal determines to turn on the gesture recognition mode, playing the first ultrasonic wave through the speaker;

Receiving, by the microphone, a second ultrasonic wave, wherein the second ultrasonic wave is waveform data obtained after collecting the first ultrasonic wave;

Performing gesture recognition according to the received second ultrasonic wave to obtain a gesture recognition result;

Corresponding gesture instructions are executed in the application according to the gesture recognition result.
The method according to claim 1, wherein the determining, according to an input control indication of the application, whether the terminal turns on a gesture recognition mode comprises:

Determining, according to a configuration file of the application, whether an input control indication of the application is a default open gesture recognition mode;

Detecting whether the application runs successfully on the terminal;

When the application runs successfully, if the terminal turns on the gesture recognition mode by default, it is determined that the terminal has turned on the gesture recognition mode.
The method according to claim 1, wherein the determining, according to an input control indication of the application, whether the terminal turns on a gesture recognition mode comprises:

Determining, according to a configuration file of the application, an input control indication of the application to turn on a gesture recognition mode for a first two-dimensional gesture, or a second two-dimensional gesture to turn off a gesture recognition mode;

When the first two-dimensional gesture is detected, determining that the terminal turns on the gesture recognition mode;

When the second two-dimensional gesture is detected, it is determined that the terminal turns off the gesture recognition mode.
The method of claim 1 wherein the method further comprises:

Recording waveform data of the specified gesture action through the microphone when the speaker plays the training ultrasonic wave;

Establishing a gesture waveform template library according to the waveform data of the specified gesture action;

Determining, according to the gesture waveform template library, a gesture recognition result corresponding to the specified gesture action.
The method according to claim 1, wherein after the gesture recognition is performed according to the received second ultrasonic wave to obtain a gesture recognition result, the method further includes:

The prompt information for successful gesture recognition is displayed through the display interface of the application.
The method of claim 1 wherein said playing the first ultrasonic wave through said speaker comprises:

Playing a first ultrasonic wave of N frequencies through the speaker, the N being a positive integer greater than or equal to 1;

Performing gesture recognition according to the received second ultrasonic wave to obtain a gesture recognition result, including:

When the value of N is greater than or equal to 2, the moving distance of the gesture is calculated for each of the N second ultrasonic waves;

Excluding an abnormal moving distance from the moving distances of the N gestures, and performing one-dimensional gesture recognition on the retained moving distance to obtain a first gesture recognition result, where the first gesture recognition result includes: the gesture is close to the terminal Or moving away from the terminal, or moving to the left relative to the terminal, or moving to the right relative to the terminal, or moving upward relative to the terminal, or moving downward relative to the terminal.
The method of claim 1, wherein the receiving the second ultrasonic wave through the microphone comprises:

Collecting a second ultrasonic wave through two microphones of the terminal;

Performing gesture recognition according to the received second ultrasonic wave to obtain a gesture recognition result, including:

Calculating a relative position and an initial position of the gesture according to the second ultrasonic waves respectively received by the two microphones;

Performing a two-dimensional gesture recognition according to the calculated relative position and the initial position, and obtaining a second gesture recognition result, where the second gesture recognition result includes: two-dimensional coordinates of the gesture change.
A terminal is configured with a speaker and a microphone, and an application is installed on the terminal, and the terminal includes:

a mode determining module, configured to determine, according to an input control indication of the application, whether the terminal turns on a gesture recognition mode;

An ultrasonic transmitting module, configured to: when the terminal determines to enable the gesture recognition mode, play the first ultrasonic wave through the speaker;

An ultrasonic acquisition module, configured to receive a second ultrasonic wave by using the microphone, where the second ultrasonic wave is waveform data obtained after collecting the first ultrasonic wave;

a gesture recognition module, configured to perform gesture recognition according to the received second ultrasonic wave, to obtain a gesture recognition result;

And an instruction execution module, configured to execute a corresponding gesture instruction in the application according to the gesture recognition result.
A terminal comprising: a speaker, a microphone, a processor, and a memory;

The processor and the speaker, the microphone, and the memory communicate with each other;

The memory is for storing instructions;

The speaker for playing a first ultrasonic wave under the control of the processor;

The microphone for receiving a second ultrasonic wave under the control of the processor;

The processor, for executing the instructions in the memory, performing the method of any one of claims 1 to 7.
A computer readable storage medium comprising instructions which, when executed on a computer, cause the computer to perform the method of any of claims 1-7.