CN108351708B - Three-dimensional gesture unlocking method, gesture image obtaining method and terminal equipment - Google Patents

Abstract

The application provides a three-dimensional gesture unlocking method, a method for acquiring a gesture image and terminal equipment. The three-dimensional gesture unlocking method comprises the following steps: acquiring a current gesture image of a user; acquiring contour data of a current gesture in the current gesture image according to the current gesture image; according to the contour data of the current gesture, determining fingertip pixel points of the current gesture and/or finger root pixel points of the current gesture; determining the characteristic data of the current gesture according to the fingertip pixel point of the current gesture and/or the finger root pixel point of the current gesture; and unlocking the terminal equipment when the feature data of the current gesture is successfully matched with the feature data of the preset gesture. According to the three-dimensional gesture unlocking method, a brand-new unlocking mode with strong interestingness, high accuracy and good rapidness is provided for a user.

Description

Three-dimensional gesture unlocking method, gesture image obtaining method and terminal equipment

The present application claims priority from chinese patent application entitled "method and apparatus for contactless three-dimensional gesture unlocking of intelligent interactive device" filed by the chinese patent office on 14/10/2016 under the reference number 201610898260.4, the entire contents of which are incorporated herein by reference.

Technical Field

The present application relates to the field of image recognition, and more particularly, to a three-dimensional gesture unlocking method, a method for acquiring a gesture image, and a terminal device.

Background

Generally, a terminal device is in a locked non-interactive state when unmanned interaction is performed, and when a user needs to operate the terminal device, the terminal device needs to be unlocked in a certain identity authentication mode first, so that the terminal device enters an interactive state. And unlocking the terminal equipment after the user has the right authority so as to ensure the safety of the information of the terminal equipment.

The currently common unlocking mode of the terminal equipment comprises the following steps: digital password unlocking, pattern password unlocking, fingerprint unlocking, iris unlocking and face unlocking. The password is fixed in the digital password unlocking mode, the unlocking can be realized as long as the password is input correctly, and the input password cannot be distinguished, so that anyone can operate and unlock the terminal equipment as long as the owner sees the input process. The unlocking mode of the pattern password also has the defect of immobility, and the pattern password unlocking mode is more visible and easier to be remembered by other people due to the more visual input process. The fingerprint unlocking mode has strong dependence on fingerprints, and the finger is one of the body parts which are most contacted with the outside, so that the fingerprint is easy to be identified by the terminal equipment during unlocking after being changed due to accidental friction, collision and water. Moreover, fingerprint identification belongs to contact measurement, and is insanitary when being touched by a plurality of people, and the influence of the stain is serious. The iris unlocking mode needs to collect iris images by a camera, so that the dependency on environment brightness is high, clear iris images are difficult to obtain when external light is weak, and the accuracy of identification is influenced. Moreover, in the image acquisition process, the eyes of the user need to be located at the optimal imaging position, and the user with smaller eyes needs to open the larger eyes for acquisition, which causes certain complexity. Because the iris is small in area, the requirement on the camera pixels is high, and more complex operation amount is brought while the hardware cost is improved. The face unlocking mode also needs a camera to collect a face image, has high dependence on environment brightness, and is difficult to be applied to an environment with weak external light. Meanwhile, the human face is easily affected by cosmetics, hair, eyes and the like during face recognition, so that the recognition rate is reduced.

Nowadays, interactive terminal equipment is more and more intelligent, and people have higher and higher requirements on interestingness, accuracy and rapidity of the operation of the terminal equipment. Therefore, aiming at the defects of the existing unlocking mode of the terminal equipment, a brand-new unlocking mode with strong interest, high accuracy and good rapidity is needed.

Disclosure of Invention

The application provides a three-dimensional gesture unlocking method, a method for acquiring a gesture image and a terminal device, which can improve user experience.

In a first aspect, a three-dimensional gesture unlocking method is provided, and the method can be applied to a terminal device, and the method includes:

acquiring a current gesture image of a user;

acquiring contour data of a current gesture in the current gesture image according to the current gesture image;

according to the contour data of the current gesture, determining fingertip pixel points of the current gesture and/or finger root pixel points of the current gesture;

determining the characteristic data of the current gesture according to the fingertip pixel point of the current gesture and/or the finger root pixel point of the current gesture;

and unlocking the terminal equipment when the feature data of the current gesture is successfully matched with the feature data of the preset gesture.

According to the three-dimensional gesture unlocking method, the three-dimensional gesture image displayed in the three-dimensional space in front of the camera by the user is obtained in real time, the gesture of the user in the gesture image is extracted, and the purpose of unlocking the terminal device is achieved by matching with the unlocking gesture set by the user in advance. Therefore, a brand-new unlocking mode with strong interestingness, high accuracy and good rapidness is provided for the user.

In a possible implementation manner, the determining the feature data of the current gesture according to the fingertip pixel point of the current gesture and/or the finger root pixel point of the current gesture includes:

determining at least one of the following information in the current gesture according to the fingertip pixel point of the current gesture and/or the finger root pixel point of the current gesture:

the method comprises the following steps of finger number, finger length, fingertip position characteristic data and finger width characteristic data, wherein the fingertip position characteristic data are used for indicating relative positions of fingers, and the finger width characteristic data are used for indicating finger width.

In a possible implementation manner, the obtaining, according to the current gesture image, profile data of a current gesture in the gesture image includes:

determining whether contour extraction needs to be carried out on the current gesture according to the current frame number of the current gesture image;

when the current gesture needs to be subjected to contour extraction, searching contour pixel points of the current gesture until all contour pixel points of the current gesture are searched or the number of the searched pixel points is larger than a preset detection threshold value, so as to obtain contour data of the current gesture.

The preset detection threshold value is set, so that searching of other contour pixel points is reduced under the condition that all finger boundary pixel points are found, and the efficiency is improved.

In one possible implementation, the method further includes:

when determining that contour extraction is not needed to be carried out on the current gesture, determining a predicted fingertip pixel point of the current gesture and a predicted finger root pixel point of the current gesture according to an exponential Average (EMA) algorithm;

and acquiring the contour data of the current gesture according to the predicted fingertip pixel point of the current gesture and the predicted finger root pixel point of the current gesture.

In a possible implementation manner, the determining, according to the contour data of the current gesture, a fingertip pixel point of the current gesture and/or a base pixel point of the current gesture includes:

determining a v-segment arc in the current gesture according to the contour data of the current gesture, wherein v is an integer greater than or equal to 1;

determining a central pixel point of the s-th arc according to a starting pixel point of the s-th arc and an ending pixel point of the s-th arc in the current gesture, and traversing s through an integer in [1, v ];

determining a central vector p of the s-th arc according to a vector m formed by k pixel points before the central pixel point of the s-th arc to the central pixel point of the s-th arc and a vector n formed by k pixel points after the central pixel point of the s-th arc to the central pixel point of the s-th arc, wherein the central vector p bisects a smaller angle of two angles formed by the vector m and the vector n, and k is an integer greater than or equal to 1;

when the distance between the position of the pixel point in the direction pointed by the central vector p and the depth camera is within a preset distance interval, determining the central pixel point corresponding to the central vector p as a fingertip pixel point, or

And when the distance between the position of the pixel point in the direction pointed by the central vector p and the depth camera is not within a preset distance interval, determining the central pixel point corresponding to the central vector p as a finger root pixel point.

In one possible implementation, the determining the length of the finger in the current gesture includes:

determining a first distance between the fingertip pixel point and a finger root pixel point positioned on the left side of the fingertip pixel point and a second distance between the fingertip pixel point and a finger root pixel point positioned on the right side of the fingertip pixel point;

when the absolute value of the difference between the first distance and the second distance is greater than a preset length threshold, determining the projection of the smaller of the first distance and the second distance in the finger direction of the fingertip pixel point as the finger length,

and when the absolute value of the difference between the first distance and the second distance is smaller than or equal to a preset length threshold, determining the average value of the projection of the first distance in the finger direction of the fingertip pixel point and the projection of the second distance in the finger direction of the fingertip pixel point as the finger length.

In one possible implementation, the determining the length of the finger in the current gesture includes:

when there is no finger root pixel point on the right side of the finger tip pixel point, or when there is no finger root pixel point on the left side of the finger tip pixel point,

determining a third distance between the fingertip pixel point and a finger root pixel point located on the left side of the fingertip pixel point, or

Determining a fourth distance between the fingertip pixel point and a finger root pixel point positioned on the right side of the fingertip pixel point;

determining the projection of the third distance in the finger direction of the fingertip pixel point as the finger length, or

And determining the projection of the fourth distance in the finger direction in which the fingertip pixel point is located as the finger length.

In one possible implementation, the fingertip position feature data D_jThe following formula is satisfied:

wherein k is the length of the longest finger in the current gesture, k is greater than 0, n is the number of fingers in the current gesture, n is an integer greater than or equal to 1, p is a fingertip pixel point in the current gesture, x and y are coordinates of the fingertip pixel point in the current gesture image, x and y are real numbers, z is the depth value of the fingertip pixel point in the current gesture image, and z is greater than or equal to 0.

In one possible implementation, the determining the finger width feature data in the gesture image includes:

equally dividing the length of each finger in the current gesture into m +1 parts in the finger direction, wherein m is a positive integer greater than or equal to 0;

for each finger in the current gesture, calculating the width perpendicular to the direction of the finger at the position of the bisection point to obtain m × n absolute width values, wherein n is the number of the fingers in the current gesture, and n is an integer greater than or equal to 1;

calculating the ratio of every two absolute value widths in the m x n absolute width values to obtain characteristic vector values { d) formed by mn (mn-1)/2 relative widths_i，i＝1，2，…，mn(mn-1)/2}；

The feature vector value d_iI ═ 1,2, …, mn (mn-1)/2} is determined as the finger width characteristic data.

In one possible implementation, the method further includes:

when the feature data of the current gesture is not successfully matched with the feature data of the preset gesture, determining whether the unlocking process time is overtime;

and if the unlocking process time is overtime, carrying out screen locking processing on the terminal equipment after a preset time period.

In one possible implementation, before obtaining contour data of a current gesture in the current gesture image according to the current gesture image, the method further includes:

acquiring and presenting candidate gesture images to the user based on the starting setting unlocking gesture operation of the user;

and setting the candidate gesture in the candidate gesture image as a preset gesture based on the determined operation of the user.

In one possible implementation, before the setting the candidate gesture in the candidate gesture image to be the preset gesture based on the determination operation of the user, the method further includes:

determining whether the number of fingers of the candidate gesture is greater than or equal to 3;

if the number of the fingers of the candidate gesture is greater than or equal to 3, setting the candidate gesture as the preset gesture based on the determined operation of the user;

wherein, the method also comprises:

and acquiring the characteristic data of the candidate gesture.

In a second aspect, a method for acquiring a gesture image is provided, including:

acquiring a current gesture image of a user;

acquiring contour data of a current gesture in the current gesture image according to the current gesture image;

according to the contour data of the current gesture, determining fingertip pixel points of the current gesture and/or finger root pixel points of the current gesture;

and determining the characteristic data of the current gesture according to the fingertip pixel point of the current gesture and/or the finger root pixel point of the current gesture.

According to the method for acquiring the gesture image, the fingertip pixel point of the current gesture and/or the finger root pixel point of the current gesture can be acquired according to the contour data of the current gesture in the current gesture image, and therefore the feature data of the current gesture can be determined.

In a possible implementation manner, the determining the feature data of the current gesture according to the fingertip pixel point of the current gesture and/or the finger root pixel point of the current gesture includes:

determining at least one of the following in the current gesture according to the fingertip pixel point of the current gesture and/or the finger root pixel point of the current gesture:

the method comprises the following steps of finger number, finger length, fingertip position characteristic data and finger width characteristic data, wherein the fingertip position characteristic data are used for indicating relative positions of fingers, and the finger width characteristic data are used for indicating finger width.

In a third aspect, a terminal device is provided, configured to perform the method in the first aspect or any possible implementation manner of the first aspect. In particular, the terminal device may comprise means for performing the method of the first aspect or any possible implementation manner of the first aspect.

In a fourth aspect, a terminal device is provided, configured to perform the method of the second aspect or any possible implementation manner of the second aspect. In particular, the terminal device may comprise means for performing the method of the second aspect or any possible implementation manner of the second aspect.

In a fifth aspect, there is provided a terminal device, comprising a memory for storing a computer program, a processor for calling and executing the computer program from the memory, and a display, wherein when the program is executed, the processor performs the method of the first aspect or any possible implementation manner of the first aspect.

A sixth aspect provides a terminal device, comprising a memory for storing a computer program, a processor for calling and executing the computer program from the memory, and a display, wherein when the program is executed, the processor executes the method of the second aspect or any possible implementation manner of the second aspect.

In a seventh aspect, a computer-readable medium is provided for storing a computer program comprising instructions for performing the method of the first aspect or any possible implementation manner of the first aspect.

In an eighth aspect, there is provided a computer readable medium for storing a computer program comprising instructions for performing the method of the second aspect or any possible implementation of the second aspect.

Drawings

Fig. 1 is a schematic diagram of a minimum hardware system of a terminal device implementing the three-dimensional gesture unlocking method of the present application.

Fig. 2 is a schematic flow chart of a three-dimensional gesture unlocking method according to an embodiment of the application.

Fig. 3 is a schematic diagram of search directions when searching contour pixel points according to an embodiment of the present application.

FIG. 4 is a diagram illustrating an example of detecting the order of contour pixels according to an embodiment of the present application.

FIG. 5 is a diagram illustrating a method for determining a finger tip pixel and a finger root pixel according to an embodiment of the present application.

Fig. 6 is a schematic flow chart of an embodiment of a set preset gesture in a three-dimensional gesture unlocking method according to the present application.

FIG. 7 is a schematic flow chart diagram illustrating one embodiment of a three-dimensional gesture unlocking method in accordance with the present application.

FIG. 8 is a schematic flow chart diagram of a method of obtaining a gesture image according to an embodiment of the present application.

Fig. 9 is a schematic block diagram of an example of a terminal device according to an embodiment of the present application.

Fig. 10 is a schematic block diagram of another example of a terminal device according to an embodiment of the present application.

Fig. 11 is a schematic block diagram of still another example of a terminal device according to an embodiment of the present application.

Detailed Description

Embodiments of the present application are described below with reference to the accompanying drawings.

Aiming at the defects existing in the conventional equipment unlocking mode, the depth camera is added to the terminal equipment, the three-dimensional gesture image presented by the user in the three-dimensional space in front of the depth camera is obtained in real time, the gesture of the user in the gesture image is extracted, and the purpose of unlocking the terminal equipment is achieved by matching with the unlocking gesture set by the user in advance. Therefore, a brand-new unlocking mode with strong interestingness, high accuracy and good rapidness is provided for the user.

The terminal device of the embodiments of the present application may be an access terminal, a User Equipment (UE), a subscriber unit, a subscriber station, a mobile station, a remote terminal, a mobile device, a user terminal, a wireless communication device, a user agent, or a user equipment. The terminal device may be a cellular phone, a cordless phone, a Session Initiation Protocol (SIP) phone, a Wireless Local Loop (WLL) station, a Personal Digital Assistant (PDA), a handheld device having wireless communication capabilities, a computing device or other processing device connected to a wireless modem, a vehicle mounted device, a wearable device, or the like.

Fig. 1 is a schematic diagram of a minimum hardware system 100 of a terminal device for implementing the three-dimensional gesture unlocking method of the present application. The system 100 shown in FIG. 1 includes: light source emitter 110, depth camera 120, spectral analysis module 130, color camera 140, central processor 150, touch screen 160, non-volatile memory 170, and memory 180.

The color camera 140, the light source emitter 110 and the depth camera 120 constitute a spectral input module, and the spectral analysis module 130 constitutes an image generation module. The light source emitter 110, color camera 140, and depth camera 120 may be mounted side-by-side above the device (e.g., in a central location directly above the device). The light source emitter 110 may be an infrared emitter, the depth camera 120 may be an infrared camera, and the spectral analysis module 130 may be an infrared spectral analysis module. In this case, light source emitter 110 works in conjunction with depth camera 120 to project a scene with infrared light encoded images. The light source emitter 110 outputs a common laser light source, and forms near-infrared light after being filtered by the ground glass and the infrared filter. The light source emitter 110 may continuously and comprehensively output infrared light having a wavelength of 840 nanometers (nm).

The depth camera 120 is a Complementary Metal Oxide Semiconductor (CMOS) image sensor, and is configured to receive an excitation light source, such as infrared light, reflected from the outside, and digitally encode the excitation light source to form a digital image, and transmit the digital image to the spectral analysis module 130. The spectrum analysis module 130 analyzes the speckles, calculates the distance between the corresponding pixel point of the image and the depth camera 120, and forms a depth data matrix for the driver to read.

The central processing unit 150 is used for gesture analysis, unlocking behavior determination and peripheral control. The non-volatile memory 170 is used to save program files, system files, and gesture feature information. The memory 180 is used for system and program operation caching. The touch screen 160 is used for interaction with a user. Specifically, the central processor 150 reads the depth data and extracts the user gesture feature data. When the user sets the unlocking gesture, the gesture characteristics are analyzed, the recognized user gesture is processed and then displayed on the touch screen 160, and when the user determines that the gesture displayed on the touch screen 160 is set to be the unlocking gesture, the confirmation key is pressed. Meanwhile, the central processing unit 150 further extracts the gesture feature value and stores it in the nonvolatile memory 170. And acquiring gesture data in real time when a user needs to verify a gesture to unlock the screen, extracting gesture features, comparing the extracted gesture features with the stored set gesture feature data, if the extracted gesture features are matched with the stored set gesture feature data, unlocking successfully, and otherwise, unlocking fails.

Hereinafter, a three-dimensional gesture unlocking method of a terminal device according to an embodiment of the present application will be described in detail.

FIG. 2 is a schematic flow chart diagram of a three-dimensional gesture unlocking method according to one embodiment of the present application. The method shown in fig. 2 is performed by a terminal device, for example, may be performed by the terminal device shown in fig. 1.

S210, acquiring a current gesture image of the user.

In particular, acquiring the current gesture image may be understood as acquiring the current depth image. Herein, a depth image is also called a range image (range image), and refers to an image in which distances (depths) from an image collector (e.g., a depth camera 120 in the present application) to various points in a scene are taken as pixel values, and directly reflects the geometric shape of a visible surface of a scene.

For example, when the method is executed by the terminal device shown in fig. 1, the depth camera 120 receives an excitation light source reflected from the outside, such as infrared light, and digitally encodes the excitation light source to form a digital image and transmits the digital image to the spectrum analysis module 130. The spectrum analysis module 130 analyzes the speckles, and calculates a distance z between a corresponding pixel point (x, y) in the current gesture image and the depth camera 120, so that the current gesture image can be obtained.

S220, acquiring the contour data of the current gesture in the current gesture image according to the current gesture image.

And S230, determining fingertip pixel points of the current gesture and/or finger root pixel points of the current gesture according to the contour data of the current gesture.

S240, determining the characteristic data of the current gesture according to the fingertip pixel point of the current gesture and/or the finger root pixel point of the current gesture.

Specifically, the steps S220 to S240 are used for recognizing the current gesture of the user according to the current gesture image.

For example, the profile data described herein may refer to coordinates (x, y, z) of various pixel points in the gesture image that constitute the gesture profile. Optionally, according to the current frame number, the contour data of the current gesture may be acquired by using the first mode or the second mode.

Specifically, the embodiment of the present application may employ an acceleration recognition strategy, and for each 1 st frame of the L frames (for example, each 5 frames) of the gesture images, the first way is used to acquire the contour data of the gesture in the gesture image, and for the remaining L-1 frames (for example, the remaining 4 frames) of the gesture images, the second way is used to acquire the contour data of the gesture in the gesture image.

That is, if the current gesture image is the 1 st frame of every L frames, the contour data of the current gesture is obtained by the first method; and if the current gesture image is any one of the rest L-1 frames in each L frame, acquiring the contour data of the current gesture by the mode two.

Next, a method for acquiring the contour data of the current gesture in the first and second ways will be described in detail.

The first method is as follows: and carrying out complete contour extraction on the current gesture.

The contour extraction mainly comprises the steps of finding out coordinates of pixel points at the edge of a human hand in a depth image in a current gesture, packaging the coordinates of the pixel points into a structural body, and sequentially recording the structural body into an ordered linked list for use in subsequent steps.

Specifically, contour pixel points in the current gesture are searched until all contour pixel points in the current gesture are searched, so that contour data in the current gesture are obtained. Or searching contour pixel points in the current gesture, and stopping searching until the number of the searched pixel points is larger than a preset detection threshold value, so that contour data in the current gesture are obtained.

As an example and not by way of limitation, when searching for contour pixel points in a current gesture, noise reduction processing may be performed on a current gesture image first; then, acquiring a first hand contour pixel point of the current gesture in the current gesture image subjected to the noise reduction processing; and then acquiring other hand contour pixel points in the current gesture image after the noise reduction treatment according to the first hand contour pixel point until the searching is finished.

(1) Image noise reduction

The current gesture image in the application can be a depth image, some noises can be generated in the process of obtaining the depth image, and the source of the noise generation is divided into an inner part and an outer part. Internal noise may be caused by photoelectric fundamental properties, by mechanical vibrations, or by component defects. The external noise may be caused by electromagnetic crosstalk, by impurities in the air, or by impurities attached to the hands. These noises appear as salt and pepper noises and random noises on the image.

For salt and pepper noise which is most likely to be generated in the depth image, the noise interference in the image can be reduced by adopting a median filtering mode in the embodiment of the application. As shown in formula (1), med represents a median operation, and the filtering method is to sort the sizes of pixels in the sliding filtering window 2I +1, and the median of the sequence is assigned to the current pixel value. I is a positive integer greater than or equal to 1, x_kRepresenting a pixel point.

x_k＝med{x_k-I，x_k-I+1，…x_k，…x_k+I} (1)

It should be understood that the method adopted when performing image noise reduction is not limited in the embodiments of the present application. For example, the present application may also perform denoising processing on the depth image by using a mean filtering method, a wavelet filtering method, and the like. In particular, the noise reduction processing mode can refer to the prior art, and is not described in detail here.

(2) Searching for first contour pixel point in current gesture

Firstly, judging whether the acquired gesture image, namely the current gesture image has a gesture. In other words, it is determined whether the current gesture exists. If a pixel point with the distance from the depth camera within a preset distance interval exists in the current gesture image, and the distance from at least one adjacent pixel point above, below, left or right to the depth camera exceeds the preset distance interval, the gesture is considered to exist in the current gesture image.

It should be understood that the preset distance interval may be set by the user when setting the unlocking gesture phase, and may also be defined by the system. The upper and lower limit values of the preset distance interval may be defined or set according to the distance value between most user gestures and the depth camera. For example, the distance interval may be 20cm to 40 cm. The lower limit of 20cm is selected according to the wide angle of the depth camera and the size of a common human hand, and the purpose is to enable the depth image of the human hand to be acquired by the camera. The upper limit of 40cm is selected in consideration of the fine degree of data acquired by the hand after the distance is far, and the range of the available distance of the user is widened as much as possible on the premise of ensuring accurate acquisition of the fine data of the hand.

When determining whether a gesture exists in the current gesture image, for example, it may be determined whether a distance between a central position pixel point of the current gesture image, that is, a central position pixel point of the current frame, and the depth camera is within the preset distance interval. If the distance between the central part of the depth image and the depth camera is not in the preset distance interval, sequentially detecting the left direction, the right direction, the upper direction, the lower direction, the upper left direction, the upper right direction, the lower left direction and the lower right direction by taking the distance of a plurality of (for example 10) pixel points as intervals. When two adjacent pixel points in the preset distance interval are found, the detection is finished, and the first human hand contour pixel point is found. And if two adjacent pixel points in the preset distance interval are not found, the current gesture image is indicated to have no gesture.

In the above steps, after two adjacent pixels in the preset distance interval are found, the first contour pixel can be found from the effective pixel, that is, any one of the two pixels is searched to the left.

For example, the effective pixel point may be searched to the left, and whether the depth value of the current pixel point is at the boundary point may be determined in a binary search manner. The method for judging whether the depth value of the current pixel point is at the boundary point is that the distance between the current pixel point and the depth camera is in a preset distance interval, and the distance between the pixel point adjacent to the current pixel point and the depth camera is not in the preset distance interval. Here, adjacent means a pixel above, below, to the left, or to the right of the current pixel.

When searching is carried out in a half-folding searching mode, if the distance between the current pixel point and the depth camera is in a preset distance interval, indicating that the pixel point of the contour of the human hand is on the left side of the pixel point, continuing half-folding searching to the left; if the distance between the current pixel point and the depth camera is not within the preset distance range, the fact that the contour pixel point of the human hand is on the right of the pixel point is indicated, and then the search is performed by half to the right. And if the pixel point at the boundary point is found, recording the position of the pixel point and storing the position, wherein the pixel point is the first contour pixel point, and tracking and detecting other contour pixel points of the current gesture. If the pixel point at the boundary point is not found, the current gesture is too large, and the frame of gesture image, namely the current gesture image, does not meet the requirements, so that the processing is not continued.

It should be understood that the binary search method is only one specific example of searching for the contour pixel point, and the binary search method can improve the search efficiency, but the embodiment of the present application does not specifically limit the manner of searching for the contour pixel point.

(3) Tracking other contour pixel points of detecting current gesture

After the first contour pixel point is found, other contour pixel points close to the first contour pixel point can be tracked and detected. In this process, first, the seek direction is determined.

The present application may define four search directions, for example as shown in fig. 3, to speed up tracking.

As shown in fig. 3, the gray boxes indicate the currently valid pixels (contour pixels), fig. 3 (a) shows a search to the upper left, fig. 3 (b) shows a search to the upper right, fig. 3 (c) shows a search to the lower left, and fig. 3 (d) shows a search to the lower right. The numbers in the figure represent the order of detection of adjacent pixel points. Because the finger boundaries are mostly distributed in the vertical direction, the search in the vertical direction is enhanced by the four search modes. And for one pixel point, searching is started according to the searching direction of the previous pixel, searching to the upper right when the upper left searching is not available, searching to the lower right when the upper right searching is not available, searching to the upper right when the lower right searching is not available, and trying to search from all directions when the lower right searching is not available.

If the pixel point recorded before is found, the direction is changed to the direction corresponding to the found previous pixel point, and the counter-clockwise searching is carried out. The method for judging whether a pixel point is stored is to maintain a hash table, add the valid point into the hash table by taking the position value as key every time the valid point is found, and then judge whether the pixel point is found by using the position parameter.

If the pixel point recorded before is still found, whether the current pixel point is on the pixel line with the vertical or horizontal edge is judged. If the contour pixel point is not located at the pixel line, all contour pixel points are searched. If the pixel point right above and below the pixel point is not the effective point, the pixel point is in a horizontal single pixel line; and if the pixel point on the right left side and the pixel point on the right side of the pixel point are not valid points, the pixel point is in a vertical single pixel line. The case of a single pixel line is processed next.

As shown in (a) of fig. 4, the 5 th pixel point can be detected after the 6 th pixel point is detected, and the pixel point has been detected, and the pixels above and below are not effective pixel points, then the 6 th pixel point is on the horizontal single pixel line, then to the right-hand detection of 5 positions.

As shown in fig. 4 (b), when the 4 th pixel is detected after the 5 th pixel is detected, the pixel has been detected, and the pixels on the left and right sides are not valid pixels, the 5 th pixel is on the vertical single pixel line, and the detection is performed to the upper side of the 4 th position.

When the search end condition is satisfied, the search operation is ended.

The condition of ending the search may be that all contour pixel points of the current gesture in the current gesture image are searched, or that the number of the searched pixel points is greater than a preset detection threshold. That is, if all contour pixel points in the current gesture image are searched, the search operation is ended. Or, when the number of the searched pixel points is greater than a preset detection threshold, for example, the number of the searched pixel points is greater than 750, the search operation is ended.

The upper limit of the detection number of the effective points is limited, namely, a preset detection threshold value is set, so that the searching of other contour pixel points is reduced under the condition that all finger boundary pixel points are found, and the efficiency is improved.

The gesture image in the embodiment of the present application may be a 640 × 480 resolution depth map, and for the 640 × 480 resolution depth map, the preset detection threshold may be set to 750.

The second method comprises the following steps: the current gesture is not subjected to a complete contour extraction.

By way of example and not limitation, a predicted fingertip pixel point and a predicted finger root pixel point of a current gesture in a current gesture image can be determined through an exponential average EMA algorithm; and then, acquiring the contour data of the current gesture according to the predicted fingertip pixel point and the predicted finger root pixel point.

For example, the EMA algorithm may be used to perform position identification of fingertips and fingerroots on the current gesture without performing complete contour extraction, and perform position prediction on each fingertip point and fingerroot point in the current frame by using equation (2). In the formula (2), s_t+1Is the predicted position at time t +1, s_tIs the predicted position at time t, o_tIs the actual position at time t and ω is a weighting factor (0 < ω < 1).

s_t+1＝ωo_t+(1-ω)s_t(2)

The method needs a series of predicted values, the embodiment of the application can maintain an ordered linked list of the number of the finger points, the finger root points and the fingers in a plurality of frames (for example, 10 frames) of recent images, and when the predicted position of a certain point in the next frame of image needs to be calculated, the position information of the corresponding point in all the plurality of frames (for example, 10 frames) of images is input to the EMA algorithm.

For example, if an ordered linked list of the number of fingers, finger tips, finger roots and fingers in an image of approximately 10 frames is maintained, 10 is used to define the value 2/(10+1) of the coefficient ω, ω. The positions of the finger tip and the finger root in the 1 st frame image in the 10 frame images are the actual positions.

After the predicted fingertip pixel points and the predicted finger root pixel points are calculated, contour pixel points around the predicted fingertip pixel points and the predicted finger root pixel points can be obtained, and therefore contour data of the gesture image is obtained. For example, the n-dimensional matrix can be used as a range to detect the pixel points in the gesture outline in the matrix, then the pixel points of other gesture outlines with a certain number are respectively tracked forwards and backwards, and the tracked points are added into the outline ordered linked list, so that the outline ordered linked list around the fingertip pixel point and the finger root pixel point can be obtained on the premise of not carrying out a complete outline extraction step.

After the contour data of the current gesture is acquired through the first mode or the second mode, a fingertip pixel point of the current gesture and/or a finger root pixel point of the current gesture can be further determined, and finally the feature data of the current gesture is acquired.

Optionally, as an embodiment of the present application, a fingertip pixel point of the current gesture and/or a finger root pixel point of the current gesture may be determined through the following steps:

determining v arc sections in the current gesture according to the contour data of the current gesture, wherein v is an integer greater than or equal to 1;

determining a central pixel point of the s-th arc according to a starting pixel point of the s-th arc and an ending pixel point of the s-th arc in the current gesture, and traversing s through an integer in [1, v ];

determining a central vector p of the s-th arc according to a vector m formed by k pixel points from the central pixel point of the s-th arc to the central pixel point of the s-th arc and a vector n formed by k pixel points from the central pixel point of the s-th arc to the central pixel point of the s-th arc, wherein the central vector p bisects a smaller angle of two angles formed by the vector m and the vector n, and k is an integer greater than or equal to 1;

when the distance between the position of the pixel point in the direction pointed by the central vector p and the depth camera is within a preset distance interval, determining the central pixel point corresponding to the central vector p as a fingertip pixel point, or

And when the distance between the position of the pixel point in the direction pointed by the central vector p and the depth camera is not within the preset distance interval, determining the central pixel point corresponding to the central vector p as a finger root pixel point.

It should be understood that the v-arc is represented as the number of arcs in the current gesture is v.

Specifically, first, the arc in the current gesture may be obtained first through a K-curvature algorithm. The K curvature algorithm defines a constant K for describing the distance between the points, and an angle a for describing the angle formed by the radian and the pixel points on the two sides. For example, k may be defined as 25 degrees, and the angle a may be defined as 50 degrees, which is not particularly limited in the embodiments of the present application. For each pixel point in the contour ordered linked list, two intersecting vectors m and n can be constructed. The starting points of the two vectors are current pixel points, the end point of the vector m is the kth pixel point before the current pixel point, and the end point of the vector n is the kth pixel point after the current pixel point. And then, calculating an included angle between the two vectors, and marking the current pixel point as a radian point when the included angle is smaller than the angle a. In this way, all the arc points in the current gesture can be obtained, with adjacent arc points forming an arc.

In addition, in the first method, that is, in the complete contour extraction process, when the first contour pixel point is located at the edge of the front arm and is far away from the first finger, due to the limitation of the number of detected valid points, the situation that the valid points are terminated because the valid points reach the upper limit when all fingers in the current gesture are not detected may be caused. In order to avoid this situation, in this embodiment of the application, after the first arc line point is extracted, a difference value between the first arc line point and the first human contour pixel point may be calculated, and if it is determined that the difference value is too large, for example, larger than a certain preset value, the number of valid point detections, that is, the preset detection threshold value is corrected, and the preset detection threshold value is increased, so that the detection of the contour pixel points can be continuously tracked, and all detectable contour pixel points can be extracted.

And then, identifying the fingertip and the finger root, and determining the fingertip pixel point and the finger root pixel point.

The specific method is that the contour pixel point chain table is traversed, and an arc starting pixel point and an arc ending pixel point are searched. Due to the central symmetry of the finger tip and the finger root arc, the central point of the arc is set as a finger tip pixel point or a finger root pixel point.

For example, as shown in fig. 5, a central vector p of two intersecting vectors m, n, m, n of the central point of the arc is taken, and if the distance between the pixel point in the direction pointed by the vector p and the depth camera exceeds a preset distance interval, the central point is the central point at the bottom of the finger, i.e., the pixel point at the finger root, otherwise, the central point is the pixel point at the fingertip. Where the finger direction is described by a vector p.

Because the gesture change has diversity and rapidity, in order to accurately detect the fingertip pixel points and/or the finger root pixel points in real time, optionally, kalman filtering may be used to predict each fingertip pixel point and/or each finger root pixel point so as to determine the accurate position of the fingertip or the finger root.

First, a state vector x of a fingertip at time t is defined_tAs shown in formula (3). x, y refer to the coordinates of the cusp in the image, v_x、v_yThe moving speed of the sharp point on the x and y axes is referred to.

Estimation x of fingertip state vector at time t +1_t+1As shown in formula (4). The state transition matrix a is defined as equation (5), and the noise variation matrix B is defined as equation (6). w is a_tIs the noise in the estimation process, obeys the Gaussian distribution N (0, Q)_t) The mean value is 0, Q is defined as equation (7), and the last two 8 on the diagonal line represent the magnitude of the change speed of the fingertip point coordinates.

x_t+1＝Ax_t+Bw_t(4)

The definition of the fingertip identification result calculated in the previous step at the time t is shown as formula (8), the calculation matrix C is defined as formula (9), and v is_tIs the error of the calculated result from the true position, and also obeys the Gaussian distribution N (0, Q)_t) The average value is 0. w is a_t、v_tRespectively are

Wherein I_2×2Is a 2 x 2 identity matrix.

y_t＝Cx_t+v_t(8)

The kalman filter used in the present invention is defined next. Filter gain equation K_tDefined as equation (10), represents the weight of the difference between the calculated value and the estimated value. Equation of state one-step prediction

Is equivalent to

Defined as formula (11) and represents y_t-1And (5) predicting the value. One step prediction equation for mean square error

Is equivalent to

Is defined as formula (12) and represents

Is equal to BB^T. By

Result of calculation of (2)

The exact position (x, y) of the fingertip can be obtained.

Therefore, more accurate fingertip pixel points and/or finger root pixel points of the current gesture can be obtained.

And then, determining the characteristic data of the current gesture according to the fingertip pixel point of the current gesture and/or the finger root pixel point of the current gesture.

Illustratively, the feature data of the current gesture mainly includes at least one of the following information in the current gesture: the number of fingers, the length of the fingers, the characteristic data of the positions of the fingertips and the characteristic data of the widths of the fingers. Wherein the fingertip position feature data is used for indicating relative positions of fingers, and the finger width feature data is used for indicating finger widths.

In other words, after obtaining the fingertip pixel point of the current gesture and/or the finger root pixel point of the current gesture, at least one of the following in the current gesture may be determined: the number of fingers, the length of the fingers, the characteristic data of the positions of the fingertips and the characteristic data of the widths of the fingers.

In the embodiment of the application, after the fingertip pixel points of the current gesture are obtained according to the method, the number of fingers in the current gesture can be determined. That is, there are several fingertip pixel points, which means that there are several fingers in the current gesture.

In the embodiment of the application, in order to improve the detection precision, the number of the fingers can be set to be 3-5. Then, when the number of the fingers in the current gesture is determined to be less than 3, the current gesture image is abandoned, and the gesture image of the next frame is continuously identified. And if the number of the fingers in the current gesture is determined to be greater than or equal to 3, continuously identifying the finger length, the finger tip position characteristic data and the finger width characteristic data in the current gesture.

In the following, how to determine the finger length, the fingertip position feature data, and the finger width feature data will be described in detail.

Optionally, when determining the length of the finger in the current gesture, if finger root pixel points exist on both the left and right sides of the fingertip pixel point, first determining a first distance between the fingertip pixel point and the finger root pixel point located on the left side of the fingertip pixel point, and a second distance between the fingertip pixel point and the finger root pixel point located on the right side of the fingertip pixel point. When the absolute value of the difference between the first distance and the second distance is larger than a preset length threshold, determining the projection of the smaller of the first distance and the second distance in the finger direction where the fingertip pixel point is located as the finger length; or when the absolute value of the difference between the first distance and the second distance is smaller than or equal to a preset length threshold, determining the average value of the projection of the first distance in the finger direction where the fingertip pixel point is located and the projection of the second distance in the finger direction where the fingertip pixel point is located as the finger length. And if the right side of the fingertip pixel point does not have the finger root pixel point, determining a third distance between the fingertip pixel point and the finger root pixel point positioned on the left side of the fingertip pixel point, and determining the projection of the third distance in the finger direction of the fingertip pixel point as the finger length. If the finger root pixel point does not exist on the left side of the fingertip pixel point, a fourth distance between the fingertip pixel point and the finger root pixel point located on the right side of the fingertip pixel point needs to be determined, and the projection of the fourth distance in the finger direction where the fingertip pixel point is located is determined as the finger length.

It should be understood that the first distance, the second distance, etc. described above are merely to distinguish the distance between different objects.

In this manner, the length of all recognized fingers in the current gesture may be determined.

It should be understood that the preset length threshold may be preset by the terminal device system.

Optionally, the characteristic data D is used in determining the fingertip position_jThen, it can be determined according to the following formula:

wherein k is the length of the longest finger in the current gesture, k is greater than 0, n is the number of fingers in the current gesture, n is an integer greater than or equal to 1, p is a fingertip pixel point in the current gesture, x and y are coordinates of the fingertip pixel point in the current gesture image, z is the depth value of the fingertip pixel point in the current gesture image, and z is greater than or equal to 0.

Specifically, p is a fingertip pixel point in the current gesture, and the x and y values and the corresponding depth value z in the image of the fingertip pixel point constitute the coordinates of the point. d (p)_i，p_j) The relationship between the positions of two pixel points in the image is defined as equation (14). k is the length of the longest finger detected, p_i，p_jThe closer the two points are, d (p)_i，p_j) The smaller will be. D_jThe fingertip position characteristic used for describing the current gesture, i.e. the sum of the position relations of every two fingertips, is defined as equation (13).

The numerical values on the right side of the equation in the equation (14) can be obtained as described earlier in the present application, and therefore, the fingertip position characteristic data D can be obtained here in the equation (13)_j。

Optionally, when determining the finger width feature data in the gesture image, the length of each finger in the current gesture may be first divided into m +1 parts on average in the finger direction, where m is an integer greater than or equal to 0. For each finger in the current gesture, the width perpendicular to the direction of the finger at the position of the bisection point is calculated, and m × n absolute width values are obtained. Wherein n is the number of fingers in the current gesture, and n is an integer greater than or equal to 1. Then, the ratio of every two absolute value widths in the m x n absolute width values is calculatedObtaining the characteristic vector value { d) consisting of mn (mn-1)/2 relative widths_iI is 1,2, …, mn (mn-1)/2 }. The feature vector value d_iI ═ 1,2, …, mn (mn-1)/2} is the finger width feature data.

And S250, unlocking the terminal equipment when the feature data of the current gesture is successfully matched with the feature data of the preset gesture.

It should be understood that the preset gesture is set by the user during the gesture setting stage. And setting unlocking fingerprints in a fingerprint setting stage in the preset gesture similar to fingerprint unlocking in the gesture setting stage. The preset gesture may be a plurality of unlocking gestures set in the gesture setting stage, that is, one unlocking gesture in the plurality of preset gestures, or only one unlocking gesture set in the gesture setting stage.

Optionally, when the user sets the preset gesture, the terminal device may trigger the light source emitter to output the excitation light source based on the user's start setting unlocking gesture operation. Candidate gesture images are then captured by the depth camera and presented to the user. And finally, setting the candidate gesture in the candidate gesture image as a preset gesture based on the determination operation of the user.

For example, a User clicks on a User Interface (UI), such as a button to initiate a set unlock gesture, which triggers the light source transmitter to output an excitation light source, the depth camera to collect depth data, and the depth image, i.e., a candidate gesture image, is presented to the User. The key that determines the gesture to be set is pressed if the user is satisfied with the candidate gesture image. And the terminal equipment sets the candidate gesture in the candidate gesture image as a preset gesture.

Further, before setting the candidate gesture in the candidate gesture image to the preset gesture based on the determination operation of the user, the method may further include: and determining whether the number of fingers of the candidate gesture is greater than or equal to 3, if so, setting the candidate gesture as a preset gesture based on the determination operation of the user, and acquiring the characteristic data of the candidate gesture.

That is, the number of fingers in the candidate gesture must be greater than or equal to 3. Otherwise, the current candidate gesture is abandoned, and the next frame of candidate gesture image is collected continuously.

After the preset gesture is set in the gesture setting stage, the terminal device stores the characteristic data of the preset gesture.

The feature data of the preset gesture may be, for example, the number of fingers, the length of the fingers, the position feature data of the fingertip, and the width feature data of the fingers in the preset gesture.

In this case, as an embodiment of S250, when the number of fingers, the finger length, the fingertip position characteristic data, and the finger width characteristic data in the current gesture respectively match the number of fingers, the finger length, the fingertip position characteristic data, and the finger width characteristic data of the preset gesture, the terminal device may be unlocked.

For example, the number of fingers, the finger length, the fingertip position characteristic data, and the finger width characteristic data in the current gesture are respectively matched with the number of fingers, the finger length, the fingertip position characteristic data, and the finger width characteristic data in the preset gesture, which may mean that the number of fingers in the current gesture is equal to the number of fingers in the preset gesture, an absolute value of a difference between each finger length in the current gesture and each corresponding finger length in the preset gesture is less than or equal to a first preset threshold, a difference between the finger position characteristic data in the current gesture and the fingertip position characteristic data in the preset gesture is less than or equal to a second preset threshold, and a euclidean distance Dis between the finger width characteristic data in the current gesture and the finger width characteristic data in the preset gesture is less than or equal to a third preset threshold. And then, when the current gesture meets the condition, unlocking the terminal equipment.

Wherein, finger width characteristic data { d) in the preset gesture_jJ-1, 2, …, mn (mn-1)/2 can be determined by comparing the finger width feature data { d } in the current gesture_iI ═ 1,2, …, mn (mn-1)/2} was obtained in the same manner. That is, the length of each finger in the preset gesture is first divided equally into m +1 in the finger direction. For each finger in the preset gesture, calculating the direction perpendicular to the finger at the position of the bisectorAnd (4) obtaining m × n absolute width values according to the width. And n is the number of fingers in the preset gesture. Then, the ratio of every two absolute value widths in the m x n absolute width values is calculated to obtain characteristic vector values { d) formed by mn (mn-1)/2 relative widths_i，i＝1，2，…，mn(mn-1)/2}。

The euclidian distance Dis of the difference between the finger width characteristic data in the current gesture and the finger width characteristic data in the preset gesture satisfies equation (15):

and in step S250, when the feature data of the current gesture is successfully matched with the feature data of the preset gesture, unlocking the terminal device. If the feature data of the current gesture is not successfully matched with the feature data of the preset gesture, as an embodiment of the present application, it may be determined whether the unlocking process time is overtime. If the unlocking process time is over, for example, a Graphical User Interface (GUI) prompts the unlocking process time to be over, and the screen is locked after a preset time period. In addition, the light source emitter and the depth camera can be turned off when the time of the unlocking process is over. And if the unlocking process time is not overtime, the terminal equipment continues to identify the next frame of image.

Therefore, according to the three-dimensional gesture unlocking method provided by the embodiment of the application, the three-dimensional gesture image presented by the user in the three-dimensional space in front of the camera is obtained in real time, the gesture of the user in the gesture image is extracted, and the purpose of unlocking the terminal device is achieved by matching with the unlocking gesture set by the user in advance. Therefore, a brand-new unlocking mode with strong interestingness, high accuracy and good rapidness is provided for the user.

In order to more clearly illustrate the manner of setting the preset gesture according to the embodiment of the present application, an embodiment of setting the preset gesture in the three-dimensional gesture unlocking method according to the embodiment of the present application is described in detail below with reference to fig. 6.

It should be noted that fig. 6 and fig. 7 to be described below are both described by taking the light source emitter as an infrared emitter and the depth camera as an infrared camera as an example.

In order to ensure the quality of the unlocking gesture and reduce the time set by the gesture, when the gesture is preset, the three-dimensional gesture unlocking method for the terminal device can prompt that the hand of the user is located in the middle of the image, and the unlocking gesture contains 3-5 fingers.

S601, the terminal device detects a first operation.

Specifically, the first operation may be an operation in which the user clicks a button that sets an unlocking gesture on the UI interface. If the user clicks a button for setting an unlocking gesture on the UI interface, the terminal may detect a first operation.

And S602, triggering the infrared emitter and the infrared camera to work.

Specifically, if the terminal device detects a first operation, namely if a user clicks a button operation for setting an unlocking gesture on a UI (user interface), the terminal device triggers the infrared transmitter and the infrared camera to work.

And S603, acquiring the current gesture image in real time.

Specifically, the infrared spectrum analysis chip acquires depth data of an image acquired by the infrared camera in real time.

And S604, the terminal equipment judges whether global tracking is needed according to the current frame number, namely, the first mode described above is executed, and complete contour extraction is performed on the current gesture. If necessary, S605 is executed, otherwise S607 is executed.

And S605, the terminal equipment extracts the contour.

Specifically, the terminal device may acquire the contour data of the frame image in the manner described above.

And S606, the terminal equipment judges whether the contour is extracted successfully, namely whether a gesture exists in the frame of image and whether the gesture is too large. If the frame of image has a gesture and a gesture contour point exists, it can be judged that contour extraction is successful.

And S607, the terminal equipment performs fingertip identification.

Specifically, if the contour extraction is successful, the feature data of the gesture is recognized next, and the recognition of the fingertip is performed first. Or in S604, the terminal device determines that global tracking is not needed according to the current frame number, that is, according to the second method described above, performs fingertip position prediction without performing complete contour extraction on the current gesture, and finally obtains a gesture contour linked list similar to the contour extraction according to the obtained predicted fingertip pixel point and predicted finger root pixel point.

S608, the contour detection upper limit value is corrected based on the detected first fingertip position information.

Specifically, the step may refer to the above description, and is not repeated herein for brevity. If the contour detection upper limit value needs to be corrected, S605 is executed after the contour detection upper limit value is corrected.

And S609, converting the data format.

Since the data identifying the fingertip position is only the coordinates of the fingertip in the image, in order to show the fingertip position to the user, the terminal device can change the color data of the fingertip and the surrounding part of the pixel points to be viewed by the user. In addition, the terminal device can also change the color data of the gesture outline. S610, the terminal device fuses the fingertip positions into image data and submits the image data to the display control.

For example, the terminal device may change color values of pixel points in a corresponding region in the color image by using the fingertip position coordinate as a central point and using 10 pixels as a radius circle region, and may also change color values of corresponding positions of the gesture contour pixel points and the pixel points adjacent to the gesture contour pixel points in the color image.

S611, displaying the gesture image of the GUI interface.

And the display control displays the image processed in the S610 for the user to view.

S612, the terminal equipment judges whether the current gesture is determined as the unlocking gesture by the user

Specifically, after the fingertip recognition is completed, if the user is satisfied with the current gesture, the key for determining the setting of the gesture is pressed, the gesture is confirmed to be used as an unlocking gesture, and meanwhile, the terminal device records the frame of image. The terminal device proceeds to S614. If the user is not satisfied with the current gesture, resetting the unlocking gesture and the terminal equipment

Jump back to execution S603.

S613, the terminal equipment judges whether the number of the fingers in the gesture meets the setting requirement.

As an embodiment of the present application, the terminal device determines whether the number of fingers is greater than or equal to 3, if yes, executes S615, and if not, discards the current image and identifies the next frame of image.

And S614, the terminal equipment continues to recognize other gesture feature data.

And when the number of the fingers in the gesture meets the set requirement, the terminal equipment continues to recognize the finger length, the finger tip position characteristic data and the finger width characteristic data.

S615, the terminal device stores the gesture feature data including the number of the fingers in a nonvolatile memory for use in an unlocking stage.

Until then, the image displayed in S611 is set as the unlocking gesture.

An embodiment of a three-dimensional gesture unlocking method according to an embodiment of the present application is described in detail below with reference to fig. 7. The embodiment mainly describes the corresponding operations of the user and the terminal device in the unlocking stage.

S701, the terminal device detects a second operation.

For example, the second operation may be an operation in which the user presses a power key. In the unlocking stage, the user presses a power key to start the gesture unlocking process.

And S702, triggering the infrared emitter and the infrared camera to work.

For example, after the user presses a power key, the infrared emitter and the infrared camera are triggered to work.

And S703, acquiring the current gesture image in real time.

Specifically, the infrared spectrum analysis chip acquires depth data of an image acquired by the infrared camera in real time.

S704, the terminal equipment judges whether the first mode described in the foregoing needs to be executed according to the current frame number, that is, the current gesture is subjected to complete contour extraction. If necessary, S705 is performed, otherwise S707 is performed.

And S705, the terminal equipment performs contour extraction.

Specifically, the terminal device may acquire the contour data of the frame image according to the manner described above.

S706, the terminal device judges whether the contour is extracted successfully, namely whether a gesture exists in the frame of image and whether the gesture is too large. If the frame of image has a gesture and a gesture contour point exists, it can be judged that contour extraction is successful.

And S707, the terminal equipment performs fingertip identification.

Specifically, if the contour extraction is successful, the feature data of the gesture is recognized next, and the recognition of the fingertip is performed first. Or, in S704, the terminal device determines that global tracking is not needed according to the current frame number, that is, the manner described above is not needed for the first time, and then performs fingertip position prediction without performing complete contour extraction on the current gesture according to the manner described above, and finally obtains a gesture contour linked list similar to the contour extraction according to the obtained predicted fingertip pixel point and predicted finger root pixel point.

Optionally, after performing the fingertip identification, the method may further include: and the terminal equipment judges whether the number of the fingers in the gesture meets the set requirement.

As an embodiment of the present application, the terminal device determines whether the number of fingers is greater than or equal to 3, if so, executes S708 or S709, and if not, discards the current image and identifies the next frame of image.

S708, the contour detection upper limit value is corrected based on the detected first fingertip position information.

Specifically, the step may refer to the above description, and is not repeated herein for brevity. If the contour detection upper limit value needs to be corrected, S705 is executed after the contour detection upper limit value is corrected.

And S709, the terminal equipment continues to recognize other gesture feature data.

For example, when the number of fingers in the gesture satisfies the setting requirement, the terminal device continues to recognize the finger length, the finger tip position feature data and the finger width feature data. The latter, after performing fingertip recognition, directly performs this step.

S710, judging whether the characteristic data of the gesture and the characteristic data of the preset gesture meet the conditions.

Specifically, according to whether the number of the fingers is matched before the step, the finger length, the fingertip position characteristic data and the finger width characteristic data in the gesture are judged to be matched with the corresponding finger length, the fingertip position characteristic data and the finger width characteristic data in the preset gesture. Or judging whether the number of the fingers, the length of the fingers, the fingertip position characteristic data and the finger width characteristic data are matched with the corresponding number of the fingers, the length of the fingers, the fingertip position characteristic data and the finger width characteristic data in the preset gesture. If so, S711 is performed, otherwise S712 is performed.

And S711, if the feature data of the gesture and the feature data of the preset gesture meet the condition, unlocking the terminal device.

And S712, if the characteristic data of the gesture and the characteristic data of the preset gesture do not meet the condition, judging whether the unlocking process time is overtime. If time out, S713 is executed, otherwise, it jumps back to S703.

And S713, turning off the infrared equipment, namely turning off the infrared emitter and the infrared camera.

And S714, the terminal equipment prompts the user that the time of the unlocking process is overtime, and locks the screen after a certain period of time.

It should be understood that the certain period of time may be preset by the system, and may be, for example, 2 seconds, 4 seconds, etc.

It should also be understood that the execution of S713 and S714 are not sequential.

It should also be understood that, the step a and the step B described herein are not performed in a sequential order, which means that the step a and the step B may be performed simultaneously, or the step a may be performed first and then the step B may be performed, or the step B may be performed first and then the step a may be performed.

Fig. 8 is a schematic flow chart of a method of acquiring a gesture image according to an embodiment of the present application. The method may be applied to, but is not limited to, the system shown in fig. 1.

And S810, acquiring a current gesture image of the user.

For example, when the method is executed by the system shown in fig. 1, the depth camera 120 receives an excitation light source reflected from the outside, such as infrared light, and digitally encodes the excitation light source to form a digital image and transmits the digital image to the spectral analysis module 130. The spectrum analysis module 130 analyzes the speckles, and calculates a distance z between a corresponding pixel point (x, y) in the current gesture image and the depth camera 120, so that the current gesture image can be obtained.

S820, according to the current gesture image, acquiring the contour data of the current gesture in the current gesture image.

For example, the profile data described herein may refer to coordinates (x, y, z) of various pixel points in the gesture image that constitute the gesture profile.

Alternatively, the contour data of the current gesture in the current gesture image may be acquired in the above manner one or manner two. Reference may be made specifically to the above description, which is not repeated here for the sake of brevity.

And S830, determining the fingertip pixel point of the current gesture and/or the finger root pixel point of the current gesture according to the contour data of the current gesture.

Specifically, reference may be made to the above-described methods, which are not described herein for brevity.

And S840, determining the characteristic data of the current gesture according to the fingertip pixel point of the current gesture and/or the finger root pixel point of the current gesture.

Optionally, the determining the feature data of the current gesture according to the fingertip pixel point of the current gesture and/or the finger root pixel point of the current gesture includes:

determining at least one of the following in the current gesture according to the fingertip pixel point of the current gesture and/or the finger root pixel point of the current gesture:

the method comprises the following steps of finger number, finger length, fingertip position characteristic data and finger width characteristic data, wherein the fingertip position characteristic data are used for indicating relative positions of fingers, and the finger width characteristic data are used for indicating finger width.

Specifically, reference may be made to the above-described methods, which are not described herein for brevity.

According to the method for acquiring the gesture image, the fingertip pixel point of the current gesture and/or the finger root pixel point of the current gesture can be acquired according to the contour data of the current gesture in the current gesture image, and therefore the feature data of the current gesture can be determined. Optionally, step S840 is followed by: and when the feature data of the current gesture is successfully matched with the feature data of the preset gesture, unlocking the terminal equipment.

Fig. 9 shows a schematic block diagram of a terminal device 900 according to an embodiment of the present application. As shown in fig. 9, the terminal apparatus 900 includes: an acquisition unit 910 and a processing unit 920.

An obtaining unit 910, configured to obtain a current gesture image of a user.

The processing unit 920 is configured to obtain, according to the current gesture image, profile data of a current gesture in the current gesture image; according to the contour data of the current gesture, determining fingertip pixel points of the current gesture and/or finger root pixel points of the current gesture; determining feature data of the current gesture according to the fingertip pixel points of the current gesture and/or the finger root pixel points of the current gesture; and unlocking the terminal equipment when the feature data of the current gesture is successfully matched with the feature data of the preset gesture.

Therefore, the terminal device of the embodiment of the application obtains the three-dimensional gesture image presented by the user in the three-dimensional space in front of the camera in real time, extracts the gesture of the user in the gesture image, and achieves the purpose of unlocking the terminal device by matching with the unlocking gesture set by the user in advance. Therefore, a brand-new unlocking mode with strong interestingness, high accuracy and good rapidness is provided for the user.

Optionally, the terminal device of the embodiment of the present application includes, but is not limited to, a mobile phone, a tablet, a computer, a multimedia machine, and a game machine. All devices using a mobile communication network are within the scope of the embodiments of the present application.

The terminal device 900 in this embodiment of the present application may correspond to the terminal device in the embodiment of the method in this application, and the above and other operations and/or functions of each module in the terminal device 900 are respectively for implementing corresponding flows of the methods shown in fig. 2 to fig. 7, and are not described herein again for brevity.

Fig. 9 shows a schematic block diagram of a terminal device 1000 according to an embodiment of the present application. As shown in fig. 10, the terminal apparatus 1000 includes: an acquisition unit 1010 and a processing unit 1020.

An obtaining unit 1010, configured to obtain a current gesture image of the user.

The processing unit 1020 is configured to obtain, according to the current gesture image, profile data of a current gesture in the current gesture image; according to the contour data of the current gesture, determining fingertip pixel points of the current gesture and/or finger root pixel points of the current gesture; and determining the characteristic data of the current gesture according to the fingertip pixel point of the current gesture and/or the finger root pixel point of the current gesture.

According to the terminal equipment, according to the contour data of the current gesture in the current gesture image, the fingertip pixel point of the current gesture and/or the base finger pixel point of the current gesture can be obtained, and therefore the feature data of the current gesture can be determined.

Optionally, the terminal device of the embodiment of the present application includes, but is not limited to, a mobile phone, a tablet, a computer, a multimedia machine, and a game machine. All devices using a mobile communication network are within the scope of the embodiments of the present application.

The terminal device 1000 in this embodiment of the present application may correspond to the terminal device in this embodiment of the present application method, and the above and other operations and/or functions of each module in the terminal device 1000 are respectively for implementing the corresponding flow of the method shown in fig. 8, and are not described herein again for brevity.

Fig. 11 is another schematic block diagram of a terminal device according to an embodiment of the present application. The terminal device 1100 shown in fig. 11 includes: radio Frequency (RF) circuitry 1110, memory 1120, other input devices 1130, display 1140, sensors 1150, audio circuitry 1160, I/O subsystem 1170, processor 1180, and power supply 1190. Those skilled in the art will appreciate that the terminal device configuration shown in fig. 11 does not constitute a limitation of the terminal device, and may include more or fewer components than those shown, or combine certain components, or split certain components, or a different arrangement of components. It will be appreciated by those skilled in the art that the display 1140 pertains to a User Interface (UI) and that the terminal device 1100 may include fewer or more User interfaces than shown.

The following specifically describes each constituent element of the terminal device 1100 with reference to fig. 11:

RF circuit 1110 may be used for receiving and transmitting signals during a message transmission or call, and in particular, for receiving downlink messages from a base station and then processing the received downlink messages to processor 1180; in addition, the data for designing uplink is transmitted to the base station. Typically, the RF circuitry includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a Low Noise Amplifier (LNA), a duplexer, and the like. In addition, the RF circuitry 1110 may also communicate with networks and other devices via wireless communications. The wireless communication may use any communication standard or protocol, including but not limited to global system for Mobile communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Messaging Service (SMS), and the like.

The memory 1120 may be used to store software programs and modules, and the processor 1180 may execute various functional applications and data processing of the terminal device 1100 by operating the software programs and modules stored in the memory 1120. The memory 1120 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required by at least one function (such as a sound playing function, an image playing function, etc.), and the like; the storage data area may store data (such as audio data, a phonebook, etc.) created according to the use of the terminal device 1100, and the like. Further, the memory 1120 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid state storage device.

Other input devices 1130 may be used to receive entered numeric or character information and generate key signal inputs related to user settings and function controls of terminal device 1100. In particular, other input devices 1130 may include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, a light mouse (a light mouse is a touch-sensitive surface that does not display visual output, or is an extension of a touch-sensitive surface formed by a touch screen), and the like. Other input devices 1130 are connected to other input device controllers 1171 of the I/O subsystem 1170 and interact with the processor 1180 in signals under the control of the other device input controllers 1171.

Display screen 1140 may be used to display information entered by or provided to the user as well as various menus for terminal device 1100, and may also accept user input. Specific display 1140 may include a display panel 1141 and a touch panel 1142. The Display panel 1141 may be configured by a Liquid Crystal Display (LCD), an Organic Light-Emitting Diode (OLED), and the like. The touch panel 1142, also referred to as a touch screen, a touch sensitive screen, etc., may collect contact or non-contact operations (e.g., operations performed by a user on or near the touch panel 1142 using any suitable object or accessory such as a finger, a stylus, etc., and may also include body-sensing operations; the operations include single-point control operations, multi-point control operations, etc.) and drive the corresponding connection device according to a preset program. Alternatively, the touch panel 1142 may include two parts, i.e., a touch detection device and a touch controller. The touch detection device detects the touch direction and gesture of a user, detects signals brought by touch operation and transmits the signals to the touch controller; the touch controller receives touch information from the touch sensing device, converts the touch information into information that can be processed by the processor, sends the information to the processor 1180, and receives and executes commands sent by the processor 1180. In addition, the touch panel 1142 may be implemented by various types such as a resistive type, a capacitive type, an infrared ray, a surface acoustic wave, and the like, and the touch panel 1142 may also be implemented by any technology developed in the future. Further, touch panel 1142 covers display panel 1141, a user can operate on or near touch panel 1142 covered on display panel 1141 according to content displayed on display panel 1141 (the display content includes, but is not limited to, a soft keyboard, a virtual mouse, virtual keys, icons, etc.), touch panel 1142 detects the operation on or near touch panel 1142, and transmits the detected operation to processor 1180 through I/O subsystem 1170 to determine a user input, and processor 1180 provides a corresponding visual output on display panel 1141 through I/O subsystem 1170 according to the user input. Although in fig. 11, touch panel 1142 and display panel 1141 are shown as two separate components to implement the input and output functions of terminal device 1100, in some embodiments, touch panel 1142 and display panel 1141 may be integrated to implement the input and output functions of terminal device 1100.

Terminal device 1100 can also include at least one sensor 1150, such as a light sensor, motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor that adjusts the brightness of the display panel 1141 according to the brightness of ambient light, and a proximity sensor that turns off the display panel 1141 and/or the backlight when the terminal device 1100 moves to the ear. As one of the motion sensors, the accelerometer sensor can detect the magnitude of acceleration in each direction (generally, three axes), can detect the magnitude and direction of gravity when stationary, and can be used for applications of recognizing the posture of a mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer and tapping), and the like; as for other sensors such as a gyroscope, a barometer, a hygrometer, a thermometer, and an infrared sensor, which can be configured in the terminal device 1100, detailed descriptions thereof are omitted.

Audio circuitry 1160, speaker 1161, and microphone 1162 may provide an audio interface between a user and terminal device 1100. The audio circuit 1160 may transmit the converted signal of the received audio data to the speaker 1161, and convert the signal into a sound signal for output by the speaker 1161; on the other hand, the microphone 1162 converts collected sound signals into signals, which are received by the audio circuit 1160, converted into audio data, and then output the audio data to the RF circuit 1110 for transmission to, for example, another cell phone, or to the memory 1120 for further processing.

The external devices used by the I/O subsystem 1170 to control input and output may include other device input controllers 1171, sensor controllers 1172, and display controllers 1173. Optionally, one or more other input control device controllers 1171 receive signals from and/or transmit signals to other input devices 1130, and the other input devices 1130 may include physical buttons (push buttons, rocker buttons, etc.), dials, slide switches, joysticks, click wheels, light mice (a light mouse is a touch-sensitive surface that does not display visual output, or is an extension of a touch-sensitive surface formed by a touch screen). It is noted that other input control device controllers 1171 may be connected to any one or more of the above devices. A display controller 1173 in the I/O subsystem 1170 receives signals from and/or sends signals to the display screen 1140. Upon detection of user input by the display screen 1140, the display controller 1173 converts the detected user input into interaction with user interface objects displayed on the display screen 1140, i.e., to implement human-machine interaction. The sensor controller 1172 may receive signals from the one or more sensors 1150 and/or send signals to the one or more sensors 1150.

The processor 1180 is a control center of the terminal device 1100, connects various parts of the whole terminal device by using various interfaces and lines, and executes various functions and processes data of the terminal device 1100 by running or executing software programs and/or modules stored in the memory 1120 and calling data stored in the memory 1120, thereby performing overall monitoring of the terminal device. Optionally, processor 1180 may include one or more processing units; alternatively, the processor 1180 may integrate an application processor, which handles primarily the operating system, user interfaces, applications, etc., and a modem processor, which handles primarily the wireless communications. It will be appreciated that the modem processor described above may not be integrated within processor 1180.

The processor 1180 is configured to: acquiring a current gesture image of a user; acquiring contour data of a current gesture in the current gesture image according to the current gesture image; according to the contour data of the current gesture, determining fingertip pixel points of the current gesture and/or finger root pixel points of the current gesture; determining feature data of the current gesture according to the fingertip pixel points of the current gesture and/or the finger root pixel points of the current gesture; and unlocking the terminal equipment when the feature data of the current gesture is successfully matched with the feature data of the preset gesture.

Alternatively, the processor 1180 is configured to: acquiring a current gesture image of a user; acquiring contour data of a current gesture in the current gesture image according to the current gesture image; according to the contour data of the current gesture, determining fingertip pixel points of the current gesture and/or finger root pixel points of the current gesture; and determining the characteristic data of the current gesture according to the fingertip pixel point of the current gesture and/or the finger root pixel point of the current gesture.

Terminal device 1100 also includes a power supply 1190 (e.g., a battery) that provides power to various components, which may optionally be logically coupled to processor 1180 via a power management system to enable management of charging, discharging, and power consumption functions via the power management system.

Although not shown, the terminal device 1100 may further include a camera, a bluetooth module, and the like, which will not be described herein.

It should be understood that the terminal device 1100 may correspond to a terminal device in the three-dimensional gesture unlocking method according to the embodiment of the present application, and the terminal device 1100 may include an entity unit for performing the method performed by the terminal device or the electronic device in the above method. Moreover, each entity unit and the other operations and/or functions in the terminal device 1100 are respectively corresponding to the flows of the method, and are not described herein again for brevity.

It should also be understood that the terminal device 1100 may include a physical unit for performing the above-described method of acquiring a gesture image. Moreover, each entity unit and the other operations and/or functions in the terminal device 1100 are respectively corresponding to the flows of the method, and are not described herein again for brevity.

It should also be understood that the processor in the embodiments of the present application may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the above method embodiments may be performed by integrated logic circuits of hardware in a processor or instructions in the form of software. The processor may be a Central Processing Unit (CPU), other general purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic device, or discrete hardware components. The various methods, steps, and logic blocks disclosed in the embodiments of the present application may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in connection with the embodiments of the present application may be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software in the decoding processor. The software may be in ram, flash, rom, prom, or eprom, registers, among other storage media as is well known in the art. The storage medium is located in a memory, and a processor reads information in the memory and completes the steps of the method in combination with hardware of the processor.

It will also be appreciated that the memory in the embodiments of the subject application can be either volatile memory or nonvolatile memory, or can include both volatile and nonvolatile memory. The non-volatile Memory may be a Read-Only Memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically Erasable PROM (EEPROM), or a flash Memory. Volatile Memory can be Random Access Memory (RAM), which acts as external cache Memory. By way of example, but not limitation, many forms of RAM are available, such as Static random access memory (Static RAM, SRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic random access memory (Synchronous DRAM, SDRAM), Double Data rate Synchronous Dynamic random access memory (DDR SDRAM), Enhanced Synchronous SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and direct memory bus RAM (DR RAM). It should be noted that the memory of the systems and methods described herein is intended to comprise, without being limited to, these and any other suitable types of memory.

It will also be appreciated that the bus system may include a power bus, a control bus, a status signal bus, etc., in addition to the data bus. For clarity of illustration, however, the various buses are labeled as a bus system in the figures.

It should also be understood that in the present embodiment, "B corresponding to a" means that B is associated with a, from which B can be determined. It should also be understood that determining B from a does not mean determining B from a alone, but may be determined from a and/or other information. It should be understood that the term "and/or" herein is merely one type of association relationship that describes an associated object, meaning that three relationships may exist, e.g., a and/or B may mean: a exists alone, A and B exist simultaneously 10, and B exists alone. In addition, the character "/" herein generally indicates that the former and latter related objects are in an "or" relationship.

In implementation, the steps of the above method may be performed by integrated logic circuits of hardware in a processor or instructions in the form of software. The steps of the method for transmitting an uplink signal disclosed in the embodiments of the present application may be directly implemented by a hardware processor, or implemented by a combination of hardware and software in the processor. The software may be in ram, flash, rom, prom, or eprom, registers, among other storage media as is well known in the art. The storage medium is located in a memory, and a processor reads information in the memory and completes the steps of the method in combination with hardware of the processor. To avoid repetition, it is not described in detail here.

Embodiments of the present application also provide a computer-readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by a portable electronic device comprising a plurality of application programs, enable the portable electronic device to perform the method of the embodiment shown in fig. 2 and/or fig. 3.

Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the embodiments of the present application.

It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.

In the several embodiments provided in the present application, it should be understood that the disclosed system, apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the units is only one logical division, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.

The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit.

The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the embodiments of the present application, which essentially or partly contribute to the prior art, may be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the methods described in the embodiments of the present application. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.

The above description is only a specific implementation of the embodiments of the present application, but the scope of the embodiments of the present application is not limited thereto, and any person skilled in the art can easily conceive of changes or substitutions within the technical scope of the embodiments of the present application, and all the changes or substitutions should be covered by the scope of the embodiments of the present application. Therefore, the protection scope of the embodiments of the present application shall be subject to the protection scope of the claims.

Claims (28)

1. A three-dimensional gesture unlocking method is applied to a terminal device, and comprises the following steps:

acquiring a current gesture image of a user;

acquiring contour data of a current gesture in the current gesture image according to the current gesture image; the acquiring, according to the current gesture image, profile data of a current gesture in the gesture image includes:

determining whether contour extraction needs to be carried out on the current gesture according to the current frame number of the current gesture image, and if the current frame number of the current gesture image is the 1 st gesture image in each L frame of gesture images, contour extraction needs to be carried out;

when the current gesture needs to be subjected to contour extraction, searching contour pixel points of the current gesture until all contour pixel points of the current gesture are searched or the number of the searched pixel points is larger than a preset detection threshold value, so as to obtain contour data of the current gesture;

according to the contour data of the current gesture, determining fingertip pixel points of the current gesture and/or finger root pixel points of the current gesture;

determining feature data of the current gesture according to the fingertip pixel points of the current gesture and/or the finger root pixel points of the current gesture;

and unlocking the terminal equipment when the feature data of the current gesture is successfully matched with the feature data of the preset gesture.

2. The method of claim 1, wherein determining the feature data of the current gesture according to the fingertip pixel point and/or the root pixel point of the current gesture comprises:

determining at least one of the following information in the current gesture according to the fingertip pixel point of the current gesture and/or the finger root pixel point of the current gesture:

the method comprises the following steps of finger number, finger length, fingertip position characteristic data and finger width characteristic data, wherein the fingertip position characteristic data are used for indicating relative positions of fingers, and the finger width characteristic data are used for indicating finger width.

3. The method of claim 2, wherein the method further comprises:

when determining that contour extraction is not needed to be carried out on the current gesture, determining a predicted fingertip pixel point of the current gesture and a predicted finger root pixel point of the current gesture according to an exponential average number (EMA) algorithm;

and acquiring the contour data of the current gesture according to the predicted fingertip pixel point of the current gesture and the predicted finger root pixel point of the current gesture.

4. The method according to claim 2 or 3, wherein the determining, according to the contour data of the current gesture, a fingertip pixel point of the current gesture and/or a root pixel point of the current gesture comprises:

determining v arc sections in the current gesture according to the contour data of the current gesture, wherein v is an integer greater than or equal to 1;

determining a central pixel point of the s-th arc according to a starting pixel point of the s-th arc and an ending pixel point of the s-th arc in the current gesture, and traversing s through an integer in [1, v ];

determining a central vector p of the s-th arc according to a vector m formed by k pixel points from the central pixel point of the s-th arc to the central pixel point of the s-th arc and a vector n formed by k pixel points from the central pixel point of the s-th arc to the central pixel point of the s-th arc, wherein the central vector p bisects a smaller angle of two angles formed by the vector m and the vector n, and k is an integer greater than or equal to 1;

when the distance between the position of the pixel point in the direction pointed by the central vector p and the depth camera is within a preset distance interval, determining that the central pixel point corresponding to the central vector p is a fingertip pixel point, or

And when the distance between the position of the pixel point in the direction pointed by the central vector p and the depth camera is not within a preset distance interval, determining the central pixel point corresponding to the central vector p as a finger root pixel point.

5. The method of claim 4, wherein the determining the finger length in the current gesture comprises:

determining a first distance between the fingertip pixel point and a finger root pixel point positioned on the left side of the fingertip pixel point and a second distance between the fingertip pixel point and a finger root pixel point positioned on the right side of the fingertip pixel point;

determining the projection of the smaller of the first distance and the second distance in the finger direction in which the fingertip pixel point is located as the finger length when the absolute value of the difference between the first distance and the second distance is greater than a preset length threshold,

when the absolute value of the difference between the first distance and the second distance is smaller than or equal to a preset length threshold, determining the average value of the projection of the first distance in the finger direction where the fingertip pixel point is located and the projection of the second distance in the finger direction where the fingertip pixel point is located as the finger length.

6. The method of claim 4, wherein the determining the finger length in the current gesture comprises:

when there is no finger root pixel point on the right side of the fingertip pixel point, or when there is no finger root pixel point on the left side of the fingertip pixel point,

determining a third distance between the fingertip pixel point and a finger root pixel point positioned on the left side of the fingertip pixel point, or

Determining a fourth distance between the fingertip pixel point and a finger root pixel point positioned on the right side of the fingertip pixel point;

determining the projection of the third distance in the finger direction of the fingertip pixel point as the finger length, or

And determining the projection of the fourth distance in the finger direction where the fingertip pixel point is located as the finger length.

7. The method of claim 4, wherein said fingertip position feature data D is_jThe following formula is satisfied:

wherein k is the length of the longest finger in the current gesture, k is greater than 0, n is the number of fingers in the current gesture, n is an integer greater than or equal to 1, p is a fingertip pixel point in the current gesture, x and y are coordinates of the fingertip pixel point in the current gesture image, x and y are real numbers, z is a depth value of the fingertip pixel point in the current gesture image, and z is greater than or equal to 0.

8. The method of claim 4, wherein the determining finger width feature data in the gesture image comprises:

averagely dividing the length of each finger in the current gesture into m +1 parts in the finger direction, wherein m is a positive integer greater than or equal to 0;

for each finger in the current gesture, calculating the width perpendicular to the direction of the finger at the position of the bisection point to obtain m × n absolute width values, wherein n is the number of the fingers in the current gesture, and n is an integer greater than or equal to 1;

calculating the ratio of every two absolute value widths in the m x n absolute width values to obtain characteristic vector values { d) formed by mn (mn-1)/2 relative widths_i,i＝1,2,…,mn(mn-1)/2}；

The feature vector value { d_iI ═ 1,2, …, mn (mn-1)/2} is determined as the finger width characteristic data.

9. The method of any of claims 1 to 3, further comprising:

when the feature data of the current gesture is not successfully matched with the feature data of the preset gesture, determining whether the unlocking process time is overtime;

and if the unlocking process time is overtime, carrying out screen locking processing on the terminal equipment after a preset time period.

10. The method of any of claims 1-3, prior to obtaining profile data for a current gesture in the current gesture image from the current gesture image, the method further comprising:

acquiring and presenting candidate gesture images to the user based on the starting setting unlocking gesture operation of the user;

setting a candidate gesture in the candidate gesture image as a preset gesture based on the determined operation of the user.

11. The method of claim 10, wherein prior to the setting of the candidate gesture in the candidate gesture image to the preset gesture based on the determined operation of the user, the method further comprises:

determining whether the number of fingers of the candidate gesture is greater than or equal to 3;

if the number of fingers of the candidate gesture is greater than or equal to 3, setting the candidate gesture as the preset gesture based on the determined operation of the user;

wherein the method further comprises:

and acquiring feature data of the candidate gesture.

12. A method for acquiring gesture images, comprising:

acquiring a current gesture image of a user;

acquiring contour data of a current gesture in the current gesture image according to the current gesture image; the acquiring, according to the current gesture image, profile data of a current gesture in the gesture image includes:

determining whether contour extraction needs to be carried out on the current gesture according to the current frame number of the current gesture image, and if the current frame number of the current gesture image is the 1 st gesture image in each L frame of gesture images, contour extraction needs to be carried out;

when the current gesture needs to be subjected to contour extraction, searching contour pixel points of the current gesture until all contour pixel points of the current gesture are searched or the number of the searched pixel points is larger than a preset detection threshold value, so as to obtain contour data of the current gesture;

according to the contour data of the current gesture, determining fingertip pixel points of the current gesture and/or finger root pixel points of the current gesture;

and determining the characteristic data of the current gesture according to the fingertip pixel point of the current gesture and/or the finger root pixel point of the current gesture.

13. The method of claim 12, wherein determining the feature data of the current gesture according to the fingertip pixel point and/or the root pixel point of the current gesture comprises:

determining at least one of the following in the current gesture according to the fingertip pixel point of the current gesture and/or the finger root pixel point of the current gesture:

the method comprises the following steps of finger number, finger length, fingertip position characteristic data and finger width characteristic data, wherein the fingertip position characteristic data are used for indicating relative positions of fingers, and the finger width characteristic data are used for indicating finger width.

14. A terminal device, comprising:

the acquisition unit is used for acquiring a current gesture image of a user;

the processing unit is used for acquiring contour data of a current gesture in the current gesture image according to the current gesture image; the processing unit is specifically configured to:

determining whether contour extraction needs to be carried out on the current gesture according to the current frame number of the current gesture image, and if the current frame number of the current gesture image is the 1 st gesture image in each L frame of gesture images, contour extraction needs to be carried out;

when the current gesture needs to be subjected to contour extraction, searching contour pixel points of the current gesture until all contour pixel points of the current gesture are searched or the number of the searched pixel points is larger than a preset detection threshold value, so as to obtain contour data of the current gesture;

the processing unit is further configured to determine a fingertip pixel point of the current gesture and/or a finger root pixel point of the current gesture according to the profile data of the current gesture;

the processing unit is further configured to determine feature data of the current gesture according to the fingertip pixel point of the current gesture and/or the finger root pixel point of the current gesture;

the processing unit is further used for unlocking the terminal device when the feature data of the current gesture is successfully matched with the feature data of the preset gesture.

15. The terminal device of claim 14, wherein the processing unit is specifically configured to:

determining at least one of the following information in the current gesture according to the fingertip pixel point of the current gesture and/or the finger root pixel point of the current gesture:

the method comprises the following steps of finger number, finger length, fingertip position characteristic data and finger width characteristic data, wherein the fingertip position characteristic data are used for indicating relative positions of fingers, and the finger width characteristic data are used for indicating finger width.

16. The terminal device of claim 15, wherein the processing unit is further to:

when determining that contour extraction is not needed to be carried out on the current gesture, determining a predicted fingertip pixel point of the current gesture and a predicted finger root pixel point of the current gesture according to an exponential average number (EMA) algorithm;

and acquiring the contour data of the current gesture according to the predicted fingertip pixel point of the current gesture and the predicted finger root pixel point of the current gesture.

17. The terminal device according to claim 15 or 16, wherein the processing unit is specifically configured to:

determining v arc sections in the current gesture according to the contour data of the current gesture, wherein v is an integer greater than or equal to 1;

determining a central pixel point of the s-th arc according to a starting pixel point of the s-th arc and an ending pixel point of the s-th arc in the current gesture, and traversing s through an integer in [1, v ];

determining a central vector p of the s-th arc according to a vector m formed by k pixel points from the central pixel point of the s-th arc to the central pixel point of the s-th arc and a vector n formed by k pixel points from the central pixel point of the s-th arc to the central pixel point of the s-th arc, wherein the central vector p bisects a smaller angle of two angles formed by the vector m and the vector n, and k is an integer greater than or equal to 1;

when the distance between the position of the pixel point in the direction pointed by the central vector p and the depth camera is within a preset distance interval, determining that the central pixel point corresponding to the central vector p is a fingertip pixel point, or

And when the distance between the position of the pixel point in the direction pointed by the central vector p and the depth camera is not within a preset distance interval, determining the central pixel point corresponding to the central vector p as a finger root pixel point.

18. The terminal device of claim 17, wherein the processing unit is specifically configured to:

determining a first distance between the fingertip pixel point and a finger root pixel point positioned on the left side of the fingertip pixel point and a second distance between the fingertip pixel point and a finger root pixel point positioned on the right side of the fingertip pixel point;

determining the projection of the smaller of the first distance and the second distance in the finger direction in which the fingertip pixel point is located as the finger length when the absolute value of the difference between the first distance and the second distance is greater than a preset length threshold,

when the absolute value of the difference between the first distance and the second distance is smaller than or equal to a preset length threshold, determining the average value of the projection of the first distance in the finger direction where the fingertip pixel point is located and the projection of the second distance in the finger direction where the fingertip pixel point is located as the finger length.

19. The terminal device of claim 17, wherein the processing unit is specifically configured to:

when there is no finger root pixel point on the right side of the fingertip pixel point, or when there is no finger root pixel point on the left side of the fingertip pixel point,

determining a third distance between the fingertip pixel point and a finger root pixel point positioned on the left side of the fingertip pixel point, or

Determining a fourth distance between the fingertip pixel point and a finger root pixel point positioned on the right side of the fingertip pixel point;

determining the projection of the third distance in the finger direction of the fingertip pixel point as the finger length, or

And determining the projection of the fourth distance in the finger direction where the fingertip pixel point is located as the finger length.

20. Terminal device according to claim 17, characterized in that said fingertip position characteristic data D_jThe following formula is satisfied:

wherein k is the length of the longest finger in the current gesture, k is greater than 0, n is the number of fingers in the current gesture, n is an integer greater than or equal to 1, p is a fingertip pixel point in the current gesture, x and y are coordinates of the fingertip pixel point in the current gesture image, x and y are real numbers, z is a depth value of the fingertip pixel point in the current gesture image, and z is greater than or equal to 0.

21. The terminal device of claim 17, wherein the processing unit is specifically configured to:

averagely dividing the length of each finger in the current gesture into m +1 parts in the finger direction, wherein m is a positive integer greater than or equal to 0;

for each finger in the current gesture, calculating the width perpendicular to the direction of the finger at the position of the bisection point to obtain m × n absolute width values, wherein n is the number of the fingers in the current gesture, and n is an integer greater than or equal to 1;

calculating the ratio of every two absolute value widths in the m x n absolute width values to obtain characteristic vector values { d) formed by mn (mn-1)/2 relative widths_i,i＝1,2,…,mn(mn-1)/2}；

The feature vector value { d_iI ═ 1,2, …, mn (mn-1)/2} is determined as the finger width characteristic data.

22. The terminal device of any of claims 14 to 16, wherein the processing unit is further configured to:

when the feature data of the current gesture is not successfully matched with the feature data of the preset gesture, determining whether the unlocking process time is overtime;

and if the unlocking process time is overtime, carrying out screen locking processing on the terminal equipment after a preset time period.

23. The terminal device of any of claims 14 to 16, wherein the processing unit is further configured to:

acquiring and presenting candidate gesture images to the user based on the starting setting unlocking gesture operation of the user;

setting a candidate gesture in the candidate gesture image as a preset gesture based on the determined operation of the user.

24. The terminal device of claim 23, wherein the processing unit is further configured to:

determining whether the number of fingers of the candidate gesture is greater than or equal to 3;

if the number of fingers of the candidate gesture is greater than or equal to 3, setting the candidate gesture as the preset gesture based on the determined operation of the user;

wherein, the terminal equipment still includes:

and acquiring feature data of the candidate gesture.

25. A terminal device, comprising:

the acquisition unit is used for acquiring a current gesture image of a user;

the processing unit is used for acquiring contour data of a current gesture in the current gesture image according to the current gesture image; the processing unit is specifically configured to:

determining whether contour extraction needs to be carried out on the current gesture according to the current frame number of the current gesture image, and if the current frame number of the current gesture image is the 1 st gesture image in each L frame of gesture images, contour extraction needs to be carried out;

when the current gesture needs to be subjected to contour extraction, searching contour pixel points of the current gesture until all contour pixel points of the current gesture are searched or the number of the searched pixel points is larger than a preset detection threshold value, so as to obtain contour data of the current gesture;

the processing unit is further configured to determine a fingertip pixel point of the current gesture and/or a finger root pixel point of the current gesture according to the profile data of the current gesture;

the processing unit is further configured to determine feature data of the current gesture according to the fingertip pixel point of the current gesture and/or the finger root pixel point of the current gesture.

26. The terminal device of claim 25, wherein the processing unit is specifically configured to:

determining at least one of the following in the current gesture according to the fingertip pixel point of the current gesture and/or the finger root pixel point of the current gesture:

the method comprises the following steps of finger number, finger length, fingertip position characteristic data and finger width characteristic data, wherein the fingertip position characteristic data are used for indicating relative positions of fingers, and the finger width characteristic data are used for indicating finger width.

27. A terminal device, comprising: a memory, a processor, and a display;

a memory for storing a program;

the processor configured to execute the program stored in the memory, the processor configured to perform the method of any of claims 1-11 when the program is executed.

28. A terminal device, comprising: a memory, a processor, and a display;

the memory is used for storing programs;

the processor for executing the program stored by the memory, the processor for performing the method of claim 12 or 13 when the program is executed.

Images