CN113056718A

CN113056718A - Handheld mobile terminal control method and related device

Info

Publication number: CN113056718A
Application number: CN201880099510.5A
Authority: CN
Inventors: 叶波; 孙忠; 李大伟
Original assignee: Huawei Technologies Co Ltd
Current assignee: Huawei Technologies Co Ltd
Priority date: 2018-11-14
Filing date: 2018-11-14
Publication date: 2021-06-29
Also published as: WO2020097830A1

Abstract

The application discloses a handheld mobile terminal control method and a related device. The terminal can acquire user images in real time through the camera and can autonomously operate AI perception capability. When a user needs to use the terminal, the terminal can autonomously perform gesture recognition only by inputting a user gesture, and corresponding operation is executed according to the input user gesture. According to the method and the device, the control of the terminal can be more intelligent, and the user experience is improved.

Description

Handheld mobile terminal control method and related device

Technical Field

The present disclosure relates to the field of terminals and Artificial Intelligence (AI), and more particularly, to a method and an apparatus for controlling a handheld mobile terminal.

Background

With the development of terminal technology, the functions of the terminal are more and more. Most terminals are configured with touch screens, and users can control the terminals through touch gestures on the touch screens to use various functions of the terminals. For example, a user may click on an application icon on a touch screen to launch an application, may switch display interfaces by sliding a finger left or right on the touch screen, and so forth.

However, in many scenarios, it is inconvenient for a user to directly touch the touch screen with a hand to operate the terminal. For example, when a user is cooking, when the user's hands are stained with liquid or dirt, and when the user has too many things to vacate the hands, the user is inconvenient to manipulate the terminal through touch gestures.

When a user is inconvenient to directly contact the touch screen with a hand, how the user operates the terminal to use various functions provided by the terminal becomes an urgent problem to be solved.

Disclosure of Invention

The application provides a handheld mobile terminal control method and a related device, when a user turns off or lights a screen of a terminal, the user can input user gestures to enable the terminal to execute corresponding operations, so that the terminal can be controlled more intelligently, and user experience is improved.

In a first aspect, the present application provides a method for operating a handheld mobile terminal, where the method is applied to a handheld mobile terminal, the handheld mobile terminal has a display screen and a camera, and the method may include: the terminal acquires a first image through a camera, and the camera is kept on when a display screen of the terminal is turned off or turned on so as to acquire the image; the terminal analyzes whether the first image comprises a valid user gesture; in a case where a valid user gesture is included in the first image, the terminal performs an operation corresponding to the valid user gesture.

By implementing the method of the first aspect, the user can input the user gesture to enable the terminal to execute the corresponding operation when the terminal is turned off or on, so that the terminal can be controlled more intelligently, and the user experience is improved.

In the method of the first aspect, the turning off of the display screen means that the display function of the touch screen is fully or partially turned off, and when the terminal is not turned off, the terminal is turned on.

Whether the display is lit and whether the terminal is locked are independent of each other. That is, in some embodiments, the display screen may also be in the locked or unlocked state while the display screen is off. In some embodiments, when the display screen is on, the display screen can be in a locked state or an unlocked state at the same time.

In the method of the first aspect, the camera is based on the terminal architecture provided by the application, and low power consumption and constant opening can be realized. For the terminal architecture provided in the present application, reference may be made to the related description of the second aspect. The camera is used for collecting images, and the images can comprise static images or dynamic videos, three-dimensional stereo images or plane images.

In the method of the first aspect, the valid user gesture is a gesture for controlling the terminal to execute a corresponding operation, and the gesture unrelated to the control of the terminal to execute the corresponding operation is an invalid user gesture.

In conjunction with the first aspect, in some embodiments, the valid user gesture may be set by the terminal default or may be set by the user. For example, a valid user gesture may be a default setting at the time of terminal factory shipment. For another example, the user may select a user gesture as an effective user gesture from a setup menu provided by the terminal that includes a variety of user gestures.

In connection with the first aspect, the valid user gestures may be static or dynamic. In some embodiments, valid user gestures may include: palm, back of hand, swipe gesture, gesture to draw a pattern or letter. Further, the valid user gesture may be only a finger motion, or may be a combination of a finger motion with a duration and a distance. For example, the user gesture may be a hand back provided by the user, or a time period in which the user provides the hand back exceeds a specified time period.

With reference to the first aspect, in some embodiments, the operation corresponding to the valid user gesture is set by default by the terminal or set by the user.

In combination with the first aspect, in some embodiments, the operation corresponding to the valid user gesture includes at least one of: and executing unlocking operation, lighting up a screen, turning off the screen, turning up/down the volume, turning up/down the screen brightness, and turning on/off the flight mode, and performing screen capture operation on the current screen display content, starting an application or starting an internal function of the application.

With reference to the first aspect, in some embodiments, before the terminal performs an operation corresponding to a valid user gesture, it may be verified whether the current user has an authority to use the terminal. And under the condition that the current user has the authority to use the terminal, the terminal and the effective user gesture correspond to each other.

In the above embodiment, the terminal may verify whether the current user has the right to use the terminal by: the terminal passes through a second image of the camera; recognizing the face of the user in the second image; and if the face of the user in the second image is matched with face information prestored by the terminal, determining that the current user has the authority to use the terminal. Whether the user has the authority to use the terminal is verified in a face verification mode, and the terminal can be guaranteed to be used by the user with the authority, so that the data safety of the terminal is guaranteed.

In some embodiments, if the face of the user in the second image does not match the face information pre-stored in the terminal, that is, the user does not pass the face authentication, the terminal may prompt the user to perform the second permission authentication. Exemplarily, the terminal may prompt the user to input authentication information, and if the authentication information input by the user matches with pre-stored authentication information, it is determined that the current user has an authority to use the terminal; and if the verification information input by the user is not matched with the pre-stored verification information, determining that the current user does not have the authority to use the terminal. The authentication information includes any one of: passwords, irises, fingerprints, and voices; wherein the pre-stored authentication information corresponding to the sound includes: semantic information of the sound and/or voiceprint information of the sound. When the face verification fails, the terminal prompts the user to perform secondary verification, so that the user with the authority can use the terminal.

In conjunction with the first aspect, in some embodiments, the terminal may invoke an AI algorithm to analyze whether a valid user gesture is included in the first image.

In some embodiments, when the terminal invokes the AI algorithm to analyze whether the first image includes a valid user gesture, the first image may be input into the AI algorithm model, and the AI algorithm model invokes a corresponding operator in the AI operator library to analyze whether the first image includes a valid user gesture.

In some embodiments, the AI operator library is solidified in the hardware of the terminal.

In a second aspect, the present application provides a handheld mobile terminal, comprising: the system comprises a coprocessor, a main processor, a camera and a display screen, wherein the coprocessor is connected with the camera and the main processor. Wherein:

the coprocessor is used for acquiring a first image through the camera, and the camera is kept on to acquire the image when the display screen is off or on; analyzing whether a valid user gesture is included in the first image; under the condition that the first image comprises the effective user gesture, sending indication information of the effective user gesture to the main processor;

and the main processor is used for executing the operation corresponding to the effective user gesture according to the indication information.

In conjunction with the second aspect, in some embodiments, the valid user gesture may be set by the terminal default or may be set by the user. For example, a valid user gesture may be a default setting at the time of terminal factory shipment. For another example, the user may select a user gesture as an effective user gesture from a setup menu provided by the terminal that includes a variety of user gestures.

In connection with the second aspect, the effective user gesture may be static or dynamic. In some embodiments, valid user gestures may include: palm, back of hand, swipe gesture, gesture to draw a pattern or letter. Further, the valid user gesture may be only a finger motion, or may be a combination of a finger motion with a duration and a distance. For example, the user gesture may be a hand back provided by the user, or a time period in which the user provides the hand back exceeds a specified time period.

In combination with the second aspect, in some embodiments, the operation corresponding to the valid user gesture is set by default by the terminal or set by the user.

In combination with the second aspect, in some embodiments, the operation corresponding to the valid user gesture includes at least one of: and executing unlocking operation, lighting up a screen, turning off the screen, turning up/down the volume, turning up/down the screen brightness, and turning on/off the flight mode, and performing screen capture operation on the current screen display content, starting an application or starting an internal function of the application.

With reference to the second aspect, in some embodiments, the main processor is further configured to verify whether the current user has permission to use the terminal before performing the operation corresponding to the valid user gesture. The main processor is specifically configured to, when the current user has an authority to use the terminal, execute an operation corresponding to the valid user gesture.

In the above embodiment, the main processor may be configured to verify whether the current user has the right to use the terminal by: recognizing the face of the user in the second image; if the face of the user in the second image is matched with face information prestored by the terminal, determining that the current user has the authority to use the terminal; wherein the second image is acquired by the camera.

With reference to the second aspect, in some embodiments, the main processor is further configured to prompt the user to input the verification information if the face of the user in the second image does not match the face information pre-stored in the terminal; if the verification information input by the user is matched with the pre-stored verification information, determining that the current user has the authority to use the terminal; and if the verification information input by the user is not matched with the pre-stored verification information, determining that the current user does not have the authority to use the terminal. The authentication information includes any one of: passwords, irises, fingerprints, and voices; wherein the pre-stored authentication information corresponding to the sound includes: semantic information of the sound and/or voiceprint information of the sound.

In combination with the second aspect, in some embodiments, the co-processor is specifically configured to invoke an AI algorithm to analyze whether a valid user gesture is included in the first image.

In some embodiments, the coprocessor may include: the system comprises an AI engine module, an AI algorithm model, an AI operator library and an AI application layer module. Wherein:

the AI engine module is used for calling a corresponding AI algorithm in the AI algorithm model to carry out AI calculation on the first image;

the AI algorithm model is used for calling a corresponding AI operator in the AI operator library to analyze whether the first image comprises an effective user gesture or not and reporting an identification result to the AI application layer module;

the AI application layer module is used for reporting the indication information of the effective user gesture to the main controller under the condition that the first image comprises the effective user gesture.

In some embodiments, the AI operator library is solidified in the hardware of the coprocessor.

In a third aspect, the present application provides a chip, where the chip is applied to a handheld mobile terminal, the handheld mobile terminal includes a camera and a display screen, and the chip includes: the coprocessor is connected with the main processor. Wherein:

In a fourth aspect, a computer-readable storage medium is provided, which has instructions stored thereon, and when the instructions are executed on a computer, the instructions cause the computer to execute the handheld mobile terminal control method described in the first aspect.

In a fifth aspect, a computer program product containing instructions is provided, which when run on a computer causes the computer to perform the method of handling a handheld mobile terminal as described in the first aspect above.

By implementing the method and the device, the user can enable the terminal to execute corresponding operation by inputting the user gesture when the terminal is turned off or on, so that the terminal can be controlled more intelligently, and the user experience is improved.

Drawings

FIGS. 1-10 are schematic diagrams of human-computer interaction provided herein;

fig. 11 is a schematic flowchart of a method for operating a handheld mobile terminal according to the present application;

fig. 12 is a schematic structural diagram of the terminal 100 provided in the present application;

fig. 13 is a schematic diagram illustrating cooperation of various components when the terminal 100 recognizes a gesture of a user and illuminates a screen.

Detailed Description

The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application.

In the description of the embodiments herein, "/" means "or" unless otherwise specified, for example, a/B may mean a or B; "and/or" herein is merely an association describing an associated object, and means that there may be three relationships, e.g., a and/or B, which may mean: a exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of the present application, "a plurality" means two or more than two.

In the following, the terms "first", "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of the embodiments of the present application, "a plurality" means two or more unless otherwise specified.

The application is applied to the terminal. The type of the terminal is not specifically limited in the present application, and the terminal may be a mobile phone, a tablet computer, a Personal Digital Assistant (PDA), a wearable device, or other portable electronic devices. Examples of portable electronic devices include, but are not limited to, portable electronic devices that carry an iOS, android, microsoft, or other operating system. The portable electronic device may also be other portable electronic devices such as laptop computers (laptop) with touch sensitive surfaces (e.g., touch panels), etc. It should also be understood that in other embodiments of the present application, the terminal may not be a portable electronic device, but may be a desktop computer having a touch-sensitive surface (e.g., a touch panel).

In the application, the terminal is configured with a touch screen and can be used for displaying the interface content currently output by the system of the terminal. The interface content may include an interface of an application program being run, a system level menu, and the like, and may specifically include the following interface elements: input type interface elements such as a button (button), a text entry box (text), a slider Bar (scroll Bar), a menu (menu), and the like; and output type interface elements such as windows, tabs, etc.

To facilitate understanding of the terminal manipulation method of the present application, the AI technique to which the present application is applied is described below.

The AI technique is a theory, method, technique and application system that uses a digital computer or a machine controlled by a digital computer to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use the knowledge to obtain the best results. In other words, artificial intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a manner similar to human intelligence. Artificial intelligence is the research of the design principle and the realization method of various intelligent machines, so that the machines have the functions of perception, reasoning and decision making. Research in the field of artificial intelligence includes robotics, natural language processing, computer vision, decision and reasoning, human-computer interaction, recommendation and search, AI basic theory, and the like.

With the development of the AI artificial intelligence technology, the AI technology is applied to the terminal more and more widely, so that the functions of the terminal are more and more intelligent. For example, as AI technology is popularized in terminals, the terminals have more and more powerful functions in the sensing field, the image processing field, the audio processing field, the language processing field, and the like.

In the prior art, an AI function is integrated in a software system of a terminal, and the AI function basically needs to be activated by a certain action of a user or the triggering of other application modules. When the terminal has service requirement, the corresponding application module will call the corresponding AI function. In other words, in the prior art, the AI function of the terminal is not continuously open, and in many cases, the terminal does not sense the user behavior, the behavior intention, the environmental change, and the like through the continuous open of the AI function.

In this application, the partial AI function at terminal can the low-power consumption continuously normally open, independently utilizes AI perception ability, and the terminal can be in real time through the change of AI technical perception user's intention to carry out corresponding function, make the terminal more intelligent, the man-machine experience is more comfortable. How the terminal implements the low-power-consumption continuous normally-open AI function can be referred to in the following description of the device embodiments, and will not be described herein again.

According to the terminal control method, the terminal can be controlled without a user contacting a touch screen of the terminal by utilizing the AI function of low power consumption continuous open of the local terminal of the terminal. The AI function of the local terminal of the terminal refers to each processing capability of the terminal based on the AI technology. Illustratively, the terminal local-end low-power-consumption constantly-on AI function may include an image processing function, an audio processing function, and the like.

And the terminal can complete the image processing task by using the low-power-consumption continuous and normally-open image processing function. The image processing tasks may include: image gesture recognition, image user hand print (fingerprint or palm print, etc.) recognition, image face expression recognition, image figure iris recognition, image semantic segmentation, etc. The image here may include a still image or a moving video, a three-dimensional stereoscopic image, or a planar image. The low-power-consumption continuous constant opening of the image processing function means that the terminal can complete the image acquisition and processing tasks with low power consumption. The image acquisition can be realized by a low-power-consumption camera of the terminal, and the image processing task can be realized by a low-power-consumption processor of the terminal. Here, how to implement the low-power-consumption continuous normally-on image processing function for the terminal may refer to the following description of the embodiment of the apparatus, which is not repeated herein.

And the terminal can complete the audio processing task by using the low-power-consumption continuous and normally-open audio processing function. The audio processing tasks may include: voiceprint recognition, audio understanding, audio semantic segmentation, and the like. Wherein audio understanding refers to semantic understanding of audio. The audio processing function is low in power consumption and is continuously and constantly opened, and the audio processing function means that the terminal can finish the tasks of collecting and processing audio with low power consumption. The audio acquisition can be realized through a low-power microphone of the terminal, and the audio processing task can be realized through a low-power processor of the terminal. Here, how to implement the low-power consumption continuous normally-on audio processing function in the terminal may refer to the following description of the embodiment of the apparatus, which is not repeated herein.

It should be noted that the application of the low-power-consumption camera or the low-power-consumption AI function is in consideration of saving power consumption, and in an embodiment that does not consider power consumption, a conventional camera may also be used, which is not limited in the present invention.

Before the terminal manipulation method of the present application is introduced, a method for manipulating a terminal through a hover gesture in the prior art is first introduced. In the prior art, the terminal is in a bright screen and unlocked state, and when the terminal enters some specific applications, a user can be supported to control the terminal through gestures. For example, when the terminal is on the screen and unlocked, and enters a reading application, the terminal may turn on the camera to capture a user gesture, and perform an operation corresponding to the user gesture, such as turning a page. In the prior art, a terminal must be in a bright screen and unlocked state, and a user can only operate the terminal through a gesture of the user, which cannot meet the requirement that the user wants to operate the terminal through the gesture in other scenes (for example, when the terminal is turned off or locked).

The terminal control method provided by the present application is described in detail below with reference to the accompanying drawings and application scenarios.

Application scenario 1: the terminal has a low-power-consumption continuous normally-open image processing function, and a user controls the terminal through a user gesture.

In this application, the user gesture may be a gesture that does not contact the touch screen, or may be a gesture that contacts the touch screen. The user gesture may be a static gesture or a dynamic gesture. Illustratively, the user gesture may be a hand gesture within a certain range of the terminal periphery. Illustratively, the user gesture may be a hand gesture in the space above the touch screen.

In the application, the operation executed by the user gesture capable of controlling the terminal can be set by default by the terminal, and can also be opened to the user in a setting module of the terminal and freely configured by the user. That is, in some embodiments, the user gesture can only manipulate the terminal to perform a partial operation.

The terminal may be configured with a low power consumption camera. The low-power consumption camera can be opened always at a specific frame rate, and images around the terminal are collected in real time, namely, the camera shoots and shoots images at a certain frequency. It should be understood that the low-power-consumption camera can be used as an infrastructure to continuously acquire images around the terminal, and provides a hardware basis for the terminal to realize an image processing function based on an AI technology.

In the application, the camera is kept on, or is always on, namely the camera shoots at a certain frequency to obtain image data. The frequency may be different when the terminal where the camera is located is in different application scenarios.

For example, in one embodiment of the present invention, the camera may be kept on when the screen is turned off or turned on, or may be kept on when both the screen is turned off and turned on, and the frequency of acquiring images when the screen is turned off is different from the frequency of acquiring images when the screen is turned on.

The method for the user to operate and control the terminal through the user gesture in the application scenario 1 is described below in conjunction with different states of the terminal. The state of the terminal includes: whether the screen of the terminal is lit, whether the terminal is locked, etc. The respective states of the terminal are described below.

In some embodiments, the terminal may be in a screen-off state. The terminal is in the screen-off state, which means that the display function of the touch screen is completely or partially closed. The display function of the touch screen is completely closed, which means that the touch screen does not display any interface content and does not consume electric quantity to emit light. The display function of the touch screen is partially turned off, namely that a part of the area of the touch screen can display some interface elements (such as time, calendar and notification message), the part of the area consumes a small amount of electricity to emit light, and the rest of the area of the touch screen does not consume the electricity to emit the light. For example, the touch screen is an organic light-emitting diode (OLED) display screen, a part of pixel points on the OLED display screen are lit for displaying time, and when the remaining pixel points are in an off state, the terminal can still be regarded as being in an off state.

In some embodiments, the terminal may be in a bright screen state. The screen lightening state and the screen extinguishing state of the terminal are opposite, and when the terminal is not in the screen extinguishing state, the terminal is in the screen lightening state.

In some embodiments, the terminal may be in a lock screen state. The terminal is in a screen locking state, that is, part of functions of the terminal are locked and cannot be used, and the locked functions can be used only after a user inputs a password or unlocks the terminal through a preset unlocking mode. When the terminal is in the screen locking state, if the touch screen is lightened, the touch screen can display part of the content. For example, the touch screen may display a status bar, date, time, lock screen, etc. The status bar under the lock screen may include a name of an operator (e.g., china mobile), a Wi-Fi icon, signal strength, current remaining power, a bluetooth icon, an alarm clock icon, a lock screen icon, and the like. When the terminal is in the screen locking state, the requirements of the available functions on data safety are low. Exemplary functions available in the lock screen state may include: answering a call, hanging up a call, playing music, pausing the playing of music, adjusting the volume of music, starting a camera application, turning on/off a flight mode, and the like. When the terminal is in the screen locking state, the requirement of the locked and unusable functions on data safety is higher. Exemplary functions that are not available in the lock screen state may include: launching applications (e.g., instant messaging applications, shopping applications, file management applications, galleries, etc.), and functions within applications (e.g., sending messages, purchasing goods, deleting pictures in galleries, etc.) and the like.

It will be appreciated that whether the screen is lit and whether the terminal is locked are independent of each other. When the terminal is in the screen-off state, the terminal may be in the screen-locking state at the same time, and may also be in the unlocking state. When the terminal is in the screen locking state, the terminal may be in the screen off state or the screen lightening state at the same time.

Because the low-power consumption camera of terminal is opened always, no matter what state the terminal is in, the low-power consumption camera all can gather the peripheral image of terminal in real time. The terminal can analyze whether a specific event occurs according to the collected image. Further, the terminal can call a corresponding AI algorithm, analyze the acquired image and determine whether a specific event occurs. When a specific event is detected to occur, the terminal executes an operation corresponding to the specific event, and the user finishes the operation of the terminal. The following describes a control method of the terminal in different states by taking a specific event as a specific user gesture as an example.

When the terminal is in the screen-off and screen-locking state, a user can control the terminal through a user gesture.

When the terminal is turned off and the screen is locked, the user can light the screen through the gesture of the user but does not unlock the screen.

As an example, referring to the left drawing of fig. 1, the terminal 100 is in the off-screen and locked state, the user's hand is suspended in the space above the touch screen, and the back of the user's hand faces the touch screen. The low power consumption camera of the terminal 100 is always on to collect images in real time. After the terminal 100 acquires the image, a gesture recognition algorithm may be called to analyze the image, and it is determined that the user gesture input by the user in the image is: the user's hand floats in the space above the touch screen, and the back of the hand faces the touch screen. Illustratively, referring to the right drawing of fig. 1, after the terminal 100 determines that the user provides the back of the hand, the touch screen may be illuminated without unlocking. As shown in the right hand drawing of fig. 1, the touch screen is illuminated but not unlocked, and the touch screen may display a status bar 110, a date and time 120, a lock screen 130, and the like. The status bar 110 under the lock screen may include the name of the operator (e.g., china mobile), Wi-Fi icon, signal strength, battery icon, and remaining power, lock screen icon, etc. In some embodiments, the status bar 110 may also include an alarm icon, a bluetooth icon, and the like. When the terminal 100 receives a notification message (e.g., an instant messenger application message), the touch screen may also display the notification message.

In some embodiments, to prevent misoperation, the terminal 100 may also determine whether to light the screen but not unlock in conjunction with the length of time the user provides the back of the hand. For example, if the user provides the back of the hand for a period of time exceeding a specified period of time (e.g., 1 second), the terminal 100 determines to light the screen but not to unlock.

In some embodiments, terminal 100 can also determine whether to illuminate the screen without unlocking the lock in conjunction with a change in the distance between the user's back of hand and the touch screen. For example, if the distance between the back of the user's hand and the touch screen is greater and greater, the terminal 100 determines to light the screen but not to unlock. The terminal 100 may detect a change in the distance between the back of the user's hand and the touch screen by: (1) the terminal 100 captures a plurality of images through a low power consumption camera and analyzes a change in the size of the user's back of hand in the plurality of images. If the back of the user's hand becomes smaller in the plurality of images, it can be confirmed that the distance between the back of the user's hand and the touch screen becomes larger. (2) After the terminal 100 determines that the user provides the back of the hand, a change in the distance between the back of the user's hand and the touch screen may be measured by the infrared ray sensor.

In some embodiments, the terminal 100 may also determine whether to light the screen but not unlock in conjunction with the length of time the user provides the back of the hand and the distance between the back of the hand and the touch screen. For example, if the duration in which the vertical distance between the back of the user's hand and the touch screen is less than the preset threshold exceeds the preset length, the terminal determines to light the screen but not to unlock. That is, the terminal may require the user's back of the hand to stay at a position close to the touch screen for a certain time, which may prevent a malfunction. The terminal 100 may detect a time period in which the user provides the back of the hand by: the terminal 100 records a duration for the low-power-consumption camera to continuously shoot the back of the hand of the user, and determines the duration to be a duration for providing the back of the hand for the user. The manner in which the terminal 100 detects the change in the distance between the back of the user's hand and the touch screen may refer to the description of (1) or (2) in the above-described embodiment.

It should be noted that the user gesture for lighting the screen without unlocking is not limited to the back of the hand in the embodiment of fig. 1, and may be other user gestures. For example, the user's fingers suspended above the touch screen may be curved around the finger root, or the user's five fingers suspended above the touch screen may be held to spread. The user gesture may be only a finger motion, or may be a combination of a finger motion and a duration and a distance. For example, the user gesture may be that the user's thumb is hovering over the touch screen for more than a specified length of time, or that the distance between the user's thumb and the touch screen is within a specified range.

By lighting up the screen by a user gesture but not unlocking the lock, the user can view information under the lock screen, such as date, time, notification messages, and the like. At this moment, the terminal is still in the screen locking state, and the data security of the terminal can be ensured.

And secondly, when the terminal is in a bright screen state, the user can control the terminal through the user gesture.

When the terminal is in the bright screen state, the terminal can also be in the screen locking state at the same time, or the terminal can also not be locked, and the application is not limited.

When the terminal is in a bright screen state, the user can extinguish the screen through the gesture of the user.

In the following, when the terminal is on and unlocked, the user turns off the screen by the user gesture.

As an example, referring to the left drawing of fig. 2, the terminal 100 is in a bright-screen and unlocked state, and the touch screen may display one or more of a status bar 210, a desktop 220 for displaying installed icons of the terminal, and a navigation bar 230. The status bar in the unlocked locked state may include one or more of the following: name of operator (e.g., china mobile), Wi-Fi icon, signal strength, battery icon, and remaining power, time alarm icon, and bluetooth icon. The navigation bar 230 may include: a return key icon (triangle in the figure), a home screen icon (circle in the figure), a multitasking key icon (square in the figure). The user's hand is hovering over the touch screen with the palm of the user's hand (which may also be referred to as the palm) facing the touch screen. The low power consumption camera of the terminal 100 is always on to collect images in real time. After the terminal 100 acquires the image, a gesture recognition algorithm may be called to analyze the image, and it is determined that the user gesture input by the user in the image is: the user's hand is suspended in the space above the touch screen with the palm facing the touch screen. Illustratively, referring to the right drawing of fig. 2, after the terminal 100 determines that the user provides the palm of the hand, the touch screen may be extinguished. The terminal is on and unlocked, and when the user turns off the screen through the user gesture, the terminal may lock the screen at the same time, or may automatically lock the screen after waiting for a specified time (e.g., 2 seconds), which is not limited herein.

Understandably, when the terminal is on screen and locked, the user can extinguish the screen through the gesture of the user. The user gesture for turning off the screen when the terminal is on and locked may be the same as the user gesture for turning off the screen when the terminal is on and unlocked. Here, the screen-on and screen-locking state of the terminal may be obtained by the user through the method shown in fig. 1, or may be obtained by the user pressing a power key of the terminal, or may be obtained by the terminal automatically locking the screen without receiving the user input for a long time but not turning off the screen.

When the terminal is in the bright screen state, in some embodiments, to prevent a malfunction, the terminal 100 may further determine whether to turn off the screen in combination with a duration of the palm provided by the user. For example, if the duration for which the user provides the palm of the hand exceeds a specified duration (e.g., 1 second), the terminal 100 determines to extinguish the screen.

In some embodiments, terminal 100 may also determine whether to turn off the screen in conjunction with a change in the distance between the user's palm and the touch screen. For example, if the distance between the palm of the user's hand and the touch screen becomes smaller and smaller, the terminal 100 determines to extinguish the screen. The terminal 100 detects a change in the distance between the palm of the user's hand and the touch screen in the same manner as the terminal 100 detects a change in the distance between the back of the user's hand and the touch screen when in the above-described first state, and reference may be made to the related description.

In some embodiments, the terminal 100 may also determine whether to extinguish the screen in combination with the duration of time the user provided the palm of the hand and the distance between the palm of the hand and the touch screen. For example, if the duration in which the vertical distance between the palm of the user's hand and the touch screen is less than a preset threshold exceeds a preset length, the terminal determines to turn off the screen. That is, the terminal may require the palm of the user's hand to stay at a position close to the touch screen for a certain period of time, which may prevent a malfunction. The terminal 100 detects the duration of the palm provided by the user in the same manner as the terminal 100 detects the duration of the back provided by the user in the first state (described above), and reference may be made to the related description. The terminal 100 detects a change in the distance between the palm of the user's hand and the touch screen in the same manner as the terminal 100 detects a change in the distance between the back of the user's hand and the touch screen when in the above-described first state, and reference is made to the related description.

It should be noted that the user gesture for turning off the screen is not limited to the palm in the embodiment of fig. 2, and may be other user gestures. For example, the user may extinguish the screen using a dynamic user gesture that changes from five fingers open to five fingers closed.

And thirdly, when the terminal is in a screen locking state, the user can control the terminal through the user gesture.

When the terminal is in the screen locking state, the terminal can also be in the screen extinguishing state at the same time, or can also be in the screen lightening state at the same time.

(1) When the terminal is in a screen locking state, a user can control the terminal to execute the unlocked function through a user gesture. The unlocked function refers to a function that the terminal can still use when the screen is locked. The functions that the terminal can still use under the lock screen include but are not limited to: answering a call, hanging up a call, playing music, pausing the playing of music, adjusting the volume of audio, starting a camera application, and the like.

In the following, a description is given by taking an example in which when the terminal is in a screen-locked and screen-bright state, the user operates the terminal to execute an unlocked function through a user gesture.

As an example, referring to fig. 3a of fig. 3, the terminal 100 is in a lock and light state, and the touch screen may display a status bar 110, a date and time 120, a lock screen 130, a notification message 140 (e.g., an instant messenger application message) received by the terminal 100, and the like. The user hands are suspended above the touch screen, and an O-shaped pattern is drawn through fingers or put out through the hands. The low power consumption camera of the terminal 100 is always on to collect images in real time. After the terminal 100 acquires the image, a gesture recognition algorithm may be called to analyze the image, and it is determined that the user gesture input by the user in the image is: the user's hand is suspended in the space above the touch screen, and the finger draws an "O" shaped pattern, or the hand swings out an "O" shaped pattern. Illustratively, referring to 3b of FIG. 3, after the terminal 100 determines that the user provides the "O" shaped pattern by hand, the camera application may be launched. Here, since the photos in the gallery belong to data with high privacy, the preview window in the lower left corner of the touch screen may not display the photos previously taken by the user after the camera application is started.

After the terminal 100 starts the camera application, the user may also manipulate the terminal 100 to perform a part of functions inside the camera by a user gesture. This part of the functionality is the functionality that can still be used when the terminal is locked, e.g. taking a picture. For example, the user's hand may be suspended above the touch screen, and the terminal may be operated to take a picture (corresponding to pressing the shutter key) by a gesture of pinching two or more fingers. Illustratively, referring to 3c of fig. 3, a photograph taken by the terminal may be displayed in the preview window in the lower left corner of the touch screen.

As another example, referring to fig. 4a and 4b, 4c of fig. 4, the terminal 100 is in a lock screen and bright screen state. The user's hand may hover over the space above the touch screen and handle the communication message 140 received by the terminal 100 through a particular gesture. For example, as shown, the user may input a user gesture in which a finger slides to the left. The low-power-consumption camera of the terminal 100 may collect an image in real time, and may call a gesture recognition algorithm after collecting the image, to determine that a user gesture input by a user in the image is: the finger slides to the left. As shown in fig. 4b, after the terminal 100 determines that the user inputs a gesture in which the finger slides to the left, the view control 141 and the cancel control 142 of the communication message 140 may be displayed.

In some embodiments, the user may also input a user gesture simulating the click cancellation control 142, and after the terminal 100 confirms that the user inputs the user gesture simulating the click cancellation control 141 through image analysis, the corresponding notification message may be cancelled to be displayed on the currently displayed interface content, for example, as shown in fig. 4 c. The user may also enter a user gesture that simulates clicking on the view control 141, and the terminal 100 may display an interface that prompts the user to unlock (e.g., a password unlock interface), which may allow the user to view a particular notification message after unlocking. When the terminal 100 analyzes the user gesture input by the user to simulate the click cancel control 141 through the image, it may be determined that the touch focus simulated by the user is on the cancel control 141 of the touch screen, and then it is determined that the simulated click operation is performed by the finger of the user at the touch focus. The touch focus is a point projected onto the touch screen by a user fingertip, and the terminal 100 may detect the user fingertip by image analysis or infrared detection, and determine the touch focus projected onto the touch screen by the user fingertip. After determining the touch focus, the terminal 100 may analyze several images collected continuously, and if the distance between the user's finger and the touch screen is closer and closer within a preset time period, it may be seen that the user's finger performs a simulated click operation on the touch focus. Here, the preset time period may be set in advance, for example, may be set to 1 second.

In addition to canceling/viewing the notification message 140, other processing of the message notification 140 may be performed by user gestures. For example, the user may click on the notification message 140 via a user gesture, and the terminal 100 may display an interface that prompts the user to unlock (e.g., a gesture unlock interface). For another example, when the terminal 100 displays a plurality of notification messages 140, the user may slide the notification messages 140 up and down by a user gesture, and the terminal 100 may slide the plurality of notification messages 140 along with the user gesture.

As still another example, referring to the left drawing of fig. 5, the terminal 100 is in a lock screen and bright screen state, the terminal 100 currently receives a voice call over a GSM network, and the touch screen displays a screen as shown in the left drawing. The user may enter a user gesture that slides to the right after seeing the screen as shown in the left drawing. The low-power-consumption camera of the terminal 100 may collect an image in real time, and may call a gesture recognition algorithm after collecting the image, to determine that a user gesture input by a user in the image is: the finger slides to the right. As shown in the right drawing of fig. 5, after the terminal 100 determines that the user inputs the gesture of sliding the finger to the right, the voice call can be accepted, and the user can talk with the other party. Similarly, the user may input a user gesture that slides to the left to reject the voice call after viewing the screen shown in the left drawing of fig. 5.

The above embodiments of fig. 4 to 5 describe the case where the user operates the terminal to execute the unlocked function through the user gesture when the terminal is in the screen-locked and screen-on state, and it can be understood that the user can also operate the terminal to execute the unlocked function through the user gesture when the terminal is in the screen-locked and screen-off state. In some embodiments, when the terminal receives a user gesture for controlling the terminal to execute an unlocked function in the screen-locked and screen-off states, the terminal may first light up the screen and then execute the unlocked function. The gesture for controlling the terminal to execute the unlocked function when the terminal is locked and the screen is off can be the same as the gesture for controlling the terminal to execute the unlocked function when the terminal is locked and the screen is on.

It should be noted that the user gesture for controlling the terminal to execute the unlocked function is not limited to the example in the embodiment in fig. 4 or fig. 5, and may be another user gesture. For example, during a call or media playing process, the user may draw the pattern "eight" by finger hovering to increase the volume of the audio played by the terminal, or the user may decrease the volume of the audio played by the terminal by a user gesture of double or multi-finger pinch. For another example, during the media playing process, the user may pause the media playing by facing the palm to the touch screen, or the user may draw the letter "M" with fingers to continue the media playing, or the user may draw the number "3" to switch the song currently played by the terminal.

(2) When the terminal is in a screen locking state, a user can unlock the terminal through a user gesture.

In the following, the example is described in which when the terminal is in the screen-locked and screen-bright state, the user operates the terminal to unlock through the user gesture.

As an example, the terminal is in a screen lock and bright state, and the touch screen displays a screen as shown in the left drawing of fig. 6. The user's hand is suspended in the space above the touch screen, and an OK gesture is performed through the hand. The low power consumption camera of the terminal 100 is always on to collect images in real time. After the terminal 100 acquires the image, a gesture recognition algorithm may be called to analyze the image, and it is determined that the user gesture input by the user in the image is: an OK gesture. Illustratively, referring to the right drawing of fig. 6, after terminal 100 determines that the user provides an OK gesture, the lock may be released. As shown in the right drawing of fig. 6, after the terminal is unlocked, the touch screen may display a desktop for displaying icons of applications installed in the terminal, and may also display a status bar and a navigation bar. The user gesture for unlocking is not limited to the OK gesture shown in fig. 6, and may be another user gesture.

Understandably, when the terminal is in the screen locking and screen extinguishing state, the user can also control the terminal to unlock through the gesture of the user. In some embodiments, when the terminal is in the screen-lock and screen-off state and receives a user gesture for controlling the terminal to unlock, the screen may be simultaneously lit up or may remain off, which is not limited herein. The gesture for controlling the terminal to unlock when the terminal locks and turns off the screen can be the same as the gesture for controlling the terminal to unlock when the terminal locks and turns on the screen.

The process of unlocking the terminal is actually equivalent to verifying whether the current user is authorized to use the terminal. And if the current user has the right to use the terminal, unlocking the terminal. The person who has the right to use the terminal can be the owner or a person who is in close relationship with the owner. Because the user gesture is not unique, a person who does not have the right to use the terminal can also give the user gesture for unlocking, and in order to ensure the safety of data and avoid the person who does not have the right to use the terminal from unlocking the terminal and checking the data of the terminal, other unlocking modes are provided.

In some embodiments, unlocking may be by a human face. Specifically, the user can provide the face to the low power consumption camera/normal power consumption camera at the terminal, and the camera collects images in real time. The terminal can call a face recognition algorithm, compare the user face in the collected image with a pre-stored user face (the user face pre-stored by the owner), and determine whether the face in the image is matched with the pre-stored face. If the face in the image is matched with a pre-stored face, the terminal can be unlocked.

In some embodiments, the unlocking may be by biometric means. The biological characteristics include: iris, palm print, fingerprint, etc. Specifically, the user can provide biometric features to the low power consumption camera/normal power consumption camera, which collects images in real time. The terminal can call an iris/palm print/fingerprint recognition algorithm to compare the biological characteristics in the collected image with the pre-stored biological characteristics. Besides collecting the biological characteristics through the camera, the terminal can also collect the biological characteristics through other modes, for example, the terminal can collect fingerprints through a fingerprint sensor, and the like. The terminal may unlock if the collected biometric matches a pre-stored biometric.

After the terminal is unlocked, a user can operate the mobile phone through a touch screen gesture (namely a gesture of directly contacting the touch screen) or a user gesture or voice. For example, when a user manipulates a mobile phone through a user gesture, a user may take a picture, send information, make a call, open a certain sub-interface of an application (e.g., a payment interface of a payment treasure), manage a file, and the like through the gesture.

In some embodiments, after the terminal is unlocked, the user may manipulate the terminal through a user gesture. These user gestures include, but are not limited to: provide a palm/back gesture, a tap gesture, a press gesture, a swipe gesture, and the like.

(3) When the terminal is in a screen locking state, a user can control the terminal to execute the locked function through a user gesture. The locked function refers to a function that the terminal cannot be used in a screen-locked state. The functions that the terminal cannot be used in the screen locking state may include: launching applications (e.g., instant messaging applications, shopping applications, file management applications, galleries, etc.), and functions within applications (e.g., sending messages, purchasing goods, deleting pictures in galleries, etc.) and the like.

In the following, a description is given by taking an example that when the terminal is in a screen-locked and screen-bright state, the user operates the terminal to execute a locked function through a user gesture.

As an example, the terminal is in a lock screen and bright screen state, and the touch screen displays a screen as shown in the left drawing of fig. 7. The user's hand is hovering over the touch screen and draws the letter "W" with the finger. The terminal 100 may analyze the captured image and determine that the user gesture input by the user in the image is: the letter "W" is drawn. Illustratively, referring to the right drawing of FIG. 7, after the terminal 100 determines that the user draws the letter "W," it may unlock and launch an application (e.g., WeChat). That is to say, when the terminal is in the screen-locked and screen-lit state, the user can operate the terminal to unlock and start an application by drawing a user gesture of the letter "W". Illustratively, to secure data of the terminal 100, the terminal 100 may prompt the user to unlock after determining that the user draws the letter "W" and launch an application (e.g., WeChat) after the user contacts the locked state of the terminal 100. Here, the manner in which the terminal 100 prompts the user to unlock may include: an unlocking picture (such as a password unlocking picture, a gesture unlocking picture and the like) is displayed on the touch screen, the unlocking is prompted to a user through voice, the unlocking is prompted to the user through the flickering of a breathing lamp, and the unlocking is prompted to the user.

Understandably, when the terminal is in the screen locking and screen extinguishing state, the user can also control the terminal to execute the locked function through the gesture of the user. The gesture for controlling the terminal to execute the locked function when the terminal is locked and turned off may be the same as the gesture for controlling the terminal to execute the locked function when the terminal is locked and turned on. For example, when the terminal is in a screen-locked and screen-off state, a user can control the terminal to unlock, screen-up and start an application by drawing a user gesture of the letter "W".

It should be noted that, when the terminal locks the screen, the user gesture for manipulating the terminal to execute the locked function is not limited to drawing the letter "W" shown in fig. 7, and may be another user gesture. For example, the user may manipulate the terminal to launch a payment application (e.g., a payment treasure) by drawing a user gesture of the letter "Z", and the user may also launch a payment interface of the payment application (e.g., a payment treasure's payment interface) by drawing the letter "P".

With reference to the embodiments of fig. 1 to 7, the user gesture for manipulating the terminal may be set in the following ways:

(1) the user gesture for manipulating the terminal may be user-set.

The user may select a user gesture as a user gesture for manipulating the terminal from a setup menu provided by the terminal including a variety of user gestures. For example, in the FIG. 1 example, a user gesture to illuminate the screen (i.e., back of hand towards the touch screen) may be user selected from a setup menu. As another example, in the FIG. 2 example, the user gesture (i.e., palm facing the touch screen) to extinguish the screen may be user selected from a setup menu. In the fig. 4-7 example, the various user gestures may also be user selected from a setup menu.

The user can customize the user gesture for controlling the terminal according to own habits. In one possible embodiment, the user may input a user gesture to the terminal in advance and specify the role of the user gesture. The user gesture of drawing the letter "W" in fig. 7 is used to initiate the WeChat as an example. Illustratively, referring to fig. 8a, the setting interface 10 of the terminal 100 may provide an option "custom user gesture" 801, the user may directly contact the touch screen and click on the option 701, and the terminal 100 enters the interface 20 of the custom user gesture as shown in fig. 8 b. Interface 20 may provide custom gesture setting options for a number of operations, which may include, but are not limited to: executing unlocking operation, lighting up a screen, turning off the screen, turning up/down the volume, turning up/down the screen brightness, answering a call, hanging up the call, switching on/off the flight mode, performing screen capture operation on the current screen display content, starting an application or starting the internal function of the application. The application can be an application such as WeChat and Payment treasured, and the internal function of the application can be to open a payment interface of the Payment treasured and the like. The user may directly contact the touch screen and click on option 802 and terminal 100 enters recording interface 30 for the user's gesture as shown in fig. 8 c. The terminal 100 turns on the low power consumption camera/normal power consumption camera while displaying the recording interface 30 as shown in fig. 8 c. The user can input a user gesture for drawing the letter "W" according to his own habit, and the terminal 100 records the user gesture through the low power consumption camera/normal power consumption camera and stores the user gesture. After the terminal 100 stores the user gesture, the user completes the user gesture of the custom start WeChat. In the fig. 1-6 example, the individual user gestures may also be user-defined.

Understandably, the user gesture of the control terminal is set by the user, so that the use habit of the user is better met, and the user experience is better.

(2) The user gesture for manipulating the terminal may be terminal-set.

In some embodiments, the user gesture for manipulating the terminal may be a default setting when the terminal leaves the factory.

Not limited to the manners described in (1) - (2) above, the user gesture for manipulating the terminal may also be determined by other manners. For example, the user gesture for manipulating the terminal may also be randomly set by the terminal, and the present application is not limited thereto.

Further, in order to facilitate the user to manipulate the terminal through the user gesture, the application provides an operation prompt for prompting the user how to cause the terminal to perform a corresponding operation based on the user gesture. The following detailed description is made with reference to the accompanying drawings.

In some embodiments, the terminal may simulate the user's touch focus and display. Specifically, the terminal may simulate a point where a fingertip of the user is projected on the touch screen as a touch focus of the user, and the user may input a user gesture such as clicking, pressing, or sliding under the indication of the touch focus. Illustratively, referring to fig. 9, the point projected onto the touch screen by the user's fingertip is modeled as a touch focus 901, and the current touch focus 901 is located on a WeChat icon on the desktop displayed by the terminal. The touch focus 901 may be a circle. The user may input a user gesture under the indication of the touch focus, for example, a user gesture of clicking may be directly input to start the WeChat, or a finger may be moved to shift the touch focus 901. Illustratively, in conjunction with fig. 4b, the terminal may project the user's fingertip onto the touch screen, and the user may click the cancel control 142 to cancel the first message of the displayed notification messages 140 according to the current touch focus.

In some embodiments, the terminal may prompt the user how to manipulate the terminal through user gestures. Fig. 10 schematically illustrates an operation prompt provided by the present application. As shown in fig. 10, the terminal is currently opened for reading a book, and the touch screen may display a prompt message for a user gesture for prompting the user to turn a page. The reminder information may be a translucent palm 1001 displayed on the touch screen. After seeing the prompt information, the user can know that when the hand is changed from the palm to the touch screen to the back of the hand to the touch screen, the user can trigger the terminal to turn pages.

In the fig. 10 embodiment, the time when the translucent palm 1001 disappears may include: 1. after detecting a user gesture for controlling the terminal to turn pages (such as a user operation of changing a hand from palm to touch screen) to a user operation of changing the back of the hand to touch screen); 2. when detecting that the user finger contacts the touch screen (such as clicking the touch screen by the user finger); 3. when the display of the translucent palm 1001 is continued for a time exceeding a certain length of time (e.g., 2 seconds).

It is understood that the terminal is not limited to the examples in fig. 9 to 10, and the terminal may give the operation prompt in other scenarios. For example, in the examples of fig. 1-7, the terminal may give an operation prompt for prompting the user how to perform a corresponding operation using the terminal based on the user gesture.

According to the embodiment, different user gestures correspond to different operations. After recognizing the user gesture, the terminal may perform an operation corresponding to the user gesture.

In some embodiments, since the user gesture and the user do not have a unique corresponding relationship, in order to ensure data security of the terminal, it may be verified whether the current user has the right to use the terminal before performing an operation corresponding to the user gesture. After the current user is verified to have the right to use the terminal, the terminal can execute corresponding operations according to the recognized user gesture, such as lighting up the screen but keeping a locked state, turning off the screen, starting an application and the like.

Optionally, the terminal may verify whether the current user has the right to use the terminal through the face. Specifically, the terminal can acquire an image through a low-power-consumption camera and identify the face of a user in the image; and if the face of the user in the image is matched with face information stored in the terminal in advance, determining that the current user has the authority to use the terminal. Here, the terminal may previously store face information of a user who is authorized to use the terminal.

Further, if the face verification fails, that is, the face of the user in the image does not match the face information stored in the terminal in advance, the terminal may prompt the user to input verification information. That is, after the face authentication fails, the terminal may provide the user with a chance of secondary authentication. If the verification information input by the user is matched with the pre-stored verification information, determining that the current user has the authority to use the terminal; and if the verification information input by the user is not matched with the pre-stored verification information, determining that the current user does not have the authority to use the terminal.

The authentication information includes any one of: password, iris, fingerprint and voice. Wherein the pre-stored authentication information corresponding to the sound includes: semantic information of the sound and/or voiceprint information of the sound. Here, the terminal may previously store voiceprint information of a user who has authority to use the terminal, or the terminal may previously store voice semantic information for verifying the authority of the user. The voiceprint information of the user having the authority to use the terminal can be input into the terminal by the user, and the voice semantic information for verifying the authority of the user can be set by default by the terminal and can also be defined by the user.

The user passes the face verification or the terminal can execute the operation corresponding to the recognized user gesture after the user passes the secondary verification.

The terminal control method of the present application is described below by a specific example. In this example, the terminal performs face authentication on the user first, and after the current user is authenticated to have the authority to use the terminal, performs an operation corresponding to the recognized user gesture. Referring to fig. 11, in the embodiment of fig. 11, the terminal is unlocked by the owner identity of the face authentication device, and then the terminal is operated by the user gesture. As shown in fig. 11, the method may include the steps of:

1. the user places the face in an image acquisition area of the low-power-consumption camera, and the terminal acquires images through the low-power-consumption camera.

The terminal collects an image through a low-power-consumption camera, and the image can contain the face of a user.

2. And the terminal identifies the face contained in the acquired image through a face identification algorithm and verifies the identity of the owner.

In some embodiments, the terminal may first perform face detection on the acquired image through a face detection algorithm. If the detected facial contour features of the user in the image are complete and accord with the facial features, the appearance of a face in the image collected by the low-power-consumption camera can be judged.

The terminal can judge whether the face features in the collected image are matched with the prestored owner face image through a face recognition algorithm. If the identity of the owner is matched with the identity of the owner, the owner identity is successfully verified, and the subsequent steps are executed. If not, step 1 may be re-executed.

3. The user provides a user gesture, and the terminal collects an image through the low-power-consumption camera.

The terminal collects an image through a low-power-consumption camera, and the image can contain user gestures input by a user.

4. And the terminal identifies the gesture contained in the acquired image through a gesture identification algorithm.

In some embodiments, the terminal may first perform gesture detection on the captured image through a gesture detection algorithm. If the contour features in the image are detected to be in accordance with the contour features of the hands of the person, the gesture can be judged to appear in the image collected by the low-power-consumption camera.

The terminal may recognize a user gesture in the captured image through a gesture recognition algorithm. For example, the terminal may recognize that the user inputs a palm, back of hand, or OK gesture, etc. through a gesture recognition algorithm.

5. And the terminal executes corresponding operation according to the recognized gesture.

In some embodiments, the terminal may determine whether the user gesture in the captured image is a valid user gesture. And if so, executing corresponding operation according to the recognized gesture. If not, the terminal may re-execute step 3.

The valid user gesture is a gesture used for controlling the terminal to execute corresponding operation, and the gesture irrelevant to the operation of the control terminal is an invalid user gesture. The effective user gesture can be set by the default of the terminal or can be set by the user. In some embodiments, the terminal may store one or more valid user gestures in advance, and determine whether the captured image includes a valid user gesture according to the stored user gesture.

The following describes, with reference to fig. 1-7, an example in which the terminal performs a corresponding operation according to a valid user gesture.

For example, referring to the example of fig. 1, after the terminal recognizes that the user provides the back of the hand, the screen may be lit but the locked state is maintained.

For example, referring to the example of fig. 2, after the terminal recognizes that the user provides the palm of the hand, the screen may be extinguished.

For example, referring to the example of fig. 4, after recognizing that the user draws an "O" shaped pattern with a finger, the terminal may start a camera application; after recognizing the gesture of sliding left by the user, the terminal can control the notification message displayed on the touch screen.

For example, referring to the example of fig. 5, after receiving a voice call request, if a gesture that a user slides to the right is recognized, the terminal may receive the voice call.

For example, referring to the example of fig. 6, the terminal may unlock the screen upon recognizing that the user provides an OK gesture.

For example, referring to the example of fig. 7, the terminal may initiate a WeChat upon recognizing that the user draws the letter "W" with a finger.

In the application scenario 1, the terminal can acquire user images in real time through the low-power-consumption camera and can autonomously operate AI perception capability. When a user needs to use the terminal, the terminal can autonomously perform gesture recognition only by inputting a user gesture, and corresponding operation is executed according to the user gesture input by the user. In some embodiments, the user may not directly contact the touch screen manipulation terminal. According to the method and the device, the control of the terminal can be more intelligent, and the user experience is improved.

Application scenario 2: the terminal has a low-power consumption continuous normally-open audio processing function, and a user controls the terminal through voice.

In the application, the operation executed by the voice-controlled terminal can be set by default by the terminal, and can also be opened to a user in a setting module of the terminal and freely configured by the user. That is, in some embodiments, voice can only manipulate a portion of the operations performed by the terminal.

The terminal may be configured with a low power consumption microphone. The low-power-consumption microphone can be always started to collect audio data around the terminal in real time. It should be understood that the low power consumption microphone can be used as an infrastructure to continuously collect the audio around the terminal, and provide a hardware basis for the terminal to realize the audio processing function based on the AI technology.

Because the low-power consumption microphone of the terminal is always started, the low-power consumption microphone can acquire the audio around the terminal in real time no matter what state the terminal is in. The terminal can analyze whether a specific event occurs according to the collected audio. Further, the terminal can call a corresponding AI algorithm, analyze the collected audio, and determine whether a specific event occurs. When a specific event is detected to occur, the terminal executes an operation corresponding to the specific event, and the user finishes the operation of the terminal. The following describes the operation method in the application scenario 2 by taking a specific event as a specific voice as an example and combining different states of the terminal.

When the terminal is in the screen-off and screen-locking state, a user can control the terminal through voice.

As an example, when the terminal is off and locked, the user may input a voice "light the screen". And a low-power-consumption microphone of the terminal is always started to collect audio in real time. After the terminal collects the audio, the terminal can call a voice recognition algorithm to analyze the audio and determine that the voice input by the user is 'screen on'. After the terminal determines the voice input by the user, the touch screen can be lightened according to the voice without unlocking.

And secondly, when the terminal is in a bright screen state, the user can control the terminal through voice.

When the terminal is in a bright screen state, the user can extinguish the screen through voice.

As an example, the user may input a voice "turn off the screen" when the terminal is on the screen. And a low-power-consumption microphone of the terminal is always started to collect audio in real time. After the terminal collects the audio, the terminal can call a voice recognition algorithm to analyze the audio and determine that the voice input by the user is 'screen off'. After the terminal determines the voice input by the user, the screen can be turned off according to the voice point.

And thirdly, when the terminal is in the screen locking state, the user can control the terminal through voice.

(1) When the terminal is in the screen locking state, the user can control the terminal to execute the unlocked function through voice.

As an example, when the terminal locks the screen, the user may input a voice "increase the volume", and the terminal is manipulated to increase the volume of the playing media.

(2) When the terminal is in the screen locking state, the user can unlock through voice.

As an example, when the terminal locks the screen, the user may input a voice "unlock" to operate the terminal to unlock.

Because the voice is not unique, people who do not have the right to use the terminal can also give the voice for unlocking, and in order to ensure the safety of data, other unlocking modes are provided. For example, by a human face or biometric unlocking. Here, the manner of unlocking the human face and unlocking the biometric features may refer to the related description in the application scenario 1. In a possible embodiment, unlocking can also be effected by means of a voiceprint. Specifically, the user can provide voice to the low-power-consumption microphone/the normal-power-consumption microphone, the microphone collects the voice in real time, the terminal calls a voiceprint recognition algorithm, and voiceprints corresponding to the collected voice are compared with stored voiceprints. And if the voiceprint corresponding to the voice is matched with the pre-stored voiceprint, the terminal can be unlocked.

After the terminal is unlocked, the user can operate the mobile phone through a touch screen gesture (namely a gesture of directly contacting the touch screen) or a user gesture or voice. For example, when a user operates a mobile phone through voice, the user can take a picture through voice, send information, make a call, open a certain sub-interface of an application (for example, a payment interface of a payment treasure), manage files, and the like.

(3) When the terminal is in the screen locking state, the user can control the terminal to execute the locked function through voice.

As an example, when the terminal locks the screen, the user may input a voice "start WeChat", and operate the terminal to start the WeChat.

It should be noted that the voice used for operating the terminal is not limited to the voice in the above example (a), (b), and (c), and may be other voices, which is not limited herein.

In combination with the above embodiment of controlling the terminal by voice, the voice for controlling the terminal in the present application can be set in the following ways: (1) the voice for manipulating the terminal may be user-set. For example, the user may select a voice from a setting menu provided by the terminal and including a plurality of voices as the voice for operating the terminal, or the user may customize the voice according to his or her own habits. (2) The voice for manipulating the terminal may be terminal-set. For example, the voice for manipulating the terminal may be a default setting when the terminal is shipped from a factory.

In application scenario 2, the terminal may collect user audio data in real time through the low power consumption microphone and may autonomously operate the AI-aware capability. When a user needs to use the terminal, the terminal can autonomously perform voice recognition only by inputting voice without contacting a touch screen, and corresponding operation is executed according to the voice input by the user. According to the method and the device, the control of the terminal can be more intelligent, and the user experience is improved.

The structure of the terminal in the present application is described below.

The terminal of the present application may include: the system comprises a main processor, a coprocessor and a normally-open low-power-consumption camera.

The coprocessor and the low-power consumption camera in the terminal are always on at low power consumption. And when no service exists, the main processor in the terminal is normally dormant and standby, and enters a low power consumption mode. And after the coprocessor in the terminal reports the AI event message, the main processor is awakened. The main processor can realize various service functions according to the service requirements of the product, or transmit the event message to other related service modules, and the other service modules finish final processing.

As an example, the terminal may report the AI identification result to the main controller when determining that a specific event occurs. As one example, the coprocessor may generate an AI message at the AI application layer and may report the AI message to the main controller. This will be described in detail below with reference to fig. 12, and will not be described in detail here.

As another example, when the terminal includes a normally-open low-power camera, the coprocessor in the terminal may analyze whether a user gesture of the user can be detected in the image according to the captured image and a corresponding AI algorithm (e.g., a gesture recognition algorithm). The coprocessor may also recognize a user gesture of a user and report the recognized user gesture of the user to the main controller.

In this application, the main controller may be responsible for running various applications of the terminal, including but not limited to: user Interface (UI), human interaction interface, gesture recognition, face recognition, environment recognition, on/off screen, and the like.

The following describes in detail, with reference to fig. 12, a specific implementation manner in which the terminal may autonomously run an AI sensing capability and sense changes of user intentions, gestures, expressions, and environments in real time by cooperatively processing the main processor and the coprocessor in the embodiment of the present application.

Fig. 12 is a schematic block diagram of a hardware architecture of a terminal 100 according to an embodiment of the present disclosure. The hardware architecture shown in fig. 12 may include a main processor 210, a co-processor 220, and a low power camera 230.

The coprocessor 220: the AI capability is integrated, and the continuous operation in a low power consumption mode can be used for detecting the action intention of a user and environmental changes. The coprocessor 220 is connected to the main processor 210, and when a corresponding event is detected, the main processor 210 is triggered to wake up the main processor 210 by reporting an AI event message to the main processor 210.

The main processor 210: when there is no service, the main controller 210 system may perform a normal sleep standby state and enter a low power consumption mode. After receiving the AI event message sent by the coprocessor 220, the main processor 210 is awakened, and then receives an event reported by the coprocessor 220, and triggers a corresponding service scenario function.

Low-power normally-open camera 230: the peripheral chip software interface (driver) provided by the coprocessor 220 is connected to the coprocessor 220, and provides a data source for the coprocessor 220 to process AI services.

The system architecture of coprocessor 220 is described in detail below.

The coprocessor 220 module may be a Real Time Operating System (RTOS). When an external event or data is generated, it can be accepted and processed at a sufficiently fast speed. The processing result can control the production process within the specified time or make quick response to the processing system, and schedule all available resources to complete real-time tasks. And all real-time tasks are controlled to be coordinated and run in a consistent manner, so that the response speed is high and the reliability is high.

The RTOS system of coprocessor 220 may include: a kernel (kernel)221, a framework layer (framework layer) 222, and an APP application layer 223.

The kernel (kernel)221 includes: peripheral driver module 2211, hardware acceleration module 2212 and AI operator library module 2213.

The framework layer 222 includes: an AI application management module 2221, an AI algorithm management module 2222, and an AI algorithm model 2223.

The APP application layer 223 includes: an AI application layer module 2231, an AI engine module 2232, and an AI model management module 2233.

The above modules are described in detail below.

Peripheral driver module 2211: and a software interface can be provided for connecting various peripheral chips. For example, a low power camera 230 may be connected, and the low power camera 230 may provide a hardware basis for the coprocessor 220 to perceive user behavioral intent or environmental changes. The coprocessor 220 may analyze characteristics of the user's actions and the surrounding environment according to the images acquired by the low power consumption camera 230, and provide a data source for the coprocessor 220 to process the AI service.

Specifically, the terminal may obtain an image in real time through the normally open low power consumption camera 230 connected to the peripheral driver module 2211.

Optionally, in some embodiments, the peripheral devices that may be connected to the peripheral driver module 2211 may further include, but are not limited to: sensors (which may be used to identify user actions), low power microphones (which may be used to analyze characteristics of a user's voice, etc.), location sensors (e.g., Global Positioning System (GPS), wireless local area network (WIFI), modem, which may be used to provide location information of a user). As an example, when the terminal includes a low power microphone that is always on, the co-processor in the terminal may analyze whether the user's voice is detected in the audio according to the captured audio and a corresponding AI algorithm (e.g., a voice recognition algorithm). The co-processor may also recognize the user's speech (i.e., understand the semantics of the speech) and report the recognized speech to the main controller.

The AI application management module 2221: the data reported by the peripheral driver module 2211 may be classified. For example, the received data is classified into image class, video class, audio class, etc. so as to call the AI algorithm models 2223 of different classes for analysis processing.

The AI engine module 2232: and the AI algorithm model 2223 can be responsible for scheduling and coordinating operations. Since multiple AI algorithm models 2223 can be run at the same time, the scheduling management control of the AI engine module 2232 can ensure that the software runs in order.

The AI algorithm management module 2222: and the AI application management module 2221 is responsible for algorithm management, and can select a corresponding AI algorithm model from the multiple operating AI algorithm models 2223 for analysis according to different types of data reported by the AI application management module 2221.

AI Algorithm model 2223: may be a collection of algorithmic features of images, sounds consistent with certain services. For example, when conducting a gesture recognition service, the AI algorithm model 2223 may be a set that conforms to certain gesture characteristics. For another example, the AI algorithm model 2223 can be a set of features that conform to a face contour when conducting a face recognition service. As another example, in a speech recognition service, the AI algorithm model 2223 may be a set of features that conform to a certain speech. The AI algorithm model 2223 may be trained using large-scale images, and after the training is completed, an algorithm model is generated, and the corresponding AI operator may operate the algorithm model to perform operations such as gesture recognition or face recognition.

Specifically, after the co-processing 220 in the terminal receives the image reported by the normally-open low-power-consumption camera 230, the AI application management module 2221 may invoke a corresponding AI algorithm through the AI engine module 2232 to analyze the user gesture in the acquired image.

It should be noted that the AI algorithm model 2223 may be integrated in a software system by default, or may be updated to the coprocessor 220 through the main controller 210, which is not specifically limited in this embodiment of the present application.

AI model management module 2233: in some embodiments, master controller 210 may also optimize AI algorithm model 2223. For example, the result of the AI algorithm model 2223 may be comprehensively determined using positioning information such as GPS/WIFI/modem, so as to improve the accuracy of the AI algorithm model 2223. AI model management module 2233 can modify certain features in AI algorithm model 2223.

AI operator library module 2213: the AI engine module 2232 may operate the AI model management module 2233 by calling operators in the AI operator library module 2213 to perform gesture recognition or face recognition operations. Because the resources of the coprocessor 220 are limited, the AI operator library module 2213 for designing a large number of mathematical calculations can be solidified in hardware, most operators of the AI can be realized by the hardware, and high processor load generated by software realization operators can be avoided. The interface of the hardware curing operator may be provided by the kernel 221 to the AI model management module 2233 for use.

It should be understood that the AI operator library module 2213 solidified in hardware (software solidification) may be writing software onto a coprocessor chip, and the software written on may be run through the coprocessor chip. The software solidification is to make the software on the silicon chip (so-called firmware) to realize the software function, so that the complexity of the operating system and the language processing is shared by both the software and the hardware.

In the embodiment of the present application, the AI operator library module 2213 is fixed on the hardware of the coprocessor, and the operation of the software fixing can improve the operation speed of the whole system, improve the reliability, reduce the cost, and facilitate large-scale production and standardization.

Hardware acceleration module 2212: the process of the AI model management module 2233 can be accelerated by the AI engine module 2232 invoking operators in the AI operator library module 2213 through an acceleration mode. The AI engine module 2232 can be ensured to call the operators in the AI operator library module 2213 quickly and in real time, and capability interfaces are provided for various AI algorithms in the AI model management module 2233 of the framework layer 222.

AI application layer module 2231: can be located in the APP application layer 223, and can implement various continuous AI applications in the APP application layer 223 according to the scene requirements of the terminal service design. The AI application layer module 2231 may call various algorithms to obtain AI identification results of various devices connected to the peripheral device, and may report a corresponding AI event message to the main controller 210. If the main controller 210 is in the sleep state, the AI event message may be secondarily processed after being awakened.

Specifically, when the terminal includes a valid user gesture (for example, providing a back of hand, a palm of hand, or an OK gesture) in the image reported by the low power consumption camera 230, the AI application management module 2221 reports the gesture detection result to the AI application layer module 2231. The AI application layer module 2231, after obtaining the identification result, forms an identification event message and reports the identification event message to the AI event message manager 212 in the main controller 210.

The system architecture of the main processor 210 is described in detail below.

The main processor 210: and the system is responsible for running various applications of the terminal, including UI (user interface) man-machine interaction interface, cloud interaction and the like. When no service exists, the main controller system is in normal sleep standby mode and enters a low power consumption mode.

The main processor 210 may include: AI native 211, AI event message manager (AI service)212, Application (APP) 213, APP 214, APP 215.

AI local (AI native) 211: the AI event message reported by the coprocessor 220 may be received and the main controller 210 is awakened. The AI algorithm model 2223 optimized by the main controller 210 may also be sent to the AI engine module 2232 of the coprocessor 220, and the AI engine module 2232 may update the AI algorithm model 2223 through the AI model management module 2233.

AI event message manager (AI service) 212: the AI event message reported by the AI native 211 can be received, the AI capability interface of the terminal is managed in a unified manner, and an AI Application Program Interface (API) is provided for each service module. According to the product service requirements, various service functions are realized. For example, different business functions may be implemented according to different applications (APP213 or APP 214 or APP 215).

Specifically, the AI event message manager 212 in the main controller 210 wakes up the main controller 210 after receiving the recognition event message transmitted by the AI application layer module 2231. The main controller 210 may determine an operation corresponding to user gesture data detected in an image reported by the low power consumption camera 230. After determining a terminal operation corresponding to a user gesture detected in an image reported by the low power consumption camera 230, the operation may be performed (e.g., screen lighting, unlocking, application starting, etc.).

Optionally, in some embodiments, if large data processing is required, the AI service 212 may also transmit data to the cloud, so as to complete a low power consumption service processing mode combining the terminal and the cloud.

In the embodiment of the application, the main frequency of the operation of the coprocessor is low, a large number of AI operators of mathematical operation are integrated in a hardware solidification mode, and peripheral devices are low-power-consumption devices and can be normally opened and operate AI sensing capability in a low-power-consumption mode, so that a terminal can sense the action change or the environment change of a user without depending on specific actions.

The following describes in detail the manner in which each component in the hardware architecture shown in fig. 12 cooperates with a specific scenario illustrated in fig. 1 in application scenario 1.

Fig. 13 is a schematic flowchart of a scene in which a terminal recognizes a user gesture to turn on a screen according to an embodiment of the present application. The method shown in fig. 13 may include the steps of:

1. the low power consumption camera 230 collects images.

The low power consumption camera 230 connected to the co-processor 220 continuously collects images around the terminal. Illustratively, the image captured by the low power consumption camera 230 includes the back of the hand provided by the user.

2. The low power camera 230 reports the captured image to the co-processor 220.

Specifically, the low power consumption camera may report the acquired image to the AI application management module 2221.

3. The co-processor 220 invokes a gesture recognition algorithm model to recognize the user gesture.

In some embodiments, the coprocessor 220 may call a gesture detection algorithm model to determine whether a user gesture occurs in an image captured by the low power consumption camera 230 before step 2. Specifically, if the contour features in the image are detected to conform to the contour features of the human hand, it can be determined that the user gesture appears in the image acquired by the low-power-consumption camera.

After the co-processing 220 receives the image reported by the low power consumption camera 230, the AI application management module 2221 may call the corresponding AI gesture recognition algorithm model 2223 to analyze the user gesture in the acquired image according to the data to be processed by the AI engine module 2232.

Specifically, the AI gesture recognition algorithm model 2223 may call a corresponding AI operator in the AI operator library module 2213, and operate the AI gesture recognition algorithm model 2223 to perform gesture recognition.

If the gesture recognition result is that the reported image includes a valid user gesture (for example, a back of a hand, that is, a back of a hand of a user appears in the image), the AI application management module 2221 may execute step 3.

If the gesture recognition result is that the reported image does not include any user gesture (i.e., the user gesture does not appear in the image), the AI application management module 2221 may re-execute step 1.

4. The coprocessor 220 reports the gesture recognition message to the main controller 210.

When the image reported by the low power consumption camera 230 includes a valid user gesture (for example, a back of hand), the AI application management module 2221 reports the gesture recognition result to the main controller 210.

Specifically, the AI application management module 2221 in the coprocessor 220 may report the gesture recognition result to the AI application layer module 2231. The AI application layer module 2231 forms a gesture recognition event message after obtaining the gesture recognition result, and reports the gesture recognition event message to the AI event message manager 212 in the main controller 210.

5. The master controller is awakened.

The AI event message manager 212 in the main controller 210 wakes up the main controller 210 after receiving the gesture recognition event message transmitted by the AI application layer module 2231.

6. The main controller lights the screen but maintains the screen lock state.

The main controller 210 may perform an operation corresponding to the gesture after receiving the gesture recognition event message. Here, in the example of fig. 1, the operation corresponding to the gesture of the back of the hand provided by the user is to light up the screen. That is, the main controller 210 lights up the screen after receiving the gesture recognition event message.

It can be understood that, when the terminal recognizes the user gesture in other specific scenarios and executes corresponding operation according to the user gesture, the cooperation manner of each component in the hardware architecture shown in fig. 12 is similar to that in fig. 13, and is not described in detail here.

The embodiments of the present application can be combined arbitrarily to achieve different technical effects.

In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the procedures or functions described in accordance with the present application are generated, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wire (e.g., coaxial cable, fiber optic, digital subscriber line) or wirelessly (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as a server, a data center, etc., that incorporates one or more of the available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., Solid State Disk), among others.

In short, the above description is only an example of the technical solution of the present invention, and is not intended to limit the scope of the present invention. Any modifications, equivalents, improvements and the like made in accordance with the disclosure of the present invention are intended to be included within the scope of the present invention.

Claims

A method for controlling a handheld mobile terminal is applied to the handheld mobile terminal, the handheld mobile terminal is provided with a display screen and a camera, and the method is characterized by comprising the following steps:

the terminal acquires a first image through the camera, and the camera is kept on when a display screen of the terminal is turned off or turned on so as to acquire an image;

the terminal analyzes whether a valid user gesture is included in the first image;

and under the condition that the first image comprises the effective user gesture, the terminal executes the operation corresponding to the effective user gesture.
The method of claim 1, wherein the valid user gesture comprises: palm, back of hand, swipe gesture, gesture to draw a pattern or letter.
The method according to claim 1 or 2, wherein the operation corresponding to the valid user gesture is set by the terminal by default or set by the user.
The method of any of claims 1-3, wherein the operation corresponding to the valid user gesture comprises at least one of:

executing unlocking operation, lighting up a screen, turning off the screen, turning up/down the volume, turning up/down the screen brightness, turning on/off the flight mode, performing screen capture operation on the current screen display content, starting an application or starting internal functions of the application.
The method according to any one of claims 1 to 4,

before the terminal executes the operation corresponding to the effective user gesture, the method further comprises: the terminal verifies whether the current user has the authority to use the terminal;

the terminal starts the operation corresponding to the effective user gesture, and the operation comprises the following steps: and under the condition that the current user has the authority to use the terminal, the terminal executes the operation corresponding to the effective user gesture.
The method of claim 5, wherein the terminal verifying whether the current user has the right to use the terminal comprises:

the terminal acquires a second image through the camera;

identifying a face of a user in the second image;

and if the face of the user in the second image is matched with face information prestored by the terminal, determining that the current user has the authority to use the terminal.
The method according to claim 6, characterized in that if the face of the user in the second image does not match the face information pre-stored in the terminal, the user is prompted to input authentication information;

if the verification information input by the user is matched with the pre-stored verification information, determining that the current user has the right to use the terminal;

if the verification information input by the user is not matched with the pre-stored verification information, determining that the current user does not have the right to use the terminal;

the authentication information includes any one of:

passwords, irises, fingerprints, and voices;

wherein the pre-stored authentication information corresponding to the sound includes: semantic information of the sound and/or voiceprint information of the sound.
A hand-held mobile terminal, comprising: the system comprises a coprocessor, a main processor, a camera and a display screen, wherein the coprocessor is connected with the camera and the main processor,

the coprocessor is used for acquiring a first image through a camera, and the camera is kept on to acquire the image when the display screen is turned off or turned on; analyzing whether a valid user gesture is included in the first image; sending indication information of a valid user gesture to the main processor under the condition that the valid user gesture is included in the first image;

and the main processor is used for executing the operation corresponding to the effective user gesture according to the indication information.
The terminal of claim 8, wherein the valid user gesture comprises: palm, back of hand, swipe gesture, gesture to draw a pattern or letter.
A terminal according to claim 8 or 9, characterized in that the function corresponding to the valid user gesture is set by the terminal default or by the user.
A terminal according to any of claims 8-10, characterized in that the operation corresponding to the valid user gesture comprises at least one of:

and executing unlocking operation, lighting up a screen, turning off the screen, increasing/decreasing the volume, increasing/decreasing the screen brightness, switching on and off the flight mode, and performing screen capture operation, application starting or internal functions of application starting on the current screen display content.
The terminal according to any of claims 8-11,

the main processor is further used for verifying whether the current user has the authority to use the terminal before executing the operation corresponding to the effective user gesture;

the main processor is specifically configured to execute an operation corresponding to the valid user gesture when the current user has the right to use the terminal.
The terminal of claim 12, wherein the main processor is further configured to verify whether the current user has permission to use the terminal before performing the operation corresponding to the valid user gesture, comprising:

the main processor is further used for identifying the face of the user in the second image before executing the operation corresponding to the effective user gesture; if the face of the user in the second image is matched with face information prestored by the terminal, determining that the current user has the right to use the terminal;

wherein the second image is acquired by the camera.
The terminal of claim 13,

the main processor is further used for prompting the user to input verification information if the face of the user in the second image is not matched with the face information prestored by the terminal;

if the verification information input by the user is matched with the pre-stored verification information, determining that the current user has the right to use the terminal;

if the verification information input by the user is not matched with the pre-stored verification information, determining that the current user does not have the right to use the terminal;

the authentication information includes any one of:

passwords, irises, fingerprints, and voices;

wherein the pre-stored authentication information corresponding to the sound includes: semantic information of the sound and/or voiceprint information of the sound.
The utility model provides a chip, the chip is applied to handheld mobile terminal, handheld mobile terminal includes camera and display screen, its characterized in that, the chip includes: a coprocessor and a main processor, wherein the coprocessor is connected with the main processor,

the coprocessor is used for acquiring a first image through the camera, and the camera is kept on to acquire the image when the display screen is turned off or turned on; analyzing whether a valid user gesture is included in the first image; sending indication information of a valid user gesture to the main processor under the condition that the valid user gesture is included in the first image;

and the main processor is used for executing the operation corresponding to the effective user gesture according to the indication information.
A computer storage medium, characterized in that it comprises a computer program which, when run on the terminal, causes the terminal to perform the method according to any one of claims 1-7.
A computer program product, comprising a computer program which, when run on the terminal, causes the terminal to perform the method according to any one of claims 1-7.