WO2020237617A1

WO2020237617A1 - Screen control method, device and apparatus, and storage medium

Info

Publication number: WO2020237617A1
Application number: PCT/CN2019/089489
Authority: WO
Inventors: 荀振
Original assignee: 华为技术有限公司
Priority date: 2019-05-31
Filing date: 2019-05-31
Publication date: 2020-12-03
Also published as: CN113728295A

Abstract

The present application provides a screen control method, device and apparatus, and a storage medium. Said method comprises: acquiring respective feature information of N screen control objects in a first image and respective action information of the N screen control objects, the screen control objects being body parts of a user in the first image, and N being a positive integer greater than or equal to 2; according to the respective feature information of the N screen control objects, determining display sub-screens respectively corresponding to the N screen control objects, the display sub-screens being partial areas of a display screen; and according to the respective action information of the N screen control objects, controlling the display sub-screens respectively corresponding to the N screen control objects. In this way, split-screen control of a screen is implemented, improving the utilization rate of the screen, and enhancing the user experience.

Description

Screen control method, device, equipment and storage medium

Technical field

This application relates to the field of screen control technology, and in particular to a method, device, device, and storage medium for screen control.

Background technique

With the development of science and technology and terminal equipment, equipment and systems with large screens are used in more and more fields due to their intuitiveness and ease of use. At the same time, the manipulation of large screens has brought corresponding problems to people. Therefore, how to better use and control large screen devices becomes particularly important. Normally, the control of large screens can be done through the remote control. , Buttons, gestures, and voice control.

In the prior art, the screen is controlled through gestures, usually through a single-person gesture to interact with the large screen, and the large screen is controlled correspondingly through gestures, so as to achieve the effect of controlling the screen through gestures.

However, in the prior art, the screen is controlled by gestures, and the large screen can only be controlled by a single gesture, resulting in low screen utilization.

Summary of the invention

The embodiments of the present application provide a method, device, device, and storage medium for controlling a screen, which are used to implement split-screen control of a screen by multiple screen control objects, which not only improves screen utilization, but also enhances user experience.

In the first aspect, this application provides a method for controlling a screen, including:

Acquire the respective feature information of the N screen control objects in the first image and the respective action information of the N screen control objects, where the screen control objects are the body parts of the user in the first image, and N is greater than or equal to 2 A positive integer; according to the respective characteristic information of the N screen control objects, determine the display sub-screen corresponding to each of the N screen control objects, the display sub-screen is a partial area of the display screen; according to the N control screens The respective action information of the objects controls the display sub-screens corresponding to the respective N screen control objects. In the embodiment of this application, the display sub-screen corresponding to the control object is determined according to the characteristic information of the control object, and the display sub-screen corresponding to the control object is controlled according to the action information of the control object, thereby realizing the split screen Control not only improves screen utilization, but also enhances user experience.

Optionally, before acquiring the respective feature information of the N screen control objects and the respective action information of the N screen control objects in the first image, the screen control method provided in this embodiment of the present application further includes:

Acquire a second image of the user; determine the number N of control objects in the second image that meet the preset activation action, and the preset activation action is used to activate the multi-gesture control mode; present N display sub-screens on the display screen .

In the embodiment of the present application, by determining how many display sub-screens to present the display screen according to the number of screen control objects satisfying the preset activation action in the second image of the user, the multi-gesture control mode is activated based on the preset activation action and Only the control object that satisfies the preset start action can become one of the controllers of multiple display sub-screens, which improves the anti-interference of the split screen. Illustratively, the control object is the user's hand. If there are four people in front of the screen , But only three people want to participate in the split-screen control, and only the hand that presents the preset start action can participate in the split-screen control, avoiding the fourth person’s hand from interfering with the split-screen control.

Optionally, before acquiring the respective feature information of the N screen control objects and the respective action information of the N screen control objects in the first image, the screen control method provided in the embodiment of the present application further includes:

Establish a first corresponding relationship between the feature information of the N screen control objects that satisfy the preset activation action and the N display sub-screens, the first corresponding relationship includes a one-to-one correspondence between the feature information of the control object and the display sub-screens; according to N Determining the respective display sub-screens corresponding to the N screen control objects by the respective feature information of each control screen object, including: determining the respective display corresponding to the N screen control objects according to the first correspondence and the respective feature information of the N screen control objects Sub-screen.

In the embodiment of the present application, the corresponding relationship between the feature information of a plurality of screen control objects satisfying the preset startup action and the multiple display sub-screens is established, and the control screen is determined according to the corresponding relationship and the feature information of the screen control object The display sub-screens corresponding to the objects respectively realize the corresponding control of the control object to the display sub-screens, and improve the accuracy of the control object's control of the display sub-screens.

Optionally, controlling the respective display sub-screens corresponding to the N screen control objects according to the respective action information of the N screen control objects includes:

The target control operation matching the action information of the target control object is determined in the second correspondence, and the target control object is any one of the N control objects, and the second correspondence includes multiple A one-to-one correspondence between each action information and multiple control operations; according to the target control operation, the display sub-screen corresponding to the target control screen object is controlled.

In the embodiment of the present application, by determining the target control screen operation of the target control screen object, and controlling the display sub-screen corresponding to the target control screen object according to the control screen operation corresponding to the target control screen operation, it is realized through the control screen object's action information and Characteristic information, correspondingly control the display sub-screen corresponding to the control screen object.

The following describes the screen control devices, equipment, storage media, and computer program products provided by the embodiments of the present application. For the content and effects, please refer to the first aspect and the optional method of the first aspect of the embodiments of the present application. .

In the second aspect, an embodiment of the present application provides a screen control device, including:

The first acquisition module is used to acquire the respective feature information of the N screen control objects and the respective action information of the N screen control objects in the first image. The screen control object is the body part of the user in the first image, and N is greater than or equal to 2 The first determining module is used to determine the display sub-screen corresponding to each of the N control screen objects according to the respective characteristic information of the N control screen objects, and the display sub-screen is a partial area of the display screen; the control module is used for According to the respective action information of the N screen control objects, the corresponding display sub-screens of the N screen control objects are controlled.

Optionally, the screen control device provided in the embodiment of the present application further includes:

The second acquisition module is used to acquire a second image of the user; the second determination module is used to determine the number N of control objects in the second image that meets the preset activation action, and the preset activation action is used to activate multiple gestures Control mode; segmentation module, used to obtain N display sub-screens presented on the display screen.

The establishment module is used to establish a first corresponding relationship between the feature information of the N screen control objects that meet the preset startup action and the N display sub-screens. The first corresponding relationship includes the feature information of the control object and the display sub-screen. Correspondence; the first determining module is specifically configured to: determine the display sub-screens corresponding to each of the N screen control objects according to the first correspondence and the respective characteristic information of the N screen control objects.

Optionally, the control module is specifically used for:

In the second correspondence, the target control operation that matches the action information of the target control object is determined. The target control object is any one of the N control objects. The second correspondence includes multiple action information and multiple One-to-one correspondence of the control operation; according to the target control operation, the display sub-screen corresponding to the target control object is controlled.

In a third aspect, an embodiment of the present application provides a device, including:

The processor and the transmission interface, the transmission interface, are used to receive the first image of the user obtained by the camera; the processor is used to call the software instructions stored in the memory to perform the following steps:

Acquire the respective feature information of the N screen control objects in the first image and the respective action information of the N screen control objects, where the screen control objects are the body parts of the user in the first image, and N is greater than or equal to 2 A positive integer; according to the respective characteristic information of the N screen control objects, determine the display sub-screen corresponding to each of the N screen control objects, the display sub-screen is a partial area of the display screen; according to the N control screens The respective action information of the objects controls the display sub-screens corresponding to the respective N screen control objects.

Optionally, the transmission interface is also used to receive the second image of the user obtained by the camera; the processor is also used to:

The number N of control objects satisfying the preset activation action in the second image is determined, and the preset activation action is used to activate the multi-gesture control mode; N display sub-screens presented on the display screen are obtained.

Optionally, the processor is also used to:

Establish a first corresponding relationship between the feature information of the N screen control objects that satisfy the preset startup action and the N display sub-screens, the first corresponding relationship includes a one-to-one correspondence between the feature information of the control object and the display sub-screens; the processor; , Specifically used to: determine the display sub-screens corresponding to each of the N screen control objects according to the first correspondence and the respective characteristic information of the N screen control objects.

Optionally, the processor is also used to:

In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium that stores instructions in the computer-readable storage medium. When the instructions run on a computer or a processor, the computer or the processor executes the Examples are the screen control methods provided in the first aspect and the optional methods of the first aspect.

The fifth aspect of the present application provides a computer program product containing instructions, which, when run on a computer or processor, causes the computer or processor to execute the control provided in the first aspect or alternatively provided by the first aspect.屛 method.

The screen control method, device, device, and storage medium provided by the embodiments of the present application obtain the respective characteristic information of the N screen control objects and the respective action information of the N screen control objects in the first image, and the screen control object Is the body part of the user in the first image, and N is a positive integer greater than or equal to 2; then, according to the respective characteristic information of the N screen control objects, determine the display sub-screens corresponding to each of the N screen control objects, The display sub-screen is a partial area of the display screen; finally, according to the respective action information of the N screen control objects, the display sub-screens corresponding to each of the N screen control objects are controlled. . Since the display sub-screen corresponding to the control screen object is determined according to the characteristic information of the control screen object, and the display sub-screen corresponding to the control screen object is controlled according to the action information of the control screen object, the split-screen control of the screen is realized, which not only improves Screen utilization, and enhanced user experience.

Description of the drawings

Fig. 1 is a schematic diagram of an exemplary application scenario provided by an embodiment of the present application;

Figure 2 is a schematic diagram of another exemplary application scenario provided by an embodiment of the present application;

FIG. 3 is a flowchart of a method for controlling a screen provided by an embodiment of the present application;

Fig. 4 is an exemplary neural network application architecture diagram provided by an embodiment of the present application;

FIG. 5 is a flowchart of a screen control method provided by another embodiment of the present application;

FIG. 6 is a flowchart of a screen control method provided by another embodiment of the present application;

FIG. 7 is a schematic structural diagram of a screen control device provided by an embodiment of the present application;

FIG. 8A is a schematic structural diagram of a terminal device provided by an embodiment of the present application;

FIG. 8B is a schematic structural diagram of a terminal device provided by another embodiment of the present application;

FIG. 9 is a schematic diagram of the hardware architecture of an exemplary screen control device provided by an embodiment of the present application;

FIG. 10 is a schematic structural diagram of a terminal device provided by another embodiment of the present application.

Detailed ways

It should be understood that although the terms first, second, third, fourth, etc. may be used to describe user images in the embodiments of the present invention, these user images should not be limited to these terms. These terms are only used to distinguish user images from each other. For example, without departing from the scope of the embodiments of the present application, the first user image may also be referred to as the second user image, and similarly, the second user image may also be referred to as the first user image.

With the development of science and technology and terminal equipment, equipment and systems with large screens are used in more and more fields because of their intuitiveness and ease of use. At the same time, the manipulation of large screens has brought corresponding problems to people. Therefore, how to better use and control large screen devices becomes particularly important. Normally, the control of large screens can be done through the remote control. , Buttons, gestures, and voice control. However, in the prior art, the screen is controlled by gestures, usually a single-person gesture is used to interact with the large screen, and the large screen is correspondingly controlled through gesture actions, resulting in low screen utilization. In order to solve the foregoing problems, embodiments of the present application provide a method, device, device, and storage medium for controlling a screen.

An exemplary application scenario of the embodiment of the present application will be introduced below.

The embodiments of this application can be applied to terminal devices with display screens. The terminal devices can be, for example, televisions, computers, screen projection devices, etc. The embodiments of this application do not limit this, and the terminal device is a television as an example. The application scenarios in the application embodiments are introduced. Fig. 1 is a schematic diagram of an exemplary application scenario provided by an embodiment of the present application. As shown in Fig. 1, a television 10 is connected to a camera 11 through a universal serial bus or other high-speed bus 12. When multiple users watch the television, every The TV programs that a user wants to watch are different, or there are video games in which multiple users participate. It is necessary to split the TV screen according to the characteristic information of each user, and each user can independently control the characteristic information of the user. The corresponding display sub-screen, for example, as shown in FIG. 1, the user image or video opposite to the television 10 is captured by the camera 11, and after processing and judgment, the display screen of the television 10 is divided into a display sub-screen 1 and a display sub-screen 2. Among them, display sub-screen 1 and display sub-screen 2 can display different playback content, and can continuously obtain user images through camera 11, and then process the user image to perform display sub-screen 1 and display sub-screen 2 respectively Control, the embodiment of this application is not limited to this. In addition, the television 10 may further include a video signal source interface 13, a wired network interface or a wireless network interface module 14, or a peripheral device interface 15, which is not limited in the embodiment of the present application.

The following introduces another exemplary application scenario of the embodiment of the present application.

The embodiment of this application can be applied to a display device. FIG. 2 is a schematic diagram of another exemplary application scenario provided by an embodiment of this application. As shown in FIG. 2, the display device may include a central processing unit, a system memory, and an edge artificial intelligence processor. Core and image memory, the central processing unit is connected to the system memory, the central processing unit can be used to execute the screen control method provided in the embodiments of this application, the central processing unit can also be connected to the edge artificial intelligence processor core, and the edge artificial intelligence can be used for To implement the image processing part of the screen control method provided by the embodiments of this application, the edge artificial intelligence processor core is connected to the image memory, and the image memory can be used to store the images acquired by the camera, the universal serial bus of the camera and the display device or other high-speed buses connection.

Based on this, the embodiments of the present application provide a method, device, device, and storage medium for controlling a screen.

FIG. 3 is a flowchart of a method for controlling a screen provided by an embodiment of the present application. The method can be executed by the device for controlling a screen provided by an embodiment of the present application. The controlling device may be part or all of a terminal device, for example, in a terminal device. The following uses the terminal device as the execution body as an example to introduce the screen control method provided in the embodiment of the present application. As shown in FIG. 3, the screen control method provided by the embodiment of the present application may include:

Step S101: Obtain the respective feature information of the N screen control objects and the respective action information of the N screen control objects in the first image. The screen control objects are the body parts of the user in the first image, and N is a positive integer greater than or equal to 2.

The first image of the user can be obtained through a camera or an image sensor. The camera can be set in the terminal device, or set independently of the terminal device, and be connected to the terminal device by wired or wireless connection. There are no restrictions on the installation location, as long as the user's first image can be obtained. The camera collects the first image of the user by means of video collection or image collection. The embodiment of the present application does not limit the specific method of how to obtain the first image of the user through the camera. In an optional situation, for the processor chip in the terminal device, the transmission interface of the processor chip receiving the user's image obtained by the camera or the image sensor can also be regarded as obtaining the user's first image, that is, The processor chip obtains the user's first image through the transmission interface.

After obtaining the first image of the user, according to the first image, obtain the respective characteristic information of the N screen control objects and the respective action information of the N screen control objects, where the N screen control objects are the user's body in the first image Part, the embodiment of the present application does not limit the specific part of the user's body part. The judgment of the user's body part in the first image can be realized by setting a preset body part. In a possible implementation, the preset body part may be a human hand. Correspondingly, the N screen control objects are at least one human hand in the first image, and the feature information of the N screen control objects is N in the first image. The hand feature information of a human hand, for example, the hand feature information includes but is not limited to handprints, hand shape, hand size, or hand skin color. The motion information of each of the N screen control objects is the hand motion information of the N human hands in the first image. In another possible implementation manner, the preset body part may be a human face. Correspondingly, the N screen control objects are the N faces in the first image, and the feature information of the N screen control objects is in the first image. The facial feature information of the N faces, and the respective action information of the N screen control objects are the facial action information of the N faces in the first image, such as facial expressions, and the embodiment of the present application is not limited thereto.

Among them, the feature information of each of the N screen control objects is used to distinguish the N screen control objects. Illustratively, if the screen control object is a human hand, the hand feature information of the human hand is used to distinguish different human hands. Face, the feature information of the face is used to distinguish different faces.

The embodiment of this application does not limit the specific implementation manners of how to obtain the respective characteristic information of the at least one screen control object and the respective action information of the at least one screen control object according to the first image. In a possible implementation manner, The way of machine learning, such as Convolutional Neural Network (CNN) model, etc. Exemplarily, taking the preset user's body part as a human hand as an example, by inputting the first image into the CNN model, and processing by the CNN model, the hand feature information of each human hand in the first image, and each human hand are obtained Information about your hand movements.

FIG. 4 is an exemplary neural network application architecture diagram provided by an embodiment of the present application. As shown in FIG. 4, the exemplary neural network application architecture diagram provided by an embodiment of the present application may include an application program entry 41, a model external interface 42, and The deep learning structure 43, the device driver 44, the central processing unit 45, the graphics processor 46, the network processor 47, and the digital processor 48. Among them, the application program entry 41 is used to select the neural network model, and the model external interface 42 is used To call the selected neural network model, the deep learning structure 43 is used to process the input first user image through the neural network model. Exemplarily, the deep learning structure 43 includes an environment manager 431, a model manager 432, and task scheduling 433, task performer 434, and event manager 435. Among them, the environment manager 431 is used to control the startup and shutdown of the device-related environment, the model manager 432 is used to load and unload the neural network model, and the task scheduler 433 is used to manage the neural network model. Which sequence is used for scheduling, the task executor 434 is responsible for executing tasks of the neural network model, and the event manager 435 is responsible for notifications of various events. The neural network application architecture provided by the embodiments of the present application is not limited to this.

Step S102: Determine a display sub-screen corresponding to each of the N screen control objects according to the respective characteristic information of the N screen control objects, and the display sub-screen is a partial area of the display screen.

After acquiring the respective characteristic information of the N screen control objects, determine the corresponding display sub-screens of the N screen control objects. In a possible implementation manner, 4 control objects and 4 control objects are obtained. Correspondingly, the display screen is divided into 4 display sub-screens, and each display sub-screen is bound to the characteristic information of a control object, so that the control object can only control the characteristic information of the control object. Bound display sub-screens; in a possible implementation, preset feature information can be set for each display sub-screen, and then N can be determined according to the respective feature information and preset feature information of the N control screen objects Each control screen object corresponds to the display sub-screen. For example: the screen is divided into 4 display sub-screens, and each display sub-screen has a one-to-one corresponding preset feature information, and the feature information of the control screen object is matched with the preset feature information, and then determined according to the matching result The display sub-screen corresponding to the control screen object. The embodiment of the present application does not limit the specific implementation of how to determine the display sub-screen corresponding to each of the N screen control objects according to the respective characteristic information of the N screen control objects. In addition, in a possible implementation manner, the display sub-screen may be a partial area of the display screen, for example: the display screen is divided into different display sub-screens; in another possible implementation manner, the display sub-screen is a display All areas of the screen, for example: the display screen is a multi-channel display mode, which can realize the function of outputting multiple different pictures at the same time on the same display screen, and has multi-channel audio output. Users can watch separately by wearing different glasses and headphones For two different programs, etc., the embodiment of the present application does not limit the area of the display sub-screen and the splitting method.

Determining the display sub-screens corresponding to each of the N screen control objects can be implemented according to the identification of the display sub-screen and the identification between the screen control objects. Exemplarily, firstly, each display sub-screen is identified according to the preset feature information. The embodiment of the present application does not limit the specific identification method of the display sub-screen, for example, by encoding, numbers, symbols, text, etc., for example, Display sub-screen 1 corresponds to preset feature information 1, display sub-screen 2 corresponds to preset feature information 2, and so on. Then the feature information of the N screen control objects in the first image is detected, and the N screen control objects are identified according to the feature information of the screen control object. The embodiment of this application does not limit the specific identification method of the screen control object, for example, If the feature information of the screen control object matches the preset feature information 1, the screen control object is identified as the screen control object 1, and so on.

The embodiment of this application does not limit the specific implementation of how to identify the screen control object. In a possible implementation manner, the feature information of the N screen control objects in the first image can be detected through the CNN model, and the feature information of the control screen objects can be detected according to the control. The feature information of the screen object identifies the N screen control objects. In another possible implementation, the coordinate information of each control screen object in the first image can be checked through CNN, and each control screen image can be cropped in the original image according to the coordinate information of each control screen object , Process as a separate image, detect the characteristic information of the control screen object in each separate image, and identify each separate image.

After the identification of each screen control object, the display sub-screen corresponding to each screen control object is determined according to the identification of the display sub-screen or the corresponding relationship between the identification of the screen control object and the identification of the display sub-screen. For example: the first image includes 3 screen control objects, which are respectively identified as screen control object 1, screen control object 2 and screen control object 3. The screen is divided into 3 display sub-screens, respectively identified as display sub-screen 1, display Sub-screen 2 and display sub-screen 3. The display sub-screen corresponding to control-screen object 1 is display sub-screen 1, the display sub-screen corresponding to control-screen object 2 is display sub-screen 2, and the display sub-screen corresponding to control-screen object 3 is display Sub-screen 3, the comparison of the embodiments of this application is not limited.

Step S103: According to the respective action information of the N screen control objects, control the display sub-screens corresponding to each of the N screen control objects.

After determining the respective display sub-screens corresponding to the N screen control objects, the respective display sub-screens corresponding to the N screen control objects are controlled according to the respective action information of the N screen control objects. Taking the above exemplary correspondence between the screen control object and the display sub-screen as an example for further introduction, the display sub-screen 1 is controlled according to the action information of the screen control object 1, and the sub-screen 2 is controlled according to the action information of the screen control object 2. The sub-screen 3 is controlled to be displayed according to the action information of the control object 3 The embodiment of the present application does not limit how to control the display sub-screen corresponding to the screen control object according to the action information of the screen control object.

In order to realize the control of the display sub-screen corresponding to the control screen object according to the motion information of the control screen object, in a possible implementation manner, the respective display sub-screens corresponding to the N screen control objects are controlled according to the respective motion information of the N screen control objects. Screen, including:

In the second correspondence, the target control operation that matches the action information of the target control object is determined. The target control object is any one of the N control objects. The second correspondence includes multiple action information and multiple One-to-one correspondence of the control operation; according to the target control operation, the display sub-screen corresponding to the target control object is controlled. The second correspondence includes a one-to-one correspondence between multiple action information and multiple control screen operations, where the action information may be a control instruction to the screen, and the control screen operation is used for a specific control method for displaying the sub-screen. Then establish a second correspondence between the preset action information and the control screen operation. The embodiment of this application does not limit the specific relationship between the multiple action information and the multiple control screen operations, as long as the action can be performed according to the action information The control screen corresponding to the information can be operated. For example: when the action information is the gesture "OK", the corresponding control screen operation is "OK", the action information is the gesture "single finger down", the corresponding control screen operation is "select box down", and the action information is gesture " Thumbs up", the corresponding control screen operation is "return", etc.

A target screen control operation matching the action information of the target screen control object is determined from the plurality of action information in the second corresponding relationship, and the target screen control object is any one of the N screen control objects.

In the action information of the N screen control objects, there may be invalid action information. Therefore, it is necessary to determine the target control screen operation matching the action information of the target screen control object from the multiple action information in the second correspondence relationship to determine Whether it is necessary to control the display of the sub-screen or how to control the display of the sub-screen. Specifically, a neural network model can be used to match the action information of the target screen control object with multiple action information in the second correspondence. If the action information of the target screen control object matches the multiple action information in the second correspondence If none of them match, the action information of the target screen control object is invalid action information. If the action information of the target screen control object matches any one of the action information in the second correspondence, the second correspondence is determined The action information in the relationship that matches the action information of the target screen object is the target screen operation.

Finally, the display sub-screen corresponding to the target control screen object is controlled according to the control operation corresponding to the target control screen operation.

After determining the target control screen operation of the target control screen object, determine the control screen operation corresponding to the target control screen operation according to the second correspondence, and then control the display sub-screen corresponding to the target control screen object according to the control screen operation corresponding to the target control screen operation . By determining the target control screen operation of the target control screen object, and controlling the display sub-screen corresponding to the target control screen object according to the control screen operation corresponding to the target control screen operation, the control screen can be controlled by the control screen object’s action information and characteristic information. The display sub-screen corresponding to the object performs corresponding control.

The method for controlling the screen provided by the embodiment of the application obtains the first image of the user, and according to the first image, obtains the respective characteristic information of the N screen control objects and the respective action information of the N screen control objects. The feature information of is used to distinguish N screen control objects, and then according to the respective feature information of the N screen control objects, the display sub-screen corresponding to each of the N screen control objects is determined. The display sub-screen is part or all of the area of the display screen, and finally According to the respective action information of the N screen control objects, the corresponding display sub-screens of the N screen control objects are controlled. Since the display sub-screen corresponding to the control screen object is determined according to the characteristic information of the control screen object, and the display sub-screen corresponding to the control screen object is controlled according to the action information of the control screen object, the flexible control of the screen splitting is realized. Improved screen utilization and enhanced user experience.

Optionally, in order to achieve split-screen control of the screen, a multi-gesture control screen mode needs to be activated to split the screen according to user needs. Fig. 5 is a flowchart of a screen control method provided by another embodiment of this application. The method can be executed by the screen control device provided in this embodiment of the application. The screen control device can be part or all of a terminal device. The execution subject is taken as an example to introduce the screen control method provided in the embodiment of the present application. As shown in FIG. 5, before step S101, the screen control method provided in the embodiment of the present application may further include:

Step S201: Acquire a second image of the user.

To obtain the second image of the user, refer to the description of the method of obtaining the first image of the user in the step S101, which is not repeated in this embodiment of the present application. The second image includes screen control objects that satisfy the preset startup action.

In order to save energy consumption, the camera switch may be turned on when preparing to obtain the second image of the user to obtain the second image of the user, which is not limited in the embodiment of the present application.

Step S202: Determine the number N of control objects in the second image that satisfy the preset activation action, and the preset activation action is used to activate the multi-gesture control mode.

There may be multiple screen control objects in the second image. Taking the screen object as a human hand as an example, there may be multiple hands in the second image. In the hands of these multiple people, there may be invalid screen control objects. In the control screen mode, the invalid control screen objects can be filtered out by the way of preset start actions, so as to realize accurate judgment on the number of displayed sub-screens. The embodiment of the application does not limit the specific actions of the preset activation action. In a possible implementation manner, if the screen control object is a human hand, the preset activation action may be a preset gesture action, and it is determined that the second image The number of human hands that satisfy the preset gesture action is N; in another possible implementation, if the screen control object is a face, the preset activation action may be a preset face expression, and it is determined that the second image satisfies the preset The number of facial expressions is N.

In addition, the embodiment of the present application does not limit the manner of how to determine the number N of screen control objects that satisfy the preset activation action in the second image. In a possible implementation manner, the A plurality of screen control objects and the action information of the plurality of screen control objects are obtained, and then it is determined whether the action information of the plurality of screen control objects meets the preset start action, so as to determine the control block in the second image that meets the preset start action The number of objects N. In another possible implementation manner, the number of screen control objects satisfying the preset startup action in the second image may be detected to determine the number N of screen control objects satisfying the preset startup action in the second image.

Step S203: Present N display sub-screens on the display screen.

After determining the number N of control objects satisfying the preset activation action in the second image, N display sub-screens are presented on the display screen. The embodiment of the present application has an explanation of how to present N display sub-screens on the display screen. The specific implementation manner is not limited. For example, the display screen is divided into N display sub-screens. The embodiment of the present application does not limit the specific implementation manner of dividing the display screen into N display sub-screens. In a possible implementation manner, the display screen is divided into N display sub-screens, the display screen can be equally divided into N display sub-screens, or the size and size of the N display sub-screens can be adjusted according to user needs. The position relationship is set. The embodiment of the present application does not limit the size and position relationship of each display sub-screen. In another possible implementation manner, the display screen can be divided into N multi-channels, and different images can be displayed through the multi-channels.

In the embodiment of the present application, by determining the number of screen control objects that meet the preset activation action in the user's second image, multiple display sub-screens are presented on the display screen, which realizes screen segmentation and multi-gesture control. When the screen mode is turned on, before the first image of the user is acquired, it can be determined whether the display screen can be split-screen control by detecting whether the multi-gesture control screen mode is turned on. If the multi-gesture control screen mode is not enabled, the user is required to start the multi-gesture control screen mode according to the preset start action, and then perform split-screen control of the display screen, which improves the efficiency of the user's split-screen control.

Optionally, FIG. 6 is a flowchart of a screen control method provided by another embodiment of the present application. The method can be executed by the screen control device provided in the embodiment of the present application, and the screen control device may be part or all of a terminal device, for example It may be a processor in a terminal device. The following takes the terminal device as an execution subject as an example to introduce the screen control method provided in the embodiment of the present application. As shown in FIG. 6, before step S101, the screen control method provided in the embodiment of the present application may further include:

Step S301: Establish a first corresponding relationship between the feature information of the N screen control objects that satisfy the preset activation action and the N display sub-screens, the first corresponding relationship includes a one-to-one correspondence between the feature information of the control object and the display sub-screens .

After dividing the display screen into N display sub-screens according to the number of screen control objects that meet the preset startup action in the second image, it is also necessary to create N screen control objects that meet the preset startup action and N The first correspondence between the sub-screens is displayed, and the first correspondence includes a one-to-one correspondence between the feature information of the screen control object and the displayed sub-screens. By establishing a one-to-one correspondence between the feature information of the screen control object and the display sub-screen, it is possible to determine the display sub-screen corresponding to the screen control object according to the feature information of the screen control object.

Exemplarily, in the second image, the number of screen control objects that meet the preset start action is 4, and the screen control objects are human hand 1, human hand 2, human hand 3, and human hand 4, and the display screen is divided into 4 display sub The screens are display sub-screen 1, display sub-screen 2, display sub-screen 3, and display sub-screen 4, respectively, to obtain the characteristic information of 4 control screen objects, namely, the characteristic information of human hand 1, the characteristic information of human hand 2, and the human hand The feature information of human hand 3 and the feature information of human hand 4 establish a one-to-one correspondence between the feature information of human hand and the display sub-screen. For example, the feature information of human hand 1 corresponds to display sub-screen 1, and the feature information of human hand 2 corresponds to the display sub-screen. The screen 2 corresponds, the characteristic information of the human hand 3 corresponds to the display sub-screen 3, and the characteristic information of the human hand 4 corresponds to the display sub-screen 4. The embodiment of the present application is not limited to this.

Correspondingly, step S102 may be:

Step S302: According to the first correspondence and the respective characteristic information of the N screen control objects, determine the display sub-screens corresponding to each of the N screen control objects.

The embodiment of this application does not limit how to determine the manner of displaying the sub-screen corresponding to each of the N screen control objects according to the first correspondence and the respective characteristic information of the N screen control objects. In a possible implementation manner, By acquiring the respective characteristic information of the N screen control objects, and then respectively match the characteristic information of the N screen control objects that meet the preset startup action, and finally determine the corresponding N screen control objects according to the first correspondence and the matching result The sub-screen is displayed.

Exemplarily, taking the number of screen control objects satisfying the preset activation action in step 301 as an example, the first image includes 4 human hands, namely, human hand A, human hand B, human hand C, and human hand D, which are obtained separately The feature information of the four human hands is matched with the feature information of human hand 1, human hand 2, human hand 3, and human hand 4 in the second image. After matching, the feature information of human hand A is consistent with the feature information of human hand 1. It is determined that the display sub-screen 1 corresponding to the characteristic information of the human hand 1 is the display sub-screen corresponding to the human hand A, and the display sub-screen is controlled by the motion information of the human hand A, and so on, and will not be repeated.

In the embodiment of the present application, the corresponding relationship between the feature information of a plurality of screen control objects satisfying the preset startup action and the multiple display sub-screens is established, and the control screen is determined according to the corresponding relationship and the feature information of the screen control object The respective display sub-screens corresponding to the objects realize the separate control of the display sub-screens by the control object, and improve the accuracy of the control object's control of the display sub-screens.

The following describes the screen control device, equipment, storage medium, and computer program product provided by the embodiments of the present application. For the content and effects, please refer to the screen control method provided in the foregoing embodiments of the present application, and will not be repeated.

An embodiment of the present application provides a screen control device. FIG. 7 is a schematic structural diagram of the screen control device provided by an embodiment of the present application. The screen control device may be part or all of a terminal device. The following takes the terminal device as the execution body as an example. As shown in FIG. 7, the screen control device provided by the embodiment of the present application may include:

The first acquisition module 71 is configured to acquire the respective feature information of the N screen control objects and the respective action information of the N screen control objects in the first image. The screen control object is the body part of the user in the first image, and N is greater than or equal to Positive integer of 2;

The first determining module 72 is configured to determine a display sub-screen corresponding to each of the N screen control objects according to the respective characteristic information of the N screen control objects, and the display sub-screen is a partial area of the display screen;

The control module 73 is configured to control the display sub-screens corresponding to the N screen control objects according to the respective action information of the N screen control objects.

In an optional situation, the functions of the first determining module and the control module may also be performed by a processing module. The processing module may be, for example, a processor, and the first acquiring module may be a transmission interface of the processor, or It can also be said that the first acquiring module is the receiving interface of the processor. In this case, the functions of the first determining module and the control module may also be performed by the processor.

Optionally, as shown in FIG. 7, the screen control device provided in the embodiment of the present application may further include:

The second acquisition module 74 is configured to acquire a second image of the user;

The second determining module 75 is configured to determine the number N of control objects in the second image that meet the preset activation action, and the preset activation action is used to activate the multi-gesture control mode;

The segmentation module 76 is used to obtain N display sub-screens presented on the display screen. Exemplarily, the dividing module divides the screen into a corresponding number of display sub-screens according to the number of control objects that meet the preset activation action determined by the second determining module.

In an optional situation, the second acquisition module and the first acquisition module may both be the transmission interface or the reception interface of the processor, and the functions of the second determination module and the segmentation module may both be completed by the processing module. The module may be a processor, for example. In this case, the functions of the second determining module and the splitting module may both be completed by the processor.

The establishment module 77 is used to establish a first correspondence between the characteristic information of the N screen control objects that satisfy the preset activation action and the N display sub-screens, and the first correspondence includes one of the characteristic information of the control object and the display sub-screens. One correspondence

The first determining module 72 is specifically configured to:

According to the first correspondence and the respective characteristic information of the N screen control objects, the display sub-screens corresponding to the N screen control objects are determined.

Optionally, the control module 73 is specifically used for:

The device embodiments provided in this application are merely illustrative, and the module division in FIG. 7 is only a logical function division, and there may be other division methods in actual implementation. For example, multiple modules can be combined or integrated into another system. The mutual coupling between the various modules can be realized through some interfaces. These interfaces are usually electrical communication interfaces, but it is not excluded that they may be mechanical interfaces or other forms of interfaces. Therefore, the modules described as separate components may or may not be physically separated, and may be located in one place or distributed to different locations on the same or different devices.

An embodiment of the present application provides a device. FIG. 8A is a schematic structural diagram of a terminal device provided by an embodiment of the present application. As shown in FIG. 8A, the terminal device provided by the present application includes a processor 81, a memory 82, and a transceiver 83. The memory stores software instructions or computer programs; the processor may be a chip, the transceiver 83 implements the sending and receiving of communication data by the terminal device, and the processor 81 is configured to call the software instructions in the memory to implement the above-mentioned screen control method, Please refer to the method embodiment for the content and effect.

An embodiment of the present application provides a device. FIG. 8B is a schematic structural diagram of a terminal device provided by another embodiment of the present application. As shown in FIG. 8B, the terminal device provided by the present application includes a processor 84 and a transmission interface 85. The transmission interface 85 , Is used to receive the first image of the user obtained by the camera; the processor 84 is used to call the software instructions stored in the memory to perform the following steps: obtain the respective feature information and N control objects of the N control screen objects in the first image The respective action information of the screen objects, the screen control object is the user's body part in the first image, and N is a positive integer greater than or equal to 2; according to the respective characteristic information of the N screen control objects, the corresponding display of the N screen control objects is determined Sub-screen, the display sub-screen is a partial area of the display screen; according to the respective action information of the N screen-control objects, the display sub-screens corresponding to the N screen-control objects are controlled.

Optionally, the transmission interface 85 is further configured to receive a second image of the user acquired by the camera; the processor 84 is further configured to: determine the number N of control objects in the second image that satisfy the preset activation action, and preset The start action is used to start the multi-gesture control mode; N display sub-screens presented on the display screen are obtained.

Optionally, the processor 84 is also used to:

Establish a first corresponding relationship between the feature information of the N screen control objects that satisfy the preset startup action and the N display sub-screens, the first corresponding relationship includes a one-to-one correspondence between the feature information of the control object and the display sub-screens; the processor; 84. It is specifically configured to: determine the display sub-screens corresponding to each of the N screen control objects according to the first correspondence and the respective characteristic information of the N screen control objects.

Optionally, the processor 84 is also used to:

FIG. 9 is a schematic diagram of the hardware architecture of an exemplary screen control device provided by an embodiment of the present application. As shown in FIG. 9, the hardware architecture of the screen control device 900 may be applicable to SOC and application processor (AP).

Exemplarily, the screen control device 900 includes at least one central processing unit (CPU), at least one memory, a graphics processing unit (GPU), a decoder, a dedicated video or graphics processor, and a receiver. Interface and sending interface, etc. Optionally, the screen control device 900 may also include a microprocessor and a microcontroller MCU, etc. In an optional situation, the above-mentioned parts of the screen control device 900 are coupled through a connector. It should be understood that, in the various embodiments of the present application, coupling refers to mutual connection in a specific manner, including direct connection or through other The devices are indirectly connected, such as through various interfaces, transmission lines or buses. These interfaces are usually electrical communication interfaces, but it is not excluded that they may be mechanical interfaces or other forms of interfaces, which are not limited in this embodiment. In an optional case, the above-mentioned parts are integrated on the same chip; in another optional case, the CPU, GPU, decoder, receiving interface, and transmitting interface are integrated on one chip, and the chip is The various parts of the bus access external memory. The dedicated video/graphics processor may be integrated with the CPU on the same chip, or may exist as a separate processor chip. For example, the dedicated video/graphics processor may be a dedicated image signal processor (ISP). The chip involved in the embodiments of this application is a system manufactured on the same semiconductor substrate by an integrated circuit process, also called a semiconductor chip, which can be manufactured on a substrate using an integrated circuit process (usually a semiconductor such as silicon) The outer layer of the integrated circuit formed on the material) is usually encapsulated by a semiconductor packaging material. The integrated circuit may include various types of functional devices, and each type of functional device includes transistors such as logic gate circuits, Metal-Oxide-Semiconductor (MOS) transistors, bipolar transistors or diodes, and may also include capacitors and resistors. Or inductance and other components. Each functional device can work independently or under the action of necessary driver software, and can realize various functions such as communication, calculation, or storage.

Optionally, the CPU may be a single-CPU processor or a multi-CPU processor; optionally, the CPU may be a processor group composed of multiple processors, between multiple processors Coupled to each other through one or more buses. In an optional situation, part of the processing of the image signal or video signal is done by the GPU, part is done by a dedicated video/graphics processor, and it may also be done by software code running on a general-purpose CPU or GPU.

The device may also include a memory, which can be used to store computer program instructions, including various computer program codes including an operating system (Operation System, OS), various user application programs, and program codes used to execute the solutions of the present application; the memory; It can also be used to store video data, image data, etc.; the CPU can be used to execute computer program codes stored in the memory to implement the methods in the embodiments of the present application. Optionally, the memory may be a non-power-down volatile memory, such as Embedded MultiMedia Card (EMMC), Universal Flash Storage (UFS) or Read-Only Memory (ROM) ), or other types of static storage devices that can store static information and instructions, or volatile memory (volatile memory), such as random access memory (Random Access Memory, RAM), or can store information and instructions Other types of dynamic storage devices can also be Electrically Erasable Programmable Read-Only Memory (EEPROM), CD-ROM (Compact Disc Read-Only Memory, CD-ROM) or other optical disk storage , CD storage (including compressed CDs, laser disks, CDs, digital versatile CDs, Blu-ray CDs, etc.), disk storage media or other magnetic storage devices, or can be used to carry or store program codes in the form of instructions or data structures and can be used by Any other computer-readable storage medium accessed by the computer, but not limited to this.

The receiving interface may be a data input interface of the processor chip. In an optional case, the receiving interface may be a mobile industry processor interface (MIPI) or a high-definition multimedia interface (High Definition). Multimedia Interface, HDMI) or Display Port (DP), etc.

Exemplarily, FIG. 10 is a schematic structural diagram of a terminal device provided by another embodiment of the present application. As shown in FIG. 10, the terminal device 100 may include a processor 110, an external memory interface 120, an internal memory 121, and a universal serial bus ( universal serial bus, USB) interface 130, charging management module 140, power management module 141, battery 142, antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C , A headset interface 170D, a sensor 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. It can be understood that the structure illustrated in this embodiment does not constitute a specific limitation on the terminal device 100. In other embodiments of the present application, the terminal device 100 may include more or fewer components than shown, or combine certain components, or split certain components, or arrange different components. The illustrated components can be implemented by hardware, software, or a combination of software and hardware.

The processor 110 may include one or more processing units. For example, the processor 110 may include an AP, a modem processor, a GPU, an ISP, a controller, a video codec, and a digital signal processor (DSP). , Baseband processor, and/or neural-network processing unit (NPU), etc. Among them, the different processing units may be independent devices or integrated in one or more processors. In some embodiments, the terminal device 100 may also include one or more processors 110. The controller may be the nerve center and command center of the terminal device 100. The controller can generate operation control signals according to the instruction operation code and timing signals to complete the control of fetching and executing instructions. A memory may also be provided in the processor 110 to store instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory can store instructions or data that have just been used or recycled by the processor 110. If the processor 110 needs to use the instruction or data again, it can be directly called from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and improves the efficiency of the terminal device 100 system.

In some embodiments, the processor 110 may include one or more interfaces. The interface may include an integrated circuit (inter-integrated circuit, I2C) interface, an integrated circuit built-in audio (inter-integrated circuit sound, I2S) interface, a pulse code modulation (pulse code modulation, PCM) interface, and a universal asynchronous transmitter (universal asynchronous transmitter) interface. receiver/transmitter, UART) interface, MIPI, general-purpose input/output (GPIO) interface, subscriber identity module (SIM) interface, and/or USB interface, HDMI, V-By-One Interface, DP, etc., among them, V-By-One interface is a digital interface standard for image transmission development. Among them, the USB interface 130 is an interface that complies with the USB standard specification, and specifically may be a Mini USB interface, a Micro USB interface, a USB Type C interface, and so on. The USB interface 130 can be used to connect a charger to charge the terminal device 100, and can also be used to transfer data between the terminal device 100 and peripheral devices. It can also be used to connect headphones and play audio through the headphones.

It can be understood that the interface connection relationship between the modules illustrated in the embodiment of the present application is merely a schematic description, and does not constitute a structural limitation of the terminal device 100. In other embodiments of the present application, the terminal device 100 may also adopt different interface connection modes in the foregoing embodiments, or a combination of multiple interface connection modes.

The charging management module 140 is used to receive charging input from the charger. Among them, the charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive the charging input of the wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 may receive wireless charging input through the wireless charging coil of the terminal device 100. While the charging management module 140 charges the battery 142, it can also supply power to the terminal device 100 through the power management module 141.

The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and/or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the display screen 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle times, and battery health status (leakage, impedance). In some other embodiments, the power management module 141 may also be provided in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 may also be provided in the same device.

The wireless communication function of the terminal device 100 can be implemented by the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor, and the baseband processor. The antenna 1 and the antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the terminal device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna of a wireless local area network. In other embodiments, the antenna can be used in combination with a tuning switch.

The mobile communication module 150 may provide a wireless communication solution including 2G/3G/4G/5G and the like applied to the terminal device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier, etc. The mobile communication module 150 can receive electromagnetic waves by the antenna 1, and perform processing such as filtering, amplifying and transmitting the received electromagnetic waves to the modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor, and convert it into electromagnetic waves for radiation via the antenna 1. In some embodiments, at least part of the functional modules of the mobile communication module 150 may be provided in the processor 110. In some embodiments, at least part of the functional modules of the mobile communication module 150 and at least part of the modules of the processor 110 may be provided in the same device.

The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low frequency baseband signal to be sent into a medium and high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. Then the demodulator transmits the demodulated low-frequency baseband signal to the baseband processor for processing. The low-frequency baseband signal is processed by the baseband processor and then passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to the speaker 170A, the receiver 170B, etc.), or displays an image or video through the display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be provided in the same device as the mobile communication module 150 or other functional modules.

The wireless communication module 160 can provide applications on the terminal device 100, including wireless local area networks (wireless local area networks, WLAN), Bluetooth, global navigation satellite system (GNSS), frequency modulation (FM), NFC, Infrared technology (infrared, IR) and other wireless communication solutions. The wireless communication module 160 may be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, perform frequency modulation, amplify it, and convert it into electromagnetic wave radiation via the antenna 2.

In some embodiments, the antenna 1 of the terminal device 100 is coupled with the mobile communication module 150, and the antenna 2 is coupled with the wireless communication module 160, so that the terminal device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include GSM, GPRS, CDMA, WCDMA, TD-SCDMA, LTE, GNSS, WLAN, NFC, FM, and/or IR technology. The aforementioned GNSS may include global positioning system (GPS), global navigation satellite system (GLONASS), Beidou navigation satellite system (BDS), quasi-zenith satellite system (quasi- Zenith satellite system, QZSS) and/or satellite-based augmentation systems (SBAS).

The terminal device 100 can implement a display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute instructions to generate or change display information.

The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can adopt liquid crystal display (LCD), organic light-emitting diode (OLED), active-matrix organic light-emitting diode or active-matrix organic light-emitting diode (active-matrix organic light-emitting diode). AMOLED, flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, quantum dot light-emitting diode (QLED), etc. In some embodiments, the terminal device 100 may include one or N display screens 194, and N is a positive integer greater than one.

The terminal device 100 may implement a shooting function through an ISP, one or more cameras 193, a video codec, a GPU, one or more display screens 194, and an application processor.

NPU is a neural-network (NN) computing processor. By drawing on the structure of biological neural networks, for example, the transfer mode between human brain neurons, it can quickly process input information and can continuously learn by itself. The NPU can realize applications such as intelligent cognition of the terminal device 100, such as image recognition, face recognition, voice recognition, text understanding, and so on.

The external memory interface 120 may be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the terminal device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to realize the data storage function. For example, save music, photos, videos and other data files in an external memory card.

The internal memory 121 may be used to store one or more computer programs, and the one or more computer programs include instructions. The processor 110 can run the above-mentioned instructions stored in the internal memory 121 to enable the terminal device 100 to execute the screen control methods provided in some embodiments of the present application, as well as various functional applications and data processing. The internal memory 121 may include a storage program area and a storage data area. Among them, the storage program area can store the operating system; the storage program area can also store one or more application programs (such as a gallery, contacts, etc.) and so on. The data storage area can store data (such as photos, contacts, etc.) created during the use of the terminal device 100. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, a universal flash storage (UFS), etc. In some embodiments, the processor 110 may execute instructions stored in the internal memory 121 and/or instructions stored in a memory provided in the processor 110 to cause the terminal device 100 to execute the instructions provided in the embodiments of the present application. Screen control methods, as well as various functional applications and data processing.

The terminal device 100 can implement audio functions through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the earphone interface 170D, and the application processor. For example, music playback, recording, etc. Among them, the audio module 170 is used to convert digital audio information into an analog audio signal for output, and also used to convert an analog audio input into a digital audio signal. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 may be provided in the processor 110, or part of the functional modules of the audio module 170 may be provided in the processor 110. The speaker 170A, also called a "speaker", is used to convert audio electrical signals into sound signals. The terminal device 100 can listen to music through the speaker 170A, or listen to a hands-free call. The receiver 170B, also called "earpiece", is used to convert audio electrical signals into sound signals. When the terminal device 100 answers a call or voice message, it can receive the voice by bringing the receiver 170B close to the human ear. The microphone 170C, also called "microphone", "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can approach the microphone 170C through the mouth to make a sound, and input the sound signal to the microphone 170C. The terminal device 100 may be provided with at least one microphone 170C. In other embodiments, the terminal device 100 may be provided with two microphones 170C, which can implement noise reduction functions in addition to collecting sound signals. In other embodiments, the terminal device 100 may also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify sound sources, and realize directional recording functions. The earphone interface 170D is used to connect wired earphones. The earphone interface 170D can be a USB interface 130, a 3.5mm open mobile terminal platform (OMTP) standard interface, or a cellular telecommunications industry association of the USA (CTIA) Standard interface.

The sensor 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, and an ambient light sensor 180L , Bone conduction sensor 180M and so on.

Among them, the pressure sensor 180A is used to sense pressure signals, and can convert the pressure signals into electrical signals. In some embodiments, the pressure sensor 180A may be provided on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors and so on. The capacitive pressure sensor may be composed of at least two parallel plates with conductive material. When a force is applied to the pressure sensor 180A, the capacitance between the electrodes changes. The terminal device 100 determines the intensity of the pressure according to the change in capacitance. When a touch operation acts on the display screen 194, the terminal device 100 detects the intensity of the touch operation according to the pressure sensor 180A. The terminal device 100 may also calculate the touched position according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations that act on the same touch location but have different touch operation strengths may correspond to different operation instructions. For example: when a touch operation whose intensity of the touch operation is less than the first pressure threshold is applied to the short message application icon, an instruction to view the short message is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold acts on the short message application icon, an instruction to create a new short message is executed.

The gyro sensor 180B may be used to determine the movement posture of the terminal device 100. In some embodiments, the angular velocity of the terminal device 100 around three axes (ie, x, y, and z axes) can be determined by the gyro sensor 180B. The gyro sensor 180B can be used for image stabilization. Exemplarily, when the shutter is pressed, the gyroscope sensor 180B detects the shake angle of the terminal device 100, calculates the distance that the lens module needs to compensate according to the angle, and allows the lens to counteract the shake of the terminal device 100 through a reverse movement to achieve anti-shake. The gyro sensor 180B can also be used for navigation, somatosensory game scenes and so on.

The acceleration sensor 180E can detect the magnitude of the acceleration of the terminal device 100 in various directions (generally three-axis). When the terminal device 100 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of the terminal device, and is used in applications such as horizontal and vertical screen switching, and pedometer.

Distance sensor 180F, used to measure distance. The terminal device 100 can measure the distance by infrared or laser. In some embodiments, when shooting a scene, the terminal device 100 may use the distance sensor 180F to measure the distance to achieve fast focusing.

The proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector such as a photodiode. The light emitting diode may be an infrared light emitting diode. The terminal device 100 emits infrared light to the outside through the light emitting diode. The terminal device 100 uses a photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the terminal device 100. When insufficient reflected light is detected, the terminal device 100 can determine that there is no object near the terminal device 100. The terminal device 100 can use the proximity light sensor 180G to detect that the user holds the terminal device 100 close to the ear to talk, so as to automatically turn off the screen to save power. The proximity light sensor 180G can also be used in leather case mode, and the pocket mode will automatically unlock and lock the screen.

The ambient light sensor 180L is used to sense the brightness of the ambient light. The terminal device 100 can adaptively adjust the brightness of the display screen 194 according to the perceived brightness of the ambient light. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180L can also cooperate with the proximity light sensor 180G to detect whether the terminal device 100 is in a pocket to prevent accidental touch.

The fingerprint sensor 180H (also called a fingerprint reader) is used to collect fingerprints. The terminal device 100 can use the collected fingerprint characteristics to realize fingerprint unlocking, access application locks, fingerprint photographs, fingerprint answering calls, etc. In addition, other descriptions of the fingerprint sensor can be found in the international patent application PCT/CN2017/082773 entitled "Method and Terminal Device for Processing Notification", the entire content of which is incorporated in this application by reference.

The touch sensor 180K can also be called a touch panel or a touch-sensitive surface. The touch sensor 180K may be disposed on the display screen 194, and the touch screen is composed of the touch sensor 180K and the display screen 194, which is also called a touch screen. The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. The display screen 194 may provide visual output related to touch operations. In other embodiments, the touch sensor 180K may also be disposed on the surface of the terminal device 100, which is different from the position of the display screen 194.

The bone conduction sensor 180M can acquire vibration signals. In some embodiments, the bone conduction sensor 180M can obtain the vibration signal of the vibrating bone mass of the human voice. The bone conduction sensor 180M can also contact the human pulse and receive the blood pressure pulse signal. In some embodiments, the bone conduction sensor 180M may also be provided in the earphone, combined with the bone conduction earphone. The audio module 170 can parse the voice signal based on the vibration signal of the vibrating bone block of the voice obtained by the bone conduction sensor 180M, and realize the voice function. The application processor may analyze the heart rate information based on the blood pressure beat signal obtained by the bone conduction sensor 180M, and realize the heart rate detection function.

The button 190 includes a power button, a volume button, and so on. The button 190 may be a mechanical button or a touch button. The terminal device 100 may receive key input, and generate key signal input related to user settings and function control of the terminal device 100.

The SIM card interface 195 is used to connect to the SIM card. The SIM card can be inserted into the SIM card interface 195 or pulled out from the SIM card interface 195 to achieve contact and separation with the terminal device 100. The terminal device 100 may support 1 or N SIM card interfaces, and N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, etc. The same SIM card interface 195 can insert multiple cards at the same time. The types of the multiple cards can be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 may also be compatible with external memory cards. The terminal device 100 interacts with the network through the SIM card to realize functions such as call and data communication. In some embodiments, the terminal device 100 uses an eSIM, that is, an embedded SIM card. The eSIM card can be embedded in the terminal device 100 and cannot be separated from the terminal device 100.

In addition, the embodiments of the present application also provide a computer-readable storage medium. The computer-readable storage medium stores computer-executable instructions. When at least one processor of the user equipment executes the computer-executable instructions, the user equipment executes the aforementioned various possibilities. Methods.

Among them, the computer-readable medium includes a computer storage medium and a communication medium, and the communication medium includes any medium that facilitates the transfer of a computer program from one place to another. The storage medium may be any available medium that can be accessed by a general-purpose or special-purpose computer. An exemplary storage medium is coupled to the processor, so that the processor can read information from the storage medium and can write information to the storage medium. Of course, the storage medium may also be an integral part of the processor. The processor and the storage medium may be located in the ASIC. In addition, the ASIC may be located in the user equipment. Of course, the processor and the storage medium may also exist as discrete components in the communication device.

A person of ordinary skill in the art can understand that all or part of the steps in the foregoing method embodiments can be implemented by a program instructing relevant hardware. The aforementioned program can be stored in a computer readable storage medium. When the program is executed, the steps including the foregoing method embodiments are executed; and the foregoing storage medium includes: ROM, RAM, magnetic disk, or optical disk and other media that can store program codes.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the application, not to limit them; although the application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand: It is still possible to modify the technical solutions described in the foregoing embodiments, or equivalently replace some or all of the technical features; these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the technical solutions of the embodiments of the application range.

Claims

A method for controlling a screen, characterized in that it comprises:

Acquire the respective feature information of the N screen control objects in the first image and the respective action information of the N screen control objects, where the screen control objects are the body parts of the user in the first image, and N is greater than or equal to 2 Positive integer;

Determine a display sub-screen corresponding to each of the N screen control objects according to the respective characteristic information of the N screen control objects, where the display sub-screen is a partial area of the display screen;

According to the respective action information of the N screen control objects, the display sub-screens corresponding to the N screen control objects are controlled.
The method according to claim 1, characterized in that, before said acquiring the respective feature information of the N screen control objects and the respective action information of the N screen control objects in the first image, the method further comprises:

Acquiring a second image of the user;

Determining the number N of control objects in the second image that satisfy a preset activation action, where the preset activation action is used to activate a multi-gesture control mode;

N display sub-screens are presented on the display screen.
The method according to claim 2, characterized in that before said acquiring the respective characteristic information of the N screen control objects and the respective action information of the N screen control objects in the first image, the method further comprises:

Establish a first correspondence between the characteristic information of the N screen control objects that satisfy the preset activation action and the N display sub-screens, and the first correspondence includes the characteristic information of the control object and the display sub-screen. One correspondence

The determining a display sub-screen corresponding to each of the N screen control objects according to the respective characteristic information of the N screen control objects includes:

According to the first correspondence and the respective characteristic information of the N screen control objects, the display sub-screens corresponding to the N screen control objects are determined.
The method according to any one of claims 1 to 3, wherein the controlling the display sub-screens corresponding to each of the N screen control objects according to the respective action information of the N screen control objects comprises:

The target control operation matching the action information of the target control object is determined in the second correspondence, and the target control object is any one of the N control objects, and the second correspondence includes multiple One-to-one correspondence between one action information and multiple control operations;

Control the display sub-screen corresponding to the target control screen object according to the target control operation.
A screen control device, characterized by comprising:

The first acquisition module is configured to acquire the respective feature information of the N screen control objects in the first image and the respective action information of the N screen control objects, where the screen control objects are the body parts of the user in the first image , N is a positive integer greater than or equal to 2;

A first determining module, configured to determine a display sub-screen corresponding to each of the N screen control objects according to the respective characteristic information of the N screen control objects, the display sub-screen being a partial area of the display screen;

The control module is configured to control the display sub-screens corresponding to each of the N screen control objects according to the respective action information of the N screen control objects.
The device according to claim 5, further comprising:

The second acquisition module is used to acquire the second image of the user;

A second determining module, configured to determine the number N of control objects in the second image that satisfy a preset activation action, and the preset activation action is used to activate a multi-gesture control mode;

The segmentation module is used to obtain N display sub-screens presented on the display screen.
The device according to claim 6, further comprising:

The establishment module is configured to establish a first corresponding relationship between the feature information of the N screen control objects that meet the preset activation action and the N display sub-screens, and the first corresponding relationship includes the feature information of the control object and Display the one-to-one correspondence of the sub-screens;

The first determining module is specifically configured to:

According to the first correspondence and the respective characteristic information of the N screen control objects, the display sub-screens corresponding to the N screen control objects are determined.
The device according to any one of claims 5-7, wherein the control module is specifically configured to:

The target control operation matching the action information of the target control object is determined in the second correspondence, and the target control object is any one of the N control objects, and the second correspondence includes multiple One-to-one correspondence between one action information and multiple control operations;

Control the display sub-screen corresponding to the target control screen object according to the target control operation.
A device, characterized by comprising: a processor and a transmission interface,

The transmission interface is used to receive the first image of the user obtained by the camera;

The processor is configured to call software instructions stored in the memory to execute the following steps:

Acquire the respective feature information of the N screen control objects in the first image and the respective action information of the N screen control objects, where the screen control objects are the body parts of the user in the first image, and N is greater than or equal to 2 Positive integer;

Determine a display sub-screen corresponding to each of the N screen control objects according to the respective characteristic information of the N screen control objects, where the display sub-screen is a partial area of the display screen;

According to the respective action information of the N screen control objects, the display sub-screens corresponding to the N screen control objects are controlled.
The device according to claim 9, wherein:

The transmission interface is also used to receive the second image of the user obtained by the camera;

The processor is also used for:

Determining the number N of control objects in the second image that satisfy a preset activation action, where the preset activation action is used to activate a multi-gesture control mode;

Obtain N display sub-screens presented on the display screen.
The device according to claim 10, wherein the processor is further configured to:

Establish a first correspondence between the characteristic information of the N screen control objects that satisfy the preset activation action and the N display sub-screens, and the first correspondence includes the characteristic information of the control object and the display sub-screen. One correspondence

The processor is specifically used for:

According to the first correspondence and the respective characteristic information of the N screen control objects, the display sub-screens corresponding to the N screen control objects are determined.
The device according to any one of claims 9-11, wherein the processor is further configured to:

The target control operation matching the action information of the target control object is determined in the second correspondence, and the target control object is any one of the N control objects, and the second correspondence includes multiple One-to-one correspondence between one action information and multiple control operations;

Control the display sub-screen corresponding to the target control screen object according to the target control operation.
A computer-readable storage medium, characterized in that the computer-readable storage medium stores instructions, and when the instructions run on a computer or a processor, the computer or the processor executes as claimed in claims 1 to 4 Any one of the methods.