WO2023065849A1

WO2023065849A1 - Screen brightness adjustment method and apparatus for electronic device, and electronic device

Info

Publication number: WO2023065849A1
Application number: PCT/CN2022/116557
Authority: WO
Inventors: 孙哲; 王征添
Original assignee: Oppo广东移动通信有限公司
Priority date: 2021-10-19
Filing date: 2022-09-01
Publication date: 2023-04-27
Also published as: CN113903317B; CN113903317A

Abstract

A screen brightness adjustment method and apparatus for an electronic device, and an electronic device. The screen brightness adjustment method comprises: collecting a facial image by means of a camera of an electronic device (S110); processing the facial image by means of a neural network model, so as to acquire a fixation point position of human eyes on a screen (S120); on the basis of the fixation point position and the distance between the fixation point position and the camera, determining whether a fixation point is located within the screen (S130); and when the fixation point is not located within the screen, lowering the brightness of the screen (S140). A fixation point position is determined and detected by means of a neural network model, and when it is determined, according to the fixation point position and the distance between the fixation point position and a camera, that a fixation point is not located within a screen, the brightness of the screen is lowered, thereby reducing the power consumption of an electronic device, and improving the intelligent perceptual interaction experience.

Description

Method and device for adjusting screen brightness of electronic equipment, and electronic equipment

Cross References to Related Applications

This application claims priority to Chinese application No. CN202111217558.1 filed on October 19, 2021, which is hereby incorporated by reference in its entirety for all purposes.

technical field

The present application relates to the technical field of electronic equipment, and more specifically, to a method and device for adjusting screen brightness of electronic equipment and electronic equipment.

Background technique

With the development of science and technology, the use of electronic equipment is becoming more and more extensive, with more and more functions, and has become one of the must-haves in people's daily life. In order to reduce the power consumption of the electronic device and reduce the irritation to the user's eyes, the user can manually adjust the brightness of the screen of the electronic device.

Contents of the invention

In view of the above problems, the present application proposes a method and device for adjusting screen brightness of an electronic device, and an electronic device to solve the above problems.

In the first aspect, the embodiment of the present application provides a method for adjusting screen brightness of an electronic device, which is applied to an electronic device. The method includes: collecting a face image through a camera of the electronic device; processing the face image through a neural network model. face image to obtain the gaze point position of human eyes on the screen; based on the gaze point position and the distance between the gaze point position and the camera, determine whether the gaze point is located in the screen; when the When the gaze point is not within the screen, reduce the brightness of the screen.

In a second aspect, the embodiment of the present application provides a device for adjusting screen brightness of an electronic device, which is applied to an electronic device, and the device includes: a face image acquisition module, configured to collect a face image through a camera of the electronic device; The fixation point position acquisition module is used to process the human face image through the neural network model to obtain the fixation point position of the human eye on the screen; the fixation point determination module is used for based on the fixation point position and the fixation point The distance between the position and the camera is used to determine whether the gaze point is located within the screen; the screen brightness reduction module is configured to reduce the brightness of the screen when the gaze point is not located within the screen.

In a third aspect, the embodiment of the present application provides an electronic device, including a memory and a processor, the memory is coupled to the processor, the memory stores instructions, and when the instructions are executed by the processor, the The processor executes the above method.

In a fourth aspect, the embodiment of the present application provides a computer-readable storage medium, where program code is stored in the computer-readable storage medium, and the program code can be invoked by a processor to execute the above method.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings that need to be used in the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application. For those skilled in the art, other drawings can also be obtained based on these drawings without any creative effort.

FIG. 1 shows a schematic flowchart of a method for adjusting screen brightness of an electronic device provided by an embodiment of the present application;

Figure 2 shows a schematic diagram of the first scenario provided by the embodiment of the present application;

Figure 3 shows a schematic diagram of the second scenario provided by the embodiment of this application

FIG. 4 shows a schematic flowchart of a method for adjusting screen brightness of an electronic device provided by an embodiment of the present application;

FIG. 5 shows a schematic diagram of a third scenario provided by the embodiment of the present application;

FIG. 6 shows a schematic diagram of a fourth scenario provided by the embodiment of the present application;

FIG. 7 shows a schematic flowchart of a method for adjusting screen brightness of an electronic device provided by an embodiment of the present application;

FIG. 8 shows a schematic flowchart of step S350 of the method for adjusting screen brightness of an electronic device shown in FIG. 7 of the present application;

FIG. 9 shows a schematic flowchart of a method for adjusting screen brightness of an electronic device provided by an embodiment of the present application;

FIG. 10 shows a schematic flowchart of a method for adjusting screen brightness of an electronic device provided by an embodiment of the present application;

FIG. 11 shows a schematic flowchart of a method for adjusting screen brightness of an electronic device provided by an embodiment of the present application;

FIG. 12 shows a schematic flowchart of a method for adjusting screen brightness of an electronic device provided by an embodiment of the present application;

FIG. 13 shows a block diagram of a device for adjusting screen brightness of an electronic device according to an embodiment of the present application;

FIG. 14 shows a block diagram of an electronic device used to implement a method for adjusting screen brightness of an electronic device according to an embodiment of the present application;

Fig. 15 shows a storage unit for storing or carrying program codes for implementing the method for adjusting screen brightness of an electronic device according to an embodiment of the present application according to an embodiment of the present application.

Detailed ways

In order to enable those skilled in the art to better understand the solutions of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

Wherein, in order to reduce power consumption of the electronic device and reduce irritation to the user's eyes, the user may manually adjust the brightness of the screen of the electronic device. After research, the inventor found that this adjustment method is cumbersome and the adjustment accuracy is low, which leads to the failure to reduce the power consumption of the electronic device and affects the user experience.

In view of the above problems, the inventor found after long-term research, and proposed the screen brightness adjustment method, device and electronic equipment of the electronic equipment provided in the embodiment of the present application, which judges and detects the position of the fixation point through the detection model, and according to the fixation point The location controls the brightness of the screen, thereby reducing the power consumption of electronic devices and improving the intelligent perception interaction experience. Wherein, a specific method for adjusting screen brightness of an electronic device will be described in detail in subsequent embodiments.

Please refer to FIG. 1 , which shows a schematic flowchart of a method for adjusting screen brightness of an electronic device according to an embodiment of the present application. The method is used to judge and detect the gaze point position through the neural network model, and according to the gaze point position and the distance between the gaze point position and the camera, it is determined that the gaze point is not located in the screen and the screen brightness is lowered, thereby reducing the power of the electronic device. consumption, and improve the intelligent perception interaction experience. In a specific embodiment, the method for adjusting screen brightness of an electronic device is applied to a screen brightness adjusting device 200 of an electronic device as shown in FIG. 13 and an electronic device 100 equipped with the screen brightness adjusting device 200 of an electronic device ( FIG. 14 . The following will take an electronic device as an example to illustrate the specific process of this embodiment. Of course, it can be understood that the electronic device applied in this embodiment may be a smart phone, a tablet computer, a wearable electronic device, etc., which is not limited here. In this embodiment, the electronic device includes a camera and a screen. The process shown in FIG. 1 will be described in detail below. The screen brightness adjustment method of the electronic device may specifically include the following steps:

Step S110: collecting face images through the camera of the electronic device.

In this embodiment, the electronic device may include a camera, and image collection may be performed through the camera. For example, collect face images. As a manner, the camera may be a front camera, a rear camera, or a rotating camera, etc., which is not limited herein.

In some embodiments, the camera can be an always-on device built into the electronic device, wherein the always-on device can be a low-power image acquisition device and a sound acquisition device that can be kept always on.

In some embodiments, the camera can be controlled to collect face images in real time; the camera can be controlled to collect face images at preset time intervals; the camera can be controlled to collect face images at preset time points; the camera can be controlled to collect face images based on operations command to collect face images; the camera can be controlled to collect face images according to other preset rules, which are not limited here.

As a way, when the time is close to the off-screen time set by the user, the camera can be controlled to collect face images.

Step S120: Processing the face image through a neural network model to obtain gaze points of human eyes on the screen.

In this embodiment, a neural network model can be obtained in advance, and the neural network model can be used for gaze point position detection. In one embodiment, after the face image is acquired, the face image can be input into the neural network model to obtain information output by the neural network model.

In some implementations, the neural network model may be stored on the electronic device after being trained by the electronic device, or may be stored on the electronic device after being trained by other devices or servers, which is not limited herein. Wherein, in the case where the neural network model is stored locally on the electronic device, after the face image is obtained, the neural network model can be directly called locally on the electronic device, for example, an instruction can be directly sent to the neural network model to instruct The neural network model reads the face image in the target storage area, or the electronic device can directly input the face image into the local neural network model, thereby effectively avoiding the reduction of the face image input neural network model due to the influence of network factors The speed to improve the speed of the neural network model to obtain face images and improve user experience.

In some embodiments, the neural network model may be stored in the server after being trained by the electronic device, or may be stored in the server after being trained by other devices or servers, which is not limited here. Wherein, in the case where the neural network model is stored in the server, after the face image can be obtained, an instruction can be sent to the neural network model stored in the server through the network, to instruct the neural network model to read the camera information of the electronic device through the network. The collected face image, or the electronic device can send the face image to the neural network model stored in the server through the network, so that by storing the neural network model in the server, the occupation of the storage space of the electronic device is reduced, and the Effects on the normal operation of electronic equipment.

In an embodiment of the present application, since the neural network model can be used for gaze point position detection. Therefore, when the neural network model detects that the human face image includes an eye image, the information output by the neural network model may include the gaze point position of the eyeball, which will not be repeated here.

Step S130: Based on the gaze point position and the distance between the gaze point position and the camera, determine whether the gaze point is located within the screen.

In this embodiment, when it is determined that the information output by the neural network model includes the gaze point position of the eyeball, it may be determined whether the gaze point is within the screen based on the gaze point position and the distance between the gaze point position and the camera. For example, when the position of the gaze point is within the corresponding position area of the screen, it can be determined that the gaze point is located within the screen; when the position of the gaze point is not within the corresponding position area of the screen, it can be determined that the gaze point is not located within the screen.

Step S140: Decrease the brightness of the screen when the gaze point is not located in the screen.

In an embodiment of the present application, controlling the brightness of the screen may include one of the following: increasing the brightness of the screen, maintaining the brightness of the screen, or reducing the brightness of the screen. Wherein, reducing the brightness of the screen may include turning off the brightness of the screen.

In this embodiment, when it is determined that the point of gaze is not located on the screen, the brightness of the screen can be reduced, so that when it is determined that the user does not pay attention to the content displayed on the screen of the electronic device, the brightness of the electronic device can be reduced by reducing the brightness of the screen. power consumption. In some implementations, the electronic device can preset and store the position of the camera. When the information output by the neural network model includes the gaze point position of the eyeball, the distance between the gaze point position and the camera can be obtained, and based on the The gaze point position and the distance between the gaze point position and the camera. When it is determined that the gaze point is not within the screen of the electronic device, the brightness of the screen is lowered.

Please refer to FIG. 2 . FIG. 2 shows a schematic diagram of a first scenario provided by an embodiment of the present application. As shown in FIG. 2 , the user holds the electronic device 100. At this time, the electronic device 100 is in a vertical display state, and the user's gaze point is not on the screen. At this time, the brightness of the screen can be reduced or the screen can be turned off.

Please refer to FIG. 3 . FIG. 3 shows a schematic diagram of a second scenario provided by the embodiment of the present application. As shown in FIG. 3 , the user holds the electronic device 100. At this time, the electronic device 100 is in a landscape display state, and the user's gaze point is not on the screen. At this time, the brightness of the screen can be reduced or the screen can be turned off.

The method for adjusting the screen brightness of an electronic device provided by an embodiment of the present application collects a face image through a camera of the electronic device, processes the face image through a neural network model to obtain the fixation point position of the human eye on the screen, based on the fixation point position and The distance between the gaze point position and the camera determines whether the gaze point is within the screen. When the gaze point is not within the screen, lower the brightness of the screen to judge and detect the gaze point position through the neural network model, and according to the gaze point position and The distance between the position of the gaze point and the camera is determined to reduce the brightness of the screen when the gaze point is not located on the screen, thereby reducing the power consumption of electronic devices and improving the intelligent perception interaction experience.

Please refer to FIG. 4. FIG. 4 shows a method for adjusting screen brightness of an electronic device provided by an embodiment of the present application. This method is applied to the above-mentioned electronic device. The method for adjusting the screen brightness of the device may specifically include the following steps:

Step S210: collecting face images through the camera of the electronic device.

Step S220: Processing the face image through a neural network model to obtain gaze points of human eyes on the screen.

Step S230: Based on the gaze point position and the distance between the gaze point position and the camera, determine whether the gaze point is located within the screen.

Step S240: Decrease the brightness of the screen when the gaze point is not located in the screen.

Wherein, for the specific description of step S210-step S240, please refer to step S110-step S140, which will not be repeated here.

Step S250: When the gaze point is located in the screen, maintain the current brightness of the screen.

Wherein, based on the gaze point position and the distance between the gaze point position and the camera, when it is determined that the gaze point position falls within the screen, it is determined that the gaze point is located within the screen, indicating that the user is watching the content displayed on the screen or expects to view the content of the screen , the brightness of the screen can be increased or the current brightness of the screen can be maintained, so as to meet the content viewing needs of the user.

Please refer to FIG. 5 , which shows a schematic diagram of a third scenario provided by the embodiment of the present application. As shown in FIG. 5 , the user holds the electronic device 100. At this time, the electronic device 100 is in a vertical display state, and the user's gaze point is on the screen. At this time, the brightness of the screen can be increased or maintained.

Please refer to FIG. 6 , which shows a schematic diagram of a fourth scenario provided by the embodiment of the present application. As shown in FIG. 6 , the user holds the electronic device 100. At this time, the electronic device 100 is in a landscape display state, and the user's gaze point is on the screen. At this time, the brightness of the screen can be increased or maintained.

Compared with the screen brightness adjustment method shown in FIG. 1 , the screen brightness adjustment method of an electronic device provided by an embodiment of the present application maintains the current brightness of the screen when the gaze point is located on the screen, so as to improve the user's brightness. Use experience.

Please refer to FIG. 7. FIG. 7 shows a method for adjusting the screen brightness of an electronic device provided by an embodiment of the present application. This method is applied to the above-mentioned electronic device. The method for adjusting the screen brightness of the device may specifically include the following steps:

Step S310: collecting face images through the camera of the electronic device.

Step S320: Processing the face image through a neural network model to obtain gaze points of human eyes on the screen.

Step S330: Based on the gaze point position and the distance between the gaze point position and the camera, determine whether the gaze point is located within the screen.

Wherein, for the specific description of step S310-step S330, please refer to step S110-step S130, which will not be repeated here.

Step S340: When the gaze point is not located in the screen, determine the operation mode of the electronic device.

Wherein, the operation mode of the electronic device may include a smart screen-off mode or a non-smart screen-off mode. As a method, in the smart off-screen mode, the electronic device can automatically adjust the brightness of the screen according to the needs of the user, such as increasing the brightness of the screen or reducing the brightness of the screen. As a method, in the non-intelligent off-screen mode, the electronic device can automatically control the screen to turn off or turn on the screen according to the user's needs, such as keeping the screen on or controlling the screen to turn off. In this embodiment, when the gaze point is not located on the screen, the operation mode of the electronic device can be determined, that is, it can be determined whether the operation mode of the electronic device is a smart off-screen mode or a non-smart off-screen mode.

In an embodiment of the present application, when the operating mode of the electronic device is the smart screen-off mode, if it is determined that the gaze point is not located within the screen, the brightness of the screen may be controlled in a first manner. When the operation mode of the electronic device is the non-smart screen-off mode, if it is determined that the gaze point is not located in the screen, the brightness of the screen may be controlled in a second manner. Wherein, the first manner is different from the second manner.

In some implementations, an electronic device can obtain its operating mode. For example, the electronic device may be provided with a target switch, and the target switch is used to control the electronic device to switch between the smart screen-off mode and the non-smart screen-off mode. When it is detected that the target switch is in the on state, it can be determined that the operation mode of the electronic device is the smart screen off mode, and when it is detected that the target switch is in the off state, it can be determined that the operation mode of the electronic device is the non-smart screen off mode . Wherein, the target switch may be in an on state or an off state based on a touch operation acting on the screen.

In some implementations, an electronic device can obtain its operating mode. For example, the automatic off-screen duration set by the electronic device can be obtained. When the automatic off-screen duration is greater than or equal to the duration threshold, it can be determined that the electronic device is operating in the smart off-screen mode. When the automatic off-screen duration is less than the duration threshold , it can be determined that the electronic device is operating in a non-smart screen-off mode. For example, the duration threshold is 1 minute, 2 minutes, etc. Among them, the automatic off-screen duration is used to represent the duration of the electronic device entering the off-screen without any operation when the screen is on. Therefore, if the automatic off-screen duration set by the electronic device is less than the duration threshold, it can be considered that the electronic device enters the off-screen from the bright screen. If the duration of the electronic device is short and will not cause excessive power consumption of the electronic device, you can set a non-smart off-screen mode; If the screen lasts for a long time, it will cause excessive power consumption of electronic equipment, so you can set the smart off-screen mode.

Step S350: Decrease the brightness of the screen according to the operation mode.

In some implementations, after the operation mode of the electronic device is determined, the brightness of the screen can be adjusted down based on the operation mode. For example, when it is determined that the operating mode of the electronic device is the smart screen-off mode, the brightness of the screen can be reduced in the first way; screen brightness.

Please refer to FIG. 8 , which shows a schematic flowchart of step S350 of the method for adjusting screen brightness of an electronic device shown in FIG. 7 of the present application. The process shown in Figure 8 will be described in detail below, and the method may specifically include the following steps:

Step S351: When the operation mode is the smart screen-off mode, reduce the brightness of the screen.

In some implementations, the electronic device operates in a non-smart screen-off mode.

In some implementations, when the operating mode of the electronic device is the smart off-screen mode, if it is determined that the point of gaze is not located on the screen, it can be considered that the user is not looking at the content displayed on the screen or does not expect to view the content of the screen, then the user can lower the The brightness of the screen can reduce the power consumption of the electronic device, and at the same time, keep the display of the electronic device at a low brightness, so as to avoid the loss and power consumption caused by switching the screen multiple times.

Wherein, after reducing the brightness of the screen, if it is determined that the gaze point is not on the screen within the preset time period, the screen will be turned off; increase the screen brightness.

In some implementations, after the brightness of the screen is lowered, the position of the gaze point can be continuously detected within a preset period of time, the position of the gaze point and the distance between the position of the gaze point and the camera can be obtained, and it can be judged whether the gaze point is within the screen, Wherein, the preset duration represents the duration from the current time to the set off-screen time.

Among them, when it is determined that the point of gaze is not on the screen continuously within the preset duration, it can be considered that the user has not focused on the content displayed on the screen when the current time reaches the set off-screen time, and the screen can be turned off. , to reduce the power consumption of electronic equipment.

Wherein, when it is determined within the preset duration that the point of gaze is switched from not being on the screen to being on the screen, it can be considered that the user switches from not watching the content displayed on the screen to watching the screen before the current time reaches the set off-screen time. The content displayed on the screen can be adjusted to increase the brightness of the screen.

Step S352: When the operation mode is the non-smart off-screen mode, turn off the screen.

In some implementations, when the operating mode of the electronic device is the non-smart off-screen mode, if it is determined that the point of gaze is not located on the screen, it can be considered that the user is not looking at the content displayed on the screen or does not expect to view the content of the screen, then you can directly Turn off the screen to reduce the power consumption of electronic equipment.

The method for adjusting the screen brightness of an electronic device provided by an embodiment of the present application, compared with the method for adjusting the screen brightness of an electronic device shown in FIG. 1 , this embodiment also determines the operating mode of the electronic device, and adjusts the The brightness of the screen of the electronic device is improved, thereby improving the diversity and rationality of screen brightness adjustment.

Please refer to FIG. 9. FIG. 9 shows a method for adjusting the screen brightness of an electronic device provided by an embodiment of the present application. This method is applied to the above-mentioned electronic device. In this embodiment, the gaze point position includes the gaze point in the target coordinate system The coordinate information of the target coordinate system is established with the center position of the camera as the origin. The process shown in FIG. 9 will be described in detail below. The method for adjusting the screen brightness of the electronic device may specifically include the following steps:

Step S410: collecting face images through the camera of the electronic device.

Step S420: Processing the face image through a neural network model to obtain gaze points of human eyes on the screen.

Wherein, for the specific description of step S410-step S420, please refer to step S110-step S120, which will not be repeated here.

Step S430: Obtain the distance between the gaze point position and the camera based on the coordinate information.

As an implementable manner, the electronic device may preliminarily establish a coordinate system with the center of the camera as the origin as the target coordinate system. For example, the electronic device may pre-establish a coordinate system with the center of the camera as the origin, with the vertical axis of the electronic device as the X axis, and with the horizontal axis of the electronic device as the Y axis, as the target coordinate system. For another example, the electronic device may pre-establish a coordinate system with the center of the camera as the origin, with the horizontal axis of the electronic device as the X axis, and with the vertical axis of the electronic device as the Y axis, as the target coordinate system.

In some implementations, the gaze point position output by the neural network model can be the coordinate information of the gaze point in the target coordinate system. Therefore, when obtaining the gaze point position output by the neural network model, the gaze point position can be obtained at the Coordinate information in the target coordinate system.

Wherein, after the target coordinate system is established, the coordinate range of the screen of the electronic device under the target coordinate system may be determined and the coordinate information of each position on the screen under the target coordinate system may be determined. Therefore, in this embodiment, after obtaining the coordinate information of the gaze point position in the target coordinate system, the coordinate information of the gaze point position in the target coordinate system can be compared with the coordinate range to determine that the gaze point position is within the target coordinate system. Whether the coordinate information in the coordinate system is within the coordinate range. Wherein, when it is determined that the coordinate information of the gaze point position in the target coordinate system is within the coordinate range, it can be determined that the gaze point position is on the screen, and the brightness of the screen can be maintained or increased. Wherein, when it is determined that the coordinate information of the gaze point position in the target coordinate system is not within the coordinate range, it can be determined that the gaze point position is not on the screen, and then the brightness of the screen can be reduced or the screen can be turned off.

In some implementations, since the target coordinate system is established with the center of the camera as the origin, the coordinate information of the camera can be (0, 0) by default. Therefore, after obtaining the coordinate information corresponding to the gaze point position, it can be based on this The coordinate information determines the distance of the gaze point position from the camera.

Step S440: Determine whether the gaze point is located within the screen based on the distance between the gaze point position and the camera.

In some implementation manners, after obtaining the distance between the gaze point position and the camera, it may be determined whether the gaze point is within the screen based on the distance between the gaze point position and the camera.

Step S450: Decrease the brightness of the screen when the gaze point is not located in the screen.

Wherein, for the specific description of step S450, please refer to step S140, which will not be repeated here.

In the method for adjusting the screen brightness of an electronic device provided by an embodiment of the present application, compared with the method for adjusting the screen brightness of an electronic device shown in FIG. The coordinate information in the target coordinate system, based on the coordinate information, obtains the distance between the gaze point position and the camera, and based on the distance between the gaze point position and the camera, determines whether the gaze point is within the screen, thereby improving the accuracy of determining the relationship between the gaze point position and the screen sex.

Please refer to FIG. 10. FIG. 10 shows a method for adjusting screen brightness of an electronic device provided by an embodiment of the present application. This method is applied to the above-mentioned electronic device. The method for adjusting the screen brightness of the device may specifically include the following steps:

Step S510: collecting face images through the camera of the electronic device.

Step S520: Processing the face image through a neural network model to obtain gaze points of human eyes on the screen.

Wherein, for the specific description of step S510-step S520, please refer to step S110-step S120, which will not be repeated here.

Step S530: Based on the gaze point position and the camera position, obtain the relative distance and relative direction between the gaze point position and the camera position.

In some implementations, the electronic device can preset and store the position of the camera. When the information output by the neural network model includes the position of the gaze point of the eyeball, the position of the gaze point can be obtained based on the position of the gaze point and the position of the camera. The relative distance and relative direction from the position of the camera. As a way, the electronic device can pre-set and store the coordinate information of the center of the camera. When the information output by the neural network model includes the coordinate information of the gaze point position of the eyeball, the real physical distance represented by a single screen pixel can be converted , to calculate the distance from the gaze point position to the center of the camera as the relative distance between the gaze point position and the camera position. As a method, the electronic device can be preset and stored with the coordinate information of the center of the camera. When the information output by the neural network model includes the coordinate information of the gaze point position of the eyeball, it can be based on the coordinate information of the camera center and the gaze point position. The coordinate information of the gaze point position and the direction of the center of the camera are determined as the relative direction between the gaze point position and the position of the camera.

Step S540: Based on the relative distance and the relative direction, determine whether the gaze point is located within the screen.

Wherein, after obtaining the relative distance and relative direction between the position of the gaze point and the position of the camera, it may be determined based on the relative distance and relative direction whether the gaze point is located within the screen.

For example, the camera is set at the top of the screen, when the relative direction of the gaze point relative to the position of the camera is from the top of the screen to the bottom of the screen, and the relative distance is L1. If the length of the screen of the electronic device is L2, and L2>L1, it can be determined that the gaze point is located within the screen. If the length of the screen of the electronic device is L3, and L1>L3, it can be determined that the gaze point is not located within the screen.

For another example, the camera is set at the top of the screen, and when the relative direction of the position of the gaze point relative to the position of the camera is from the bottom of the screen to the top of the screen, it can be determined that the gaze point is not located in the screen.

Step S550: Decrease the brightness of the screen when the gaze point is not located in the screen.

Wherein, for the specific description of step S550, please refer to step S140, which will not be repeated here.

Compared with the method for adjusting the screen brightness of an electronic device shown in FIG. 1 , the method for adjusting the screen brightness of an electronic device provided by an embodiment of the present application is also based on the position of the point of gaze and the position of the camera to obtain the position of the point of gaze and the position of the camera. The relative distance and relative direction of the location, based on the relative distance and relative direction, determine whether the gaze point is within the screen, thereby improving the accuracy of determining the relationship between the gaze point position and the screen.

Please refer to FIG. 11. FIG. 11 shows a method for adjusting the screen brightness of an electronic device provided by an embodiment of the present application. This method is applied to the above-mentioned electronic device. The process shown in FIG. The method for adjusting the screen brightness of the device may specifically include the following steps:

Step S610: collecting face images through the camera of the electronic device.

Wherein, for the specific description of step S610, please refer to step S110, which will not be repeated here.

Step S620: When there are multiple facial images collected by the camera, the facial image with the largest area among the multiple facial images is used as the target facial image.

In some embodiments, the image collected by the camera may or may not include a face image. Therefore, in this embodiment, the neural network model includes a face detection model and a gaze point detection model to detect whether the image collected by the camera includes a face image. As a way, the input of the face detection model can be the image collected by the camera (hereinafter referred to as the image to be detected), the output of the face detection model can be used as the input of the gaze point detection model, and the gaze point detection model can be based on the input and output Gaze point location.

In some embodiments, after the image to be detected is obtained, the image to be detected can be input into the face detection model, so as to detect whether the image to be detected includes a face image through the face detection model, and obtain the face detection model. information output by the model. Wherein, the information output by the face detection model may include one of the following: including a face image and not including a face image. As a way, when the information output by the face detection model does not include a face image, the gaze point detection model may not be invoked, and the brightness of the screen of the electronic device may be directly reduced or the screen may be turned off. As a way, when the information output by the face detection model includes a face image, the gaze point detection model can be called, and the face image can be input into the gaze point detection model to determine the gaze point position through the gaze point detection model. detection.

In some implementations, the information center output by the face detection model includes a face image may include one of the following: including a face image and including an eye image in the face image; including a face image and not including an eye image in the face image . Wherein, when the image to be detected collected by the camera is the front face image of the user, the information input by the face detection model based on the image to be detected may include a face image and the face image may include an eye image. When the image to be detected collected by the camera is a side face image of the user, the information input by the face detection model based on the image to be detected may include a face image and the face image does not include an eye image. As a way, when the information output by the face detection model includes a face image and the face image does not include an eye image, the gaze point detection model can no longer be called, and the brightness of the screen of the electronic device can be directly reduced or the control The screen goes off. As a way, when the information output by the face detection model includes a face image and the face image includes an eye image, the gaze point detection model can be called, and the face image can be input into the gaze point detection model to pass The gaze detection model performs gaze location detection.

In an embodiment of the present application, after the image to be detected is obtained, the image to be detected can be input into the face detection model, and the face detection model can detect the image to be detected, and detect the face frame in the image to be detected position and the position of multiple key points (such as the position of the center of the two eyes, the position of the center of the nose, and the position of the two corners of the mouth, etc.), and then extract the face image and eye image from the image to be detected and generate a representation of the face position mask.

In this embodiment, when it is determined that the information output by the face detection model includes a face image, the face image may be input into the gaze point detection model to obtain information output by the gaze point detection model. Wherein, the information output by the gaze point detection model may include one of the following: a gaze point position is detected, and a gaze point position is not detected. As a way, when the face image output by the face detection model is a side face image (not including the eye image), then the gaze point detection model is based on the information output by the face image for not detecting the gaze point position, then It is possible to directly reduce the brightness of the screen of the electronic device or control the screen to turn off. As a way, when the face image output by the face detection model is a frontal face image (including an eye image), the brightness of the screen can be controlled based on the gaze point position.

In some implementations, after obtaining the face image output by the face detection model, it may be determined whether the face image satisfies a preset face condition. When judging that the face image satisfies the preset face condition, the face image can be input into the gaze point detection model; when it is judged that the face image does not meet the preset face condition, the face image can not be Input the gaze point detection model, and directly reduce the brightness of the screen of the electronic device or control the screen to turn off. As a method, the preset face condition can be a complete face, that is, when the face image includes a complete face, it can be determined that the face image meets the preset face condition, and when the face image contains If the complete face is not included, it can be determined that the face image does not satisfy the preset face condition.

In some cases, the camera may capture multiple faces at the same time during the image collection process, that is, the acquired image to be detected may also include multiple faces. Therefore, the neural network model is based on the output information of the image to be detected. May include multiple face images.

In an embodiment of the present application, when the information output by the neural network model includes only one face image, the one face image may be directly input into the gaze point detection model.

In an embodiment of the present application, when the information output by the neural network model includes multiple face images, the face image of the user corresponding to the electronic device may be determined from the multiple face images as the target face image, and the The determined target face image is input into the neural network model. As a manner, when there are multiple face images collected by the camera, the face image with the largest area may be selected from the multiple face images as the target face image. As a manner, when there are multiple face images collected by the camera, the frontmost face image may be selected from the multiple face images as the target face image. As a method, when the number of face images collected by the camera is multiple, the face image matching the preset face image can be selected from the multiple face images as the target face image.

Step S630: Process the target face image through the neural network model to obtain the gaze point of human eyes on the screen.

In this embodiment, when it is determined that the target face image should be selected from the plurality of face images, the target face image is processed through the neural network model to obtain the fixation point position of the human eye on the screen.

Step S640: Based on the gaze point position and the distance between the gaze point position and the camera, determine whether the gaze point is located within the screen.

Step S650: Decrease the brightness of the screen when the gaze point is not located in the screen.

Wherein, for the specific description of step S640-step S650, please refer to step S130-step S140, which will not be repeated here.

The method for adjusting the screen brightness of an electronic device provided by an embodiment of the present application is compared with the method for adjusting the screen brightness of an electronic device shown in FIG. The face image with the largest area in the face image is used as the target face image, and the target face image is processed through the neural network model to obtain the position of the fixation point of the face on the screen, so as to improve the accuracy of determining the fixation point of the face.

Please refer to FIG. 12. FIG. 12 shows a method for adjusting screen brightness of an electronic device provided by an embodiment of the present application. This method is applied to the above-mentioned electronic device. The process shown in FIG. The method for adjusting the screen brightness of the device may specifically include the following steps:

Step S710: Obtain a training data set, wherein the training data set includes a plurality of training images and gaze point positions corresponding to the plurality of training images.

In this embodiment, a training data set can be obtained. The training data set includes a plurality of training images and a plurality of fixation point positions, wherein a plurality of training images and a plurality of fixation point positions correspond one-to-one, that is, each training image in the plurality of training images corresponds to a plurality of fixation points A fixation position in position.

In some implementations, the multiple training images can be obtained by the camera of the electronic device, can be obtained by the local storage of the electronic device, can be obtained by the electronic device from other devices, or can be obtained by the electronic device from a server. Do limited.

Wherein, the multiple gaze point positions may be manually marked on the training image by the user, or automatically marked on the training image by the electronic device, which is not limited here.

As a method, a label frame for identifying the position of the gaze point can be added to the training image to form a label image with a label frame, or the training image can be marked in the form of an xml format file, which is not limited here.

Step S720: Based on the plurality of training images and gaze point positions corresponding to the plurality of training images, train a preset model to obtain a neural network model.

In this embodiment, after the training data set is acquired, the preset model can be trained to obtain the gaze point detection model based on the training data set. As a method, after acquiring multiple training images and respective gaze point positions corresponding to the multiple training images, the preset model can be trained based on the multiple training images and the respective gaze point positions corresponding to the multiple training images Get the gaze detection model. For example, after obtaining multiple training images and multiple gaze point positions in one-to-one correspondence, multiple training images can be used as input parameters, and multiple gaze point positions can be used as output parameters to train the preset model to obtain a neural network model .

In addition, after training the neural network model, you can also verify the accuracy of the neural network model, and judge whether the output information of the neural network model based on the input data meets the preset requirements. When the neural network model based on the input data When the output information does not meet the preset requirements, you can re-collect the training data set to train the preset model, or obtain multiple training data sets to correct the trained neural network model, so that the error is as close to 0 as possible. As a manner, the preset model may be a convolutional neural network.

Step S730: collecting face images through the camera of the electronic device.

Step S740: Processing the face image through a neural network model to obtain gaze points of human eyes on the screen.

Step S750: Determine whether the gaze point is located within the screen based on the gaze point position and the distance between the gaze point position and the camera.

Step S760: Decrease the brightness of the screen when the gaze point is not located in the screen.

Wherein, for the specific description of step S730-step S760, please refer to step S110-step S140, which will not be repeated here.

Compared with the method for adjusting the screen brightness of an electronic device shown in FIG. 1 , the method for adjusting the screen brightness of an electronic device provided by an embodiment of the present application also acquires a training data set based on multiple training images and The gaze point positions corresponding to multiple training images are used to train the preset model to obtain a neural network model to improve the accuracy of model recognition.

Please refer to FIG. 13 . FIG. 13 shows a device for adjusting screen brightness of an electronic device provided by the present application. The screen brightness adjustment device 200 of the electronic equipment includes: a human face image acquisition module 210, a fixation point position acquisition module 220, a fixation point determination module 230 and a screen brightness reduction module 240, wherein:

The facial image acquisition module 210 is configured to acquire facial images through the camera of the electronic device.

The gaze point position obtaining module 220 is configured to process the face image through a neural network model to obtain the gaze point position of human eyes on the screen.

Further, the gaze point position acquisition module 220 includes: a target face image determination submodule and a gaze point position acquisition submodule, wherein:

The target human face image determination sub-module is used for taking the human face image with the largest area among the multiple human face images as the target human face image when there are multiple human face images collected by the camera.

The gaze point position obtaining submodule is used to process the target face image through the neural network model to obtain the gaze point position of human eyes on the screen.

The gaze point determination module 230 is configured to determine whether the gaze point is located within the screen based on the gaze point position and the distance between the gaze point position and the camera.

Further, the gaze point position includes coordinate information of the gaze point in a target coordinate system, and the target coordinate system is established with the center position of the camera as the origin, and the gaze point determination module 230 includes: a distance acquisition sub- module and the first fixation point determination submodule, wherein:

The distance acquisition submodule is configured to acquire the distance between the gaze point position and the camera based on the coordinate information.

The first gaze point determining submodule is configured to determine whether the gaze point is located within the screen based on the distance between the gaze point position and the camera.

Further, the gaze point determination module 230 includes: a relative relationship acquisition submodule and a second gaze point determination submodule, wherein:

The relative relationship acquisition submodule is configured to acquire the relative distance and direction between the gaze point position and the camera position based on the gaze point position and the camera position.

The second gaze point determination submodule is configured to determine whether the gaze point is located within the screen based on the relative distance and the relative direction.

The screen brightness reduction module 240 is configured to reduce the screen brightness when the gaze point is not located in the screen.

Further, the screen brightness reduction module 240 includes: an operating mode determination sub-module and a screen brightness reduction sub-module, wherein:

The operation mode determination submodule is used to determine the operation mode of the electronic device.

The screen brightness reduction sub-module is used for reducing the screen brightness according to the operation mode.

Further, the screen brightness reduction sub-module includes: a screen brightness reduction unit and a screen extinguishing unit, wherein:

The screen brightness reduction unit is configured to reduce the screen brightness when the operation mode is the smart off-screen mode.

A screen extinguishing unit, configured to extinguish the screen when the operating mode is a non-intelligent off-screen mode.

Further, the screen brightness reduction sub-module includes: an automatic off-screen duration acquisition unit, an intelligent off-screen mode determination unit, and a non-intelligent off-screen mode determination unit, wherein:

The automatic off-screen duration acquisition unit is configured to acquire the automatic off-screen duration set by the electronic device.

The smart screen-off mode determination unit is configured to determine that the electronic device's operating mode is the smart screen-off mode when the automatic screen-off duration is greater than or equal to a duration threshold.

The non-smart screen-off mode determination unit is configured to determine that the electronic device is in the non-smart screen-off mode when the automatic screen-off duration is less than the duration threshold.

Further, the screen brightness adjusting device 200 of the electronic device also includes: a screen brightness maintaining module, wherein:

The screen brightness maintaining module is used for maintaining the current brightness of the screen when the gaze point is located in the screen.

Further, the screen brightness adjusting device 200 of the electronic device also includes: a screen extinguishing module and a screen brightness increasing module, wherein:

A screen extinguishing module, configured to extinguish the screen when it is determined that the gaze point is not within the screen for a predetermined period of time.

A screen brightness increasing module, configured to increase the screen brightness when it is determined within the preset duration that the gaze point is switched from not being in the screen to being in the screen.

Further, the screen brightness adjustment device 200 of the electronic device also includes: a training data set acquisition module and a model training module, wherein:

The training data set acquisition module is configured to acquire a training data set, wherein the training data set includes a plurality of training images and gaze point positions corresponding to the plurality of training images.

A model training module, configured to train a preset model to obtain a neural network model based on the plurality of training images and gaze point positions corresponding to the plurality of training images.

Those skilled in the art can clearly understand that for the convenience and brevity of the description, the specific working process of the devices and modules described above can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

In several embodiments provided in the present application, the coupling between the modules may be electrical, mechanical or other forms of coupling.

In addition, each functional module in each embodiment of the present application may be integrated into one processing module, each module may exist separately physically, or two or more modules may be integrated into one module. The above-mentioned integrated modules can be implemented in the form of hardware or in the form of software function modules.

Please refer to FIG. 14 , which shows a structural block diagram of an electronic device 100 provided by an embodiment of the present application. The electronic device 100 may be an electronic device capable of running application programs, such as a smart phone, a tablet computer, and an e-book. The electronic device 100 in this application may include one or more of the following components: a processor 110, a memory 120, and one or more application programs, wherein one or more application programs may be stored in the memory 120 and configured to be executed by a or a plurality of processors 110, and one or more programs are configured to execute the methods described in the foregoing method embodiments.

Wherein, the processor 110 may include one or more processing cores. The processor 110 uses various interfaces and lines to connect various parts of the entire electronic device 100, and executes or executes instructions, programs, code sets or instruction sets stored in the memory 120, and calls data stored in the memory 120 to execute Various functions of the electronic device 100 and processing data. Optionally, the processor 110 may adopt at least one of Digital Signal Processing (Digital Signal Processing, DSP), Field-Programmable Gate Array (Field-Programmable Gate Array, FPGA), and Programmable Logic Array (Programmable Logic Array, PLA). implemented in the form of hardware. The processor 110 may integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU), a modem, and the like. Among them, the CPU mainly handles the operating system, user interface and application programs, etc.; the GPU is used to render and draw the content to be displayed; the modem is used to handle wireless communication. It can be understood that, the above-mentioned modem may not be integrated into the processor 110, but may be realized by a communication chip alone.

The memory 120 may include random access memory (Random Access Memory, RAM), and may also include read-only memory (Read-Only Memory). The memory 120 may be used to store instructions, programs, codes, sets of codes, or sets of instructions. The memory 120 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system and instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.) , instructions for implementing the following method embodiments, and the like. The storage data area can also store data created during use of the electronic device 100 (such as phonebook, audio and video data, chat record data) and the like.

Please refer to FIG. 15 , which shows a structural block diagram of a computer-readable storage medium provided by an embodiment of the present application. Program codes are stored in the computer-readable medium 300, and the program codes can be invoked by a processor to execute the methods described in the foregoing method embodiments.

The computer readable storage medium 300 may be an electronic memory such as flash memory, EEPROM (Electrically Erasable Programmable Read Only Memory), EPROM, hard disk, or ROM. Optionally, the computer-readable storage medium 300 includes a non-transitory computer-readable storage medium (non-transitory computer-readable storage medium). The computer-readable storage medium 300 has a storage space for program code 310 for executing any method steps in the above methods. These program codes can be read from or written into one or more computer program products. Program code 310 may, for example, be compressed in a suitable form.

To sum up, the method and device for adjusting the screen brightness of an electronic device and the electronic device provided by the embodiments of the present application collect face images through the camera of the electronic device, and process the face images through a neural network model to obtain the brightness of the human eyes on the screen. The gaze point position, based on the gaze point position and the distance between the gaze point position and the camera, determines whether the gaze point is located on the screen. When the gaze point is not located on the screen, lower the brightness of the screen, so as to judge the gaze point position through the neural network model And detection, and according to the gaze point position and the distance between the gaze point position and the camera, it is determined that the gaze point is not located in the screen and the screen brightness is reduced, thereby reducing the power consumption of electronic devices and improving the intelligent perception interaction experience.

Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: it can still Modifications are made to the technical solutions described in the foregoing embodiments, or equivalent replacements are made to some of the technical features; and these modifications or replacements do not drive the essence of the corresponding technical solutions away from the spirit and scope of the technical solutions of the various embodiments of the present application.

Claims

A method for adjusting screen brightness of an electronic device is characterized in that it is applied to an electronic device, and the method includes:

collecting face images through the camera of the electronic device;

Processing the face image through a neural network model to obtain the gaze point position of the human eye on the screen;

Determine whether the gaze point is located within the screen based on the gaze point position and the distance between the gaze point position and the camera;

When the gaze point is not located in the screen, reduce the brightness of the screen.
The method according to claim 1, further comprising:

When the gaze point is located within the screen, the current brightness of the screen is maintained.
The method according to claim 1, wherein the reducing the brightness of the screen comprises:

determining an operating mode of the electronic device; and

Adjust the brightness of the screen according to the mode of operation.
The method according to claim 1, further comprising: after said reducing the screen brightness,

When it is determined that the gaze point is not on the screen for a predetermined period of time, turn off the screen; or

When it is determined within the preset duration that the gaze point is switched from not being in the screen to being in the screen, increasing the brightness of the screen.
The method according to claim 3, wherein the reducing the brightness of the screen according to the operating mode comprises:

When the operating mode is the smart off-screen mode, reduce the brightness of the screen; or

When the operating mode is a non-smart off-screen mode, the screen is turned off.
The method according to claim 5, wherein the method further comprises:

Obtain the automatic off-screen duration set by the electronic device;

When the automatic off-screen duration is greater than or equal to a duration threshold, determine that the operation mode of the electronic device is the smart off-screen mode; or

When the automatic off-screen duration is less than the duration threshold, it is determined that the operation mode of the electronic device is the non-smart off-screen mode.
The method according to any one of claims 1-6, wherein the gaze point position comprises coordinate information of the gaze point in a target coordinate system, and the target coordinate system is based on the central position of the camera as Establishing the origin, determining whether the gaze point is located in the screen based on the gaze point position and the distance between the gaze point position and the camera, including:

Obtaining the distance between the gaze point position and the camera based on the coordinate information;

Based on the distance between the gaze point position and the camera, determine whether the gaze point is located within the screen.
The method according to any one of claims 1-6, wherein, based on the position of the point of gaze and the distance between the position of the point of gaze and the camera, it is determined whether the point of gaze is located within the screen ,include:

Based on the position of the gaze point and the position of the camera, obtain the relative distance and the relative direction between the position of the gaze point and the position of the camera;

Based on the relative distance and the relative direction, it is determined whether the gaze point is located within the screen.
The method according to any one of claims 1-6, wherein the processing the face image through a neural network model to obtain the gaze point position of the human eye on the screen comprises:

When there are multiple facial images collected by the camera, the facial image with the largest area among the multiple facial images is used as the target facial image;

The target human face image is processed by the neural network model to obtain the gaze point position of human eyes on the screen.
The method according to any one of claims 1-6, wherein the processing the face image through a neural network model to obtain the gaze point position of the human eye on the screen comprises:

When there are multiple human face images collected by the camera, the frontmost human face image in the multiple human face images is used as the target human face image;

The target human face image is processed by the neural network model to obtain the gaze point position of human eyes on the screen.
The method according to any one of claims 1-6, wherein the processing the face image through a neural network model to obtain the gaze point position of the human eye on the screen comprises:

When there are multiple facial images collected by the camera, the facial image matching the preset facial image among the multiple facial images is used as the target facial image;

The target human face image is processed by the neural network model to obtain the gaze point position of human eyes on the screen.
The method according to any one of claims 1-6, wherein, before processing the face image through a neural network model to obtain the gaze point position of the human eye on the screen, further comprising:

Obtain a training data set, wherein the training data set includes a plurality of training images and gaze point positions corresponding to each of the plurality of training images;

Based on the plurality of training images and gaze point positions corresponding to the plurality of training images, the preset model is trained to obtain a neural network model.
The method according to any one of claims 1-6, wherein the neural network model includes a human face detection model and a gaze point detection model, and the neural network model is used to process the human face image to obtain human eyes The gaze point position on the screen, including:

Obtain the image to be detected;

Inputting the image to be detected into the face detection model to obtain information output by the face detection model;

When the information output by the human face detection model includes a human face image, the human face image is input into the gaze point detection model to obtain the gaze point position of the human eyes output by the gaze point detection model on the screen .
The method according to claim 13, further comprising:

When the information output by the face detection model does not include a face image, reduce the brightness of the screen or control the screen to turn off.
The method according to claim 13, wherein when the information output by the human face detection model includes a human face image, inputting the human face image into the gaze point detection model to obtain the gaze point The gaze point position of human eyes on the screen output by the detection model includes:

When the information output by the face detection model includes a face image, it is judged whether the face image satisfies a preset face condition;

When the human face image satisfies the preset human face condition, the human face image is input into the gaze point detection model, and the gaze point position of the human eye on the screen output by the gaze point detection model is obtained .
The method according to claim 13, wherein when the information output by the human face detection model includes a human face image, inputting the human face image into the gaze point detection model to obtain the gaze point The gaze point position of human eyes on the screen output by the detection model includes:

When the information output by the human face detection model includes a human face image and when the human face image includes an eye image, the human face image is input into the gaze point detection model, and the person output by the gaze point detection model is obtained. The gaze point position of the eye on the screen.
The method according to claim 16, further comprising:

When the information output by the face detection model includes a face image and the face image does not include an eye image, reduce the brightness of the screen or control the screen to turn off.
A screen brightness adjustment device for electronic equipment, characterized in that it is applied to electronic equipment, and the device includes:

A face image acquisition module, configured to collect a face image through the camera of the electronic device;

The gaze point position acquisition module is used to process the face image through a neural network model to obtain the gaze point position of the human eye on the screen;

A gaze point determination module, configured to determine whether the gaze point is located within the screen based on the gaze point position and the distance between the gaze point position and the camera;

The screen brightness reduction module is used to reduce the screen brightness when the gaze point is not located in the screen.
An electronic device, characterized in that it includes a memory and a processor, the memory is coupled to the processor, the memory stores instructions, and when the instructions are executed by the processor, the processor performs The method described in any one of claims 1-17.
A computer-readable storage medium, characterized in that the computer-readable storage medium stores program codes, and the program codes can be invoked by a processor to execute the method according to any one of claims 1-17 .