WO2023130927A1

WO2023130927A1 - Always on display control method, electronic device, and storage medium

Info

Publication number: WO2023130927A1
Application number: PCT/CN2022/138948
Authority: WO
Inventors: 李经纬
Original assignee: 荣耀终端有限公司
Priority date: 2022-01-10
Filing date: 2022-12-14
Publication date: 2023-07-13
Also published as: CN116450068A

Abstract

Embodiments of the present application provide an always on display control method, an electronic device, and a computer-readable storage medium. The always on display control method comprises: if a display screen is in a screen-off state, an electronic device starts a face detection function, and acquires image data by means of a front camera; and if the electronic device recognizes that the image data comprises all or part of a facial image and determines that a difference between a time of the present recognition that the image data comprises all or part of a facial image and a time of a previous recognition that the image data comprises all or part of a facial image is not smaller than a threshold value, the display screen displays an always on display interface. It can be seen that the display screen displays an always on display interface only if the electronic device determines that a difference between a time of the present recognition that the image data comprises all or part of a facial image and a time of a previous recognition that the image data comprises all or part of a facial image is not smaller than a threshold value, so that the electronic device can be prevented from repeatedly displaying the always on display interface.

Description

Screen off display control method, electronic device and storage medium

This application claims the priority of a Chinese patent application filed with the State Intellectual Property Office of China on January 10, 2022, with the application number 202210021719.8, and the title of the invention is "control method for screen-off display, electronic equipment, and storage medium", the entire content of which Incorporated in this application by reference.

technical field

The present application relates to the field of display technology, and in particular to a control method for screen-off display, electronic equipment, and a computer-readable storage medium.

Background technique

The screen-off display is a display function when the screen of the electronic device is off. Compared with the conventional display, its power consumption is lower, and it can also meet the user's requirements for information such as date, time, and reminders. FIG. 1a provides an off-screen display interface of an electronic device, through which a user can know the date and time.

The startup method of the screen-off display provided by the electronic device is: the electronic device is in the screen-off state, the user touches the display screen, and the electronic device displays the screen-off display interface. However, if it is inconvenient for the user to touch the display screen, the electronic device cannot display the screen-off display interface.

Contents of the invention

The present application provides a control method for an off-screen display, an electronic device, and a computer-readable storage medium, with the purpose of realizing conveniently triggering an electronic device to display an off-screen display interface.

In order to achieve the above object, the application provides the following technical solutions:

In the first aspect, the present application provides a control method applicable to the screen-off display of an electronic device, including: when the display screen of the electronic device is in the screen-off state, the electronic device activates the face detection function, and obtains the face detection function through the front camera. Image data; the electronic device recognizes that the image data includes all or part of the face image; the time when the electronic device determines that the image data includes all or part of the face image this time is different from the time when the image data that was recognized last time includes all or part of the face image When the time difference is not less than the threshold, the display screen of the electronic device displays an off-screen display interface.

It can be seen from the above content that: the display screen is off, and the electronic device activates the face detection function. After the face detection function is activated, the electronic device can obtain image data through the front camera, and when it is determined that the image data includes part or all of the face image, and when it is determined that the image data includes all or part of the face image, When the time difference from the time when the image data including all or part of the facial image was recognized last time is not less than the threshold value, when the display screen of the electronic device displays an off-screen display interface, the display screen of the electronic device is turned off in this way, and the face approaches The display screen triggers an electronic device to display a screen-off display interface, which is a convenient trigger method.

And, when the electronic device determines that the difference between the time when the image data includes all or part of the face image this time and the time when the image data includes all or part of the face image is not less than the threshold, the display screen will display the off In this way, the off-screen display interface can also be prevented from being repeatedly displayed, and the electronic device enters the process of "detecting a face -> displaying the off-screen display interface -> after a certain period of time, the off-screen display interface disappears -> detecting a face" In an endless loop.

In a possible implementation, the electronic device determines that the difference between the time when the image data includes all or part of the face image this time and the time when the image data includes all or part of the face image the previous time is not less than a threshold , the display screen of the electronic device displays an off-screen display interface, including: the electronic device determines the reporting time of the face detection event, and when the difference between the reporting time of the previous face detection event and the previous face detection event is not less than the threshold value, the display screen of the electronic device displays the off-screen display interface. The screen display interface, the face detection event is generated by the electronic device recognizing that the image data includes all or part of the face image.

In a possible implementation, it also includes: when the electronic device determines that the image data includes all or part of the face image this time, and the difference between the time when the image data includes all or part of the face image in the previous time is less than a threshold , the display screen of the electronic device remains in an off-screen state.

In a possible implementation manner, before the display screen of the electronic device maintains the screen-off state, the method further includes: the electronic device determines that the electronic device remains in a static state.

In a possible implementation manner, the method further includes: the electronic device determines that the electronic device is not continuously in a static state, and the display screen of the electronic device displays an off-screen display interface.

In this possible implementation manner, the electronic device determines that the difference between the time when it recognizes that the image data includes all or part of the face image this time and the time when it recognized that the image data includes all or part of the face image this time is less than the threshold, but If the electronic device is not continuously in a static state, that is, the attitude of the electronic device changes, the display screen is triggered to display an off-screen display interface.

In a possible implementation manner, the electronic device determines that the electronic device is in a static state continuously, including: the electronic device determines that the electronic device is in the same posture within a predetermined time period, and the predetermined time period refers to: the moment when the face detection function is turned on, until this time A time period in which the difference between the time when it is recognized that the image data includes all or part of the face image and the time when the previous time when it was recognized that the image data includes all or part of the face image is not less than a threshold value.

In a possible implementation manner, the electronic device determining that the electronic device is in the same posture within the predetermined time period includes: the electronic device determines that the electronic device is in the same posture within the predetermined time period by using the three-axis detection data of the acceleration sensor.

In a possible implementation manner, before the display screen of the electronic device maintains the screen-off state, the method further includes: the electronic device determines that the light of the environment where the electronic device is located remains stable.

In a possible implementation manner, the method further includes: the electronic device determines that the light of the environment where the electronic device is located is not stable, and the display screen of the electronic device displays a screen-off display interface.

In this possible implementation manner, the electronic device determines that the difference between the time when it recognizes that the image data includes all or part of the face image this time and the time when it recognized that the image data includes all or part of the face image this time is less than the threshold, but The brightness of the environment where the electronic device is located is not stable, so that even if the user continues to be in front of the display screen, as long as the brightness of the environment where the electronic device is located changes, the display screen can be triggered to display an off-screen display interface.

In a possible implementation manner, the electronic device determines that the brightness of the environment where the electronic device is located remains stable, including: the electronic device detects the ambient light brightness through the ambient light sensor; the electronic device determines that the ambient light brightness detected by the ambient light sensor is within a predetermined period of time The predetermined time period refers to: the moment when the face detection function is turned on, the time until the current recognition of image data including all or part of the face image, and the time when the previous recognition of image data including all or part of the face image The time period for which the difference is not less than the threshold.

In a possible implementation manner, before the display screen of the electronic device maintains the off-screen state, the method further includes: the electronic device determines that the electronic device does not detect a change of an object approaching or moving away.

In a possible implementation manner, the method further includes: the electronic device determines that the electronic device detects a change in the approach and distance of the object, and the display screen of the electronic device displays an off-screen display interface.

In this possible implementation manner, the electronic device determines that the difference between the time when it recognizes that the image data includes all or part of the face image this time and the time when it recognized that the image data includes all or part of the face image this time is less than the threshold, but The electronic device detects the change of the object approaching and moving away, and the display screen of the electronic device displays the screen-off display interface, which can realize that even if the user continues to be in front of the display screen, as long as an object approaches or moves away from the electronic device, the display screen can be triggered to display the screen-off display interface.

In a possible implementation manner, the electronic device determines that the electronic device does not detect the change of the approach and distance of the object, including: the electronic device detects the object through the proximity light sensor, and obtains a report event; the electronic device determines that the report event lasts for a predetermined period of time Far away from the event, or continue to be an approaching event within a predetermined period of time, the predetermined period of time refers to: the moment when the face detection function is turned on, to the time when the image data is recognized this time including all or part of the face image, and the time when the image is recognized last time The data includes a time period in which the difference in time of all or part of the facial images is not less than a threshold.

In a possible implementation manner, the processor of the electronic device includes an intelligent sensor hub sensor hub, and the sensor hub includes: an AON module; when the display screen of the electronic device is in an off-screen state, the electronic device starts the face detection function, including: electronic When the display screen of the device is in the off-screen state, the electronic device starts the AON module to perform the face detection function.

In a possible implementation, the processor of the electronic device includes: a virtual camera unit Virtual AON Camera; based on this, when the display screen of the electronic device is in an off-screen state, the electronic device starts the face detection function, including: the Virtual AON Camera determines When the display screen is off, the face detection function is activated.

In a possible implementation, when the Virtual AON Camera determines that the display screen is off, before starting the face detection function, it further includes: the Virtual AON Camera receives the display state of the display screen, and records the display state of the display screen.

In this possible implementation, since the Virtual AON Camera takes over the on/off of the face detection function, it is necessary to know the display status of the display screen. Considering that the display screen is off, the AP will go to sleep, so before the AP goes to sleep, the Virtual AON Camera receives the screen off status of the display screen and records the screen off status of the display screen.

In a possible implementation, when the Virtual AON Camera determines that the display screen is off, after starting the face detection function, it also includes: when the Virtual AON Camera determines that the image data includes part or all of the image data, wake up the application processing of the electronic device controller, and send the display status of the display to the application processor.

In this possible implementation, since the display screen of the electronic device needs to rely on the application processor AP to be in the wake-up state for displaying the off-screen display interface, the Virtual AON Camera needs to wake up the AP first when determining that the image data includes part or all of the image data. Because the display status of the display screen is recorded by the Virtual AON Camera during the AP sleep process, after the AP is woken up, the Virtual AON Camera sends the display status of the display screen recorded by itself to the AP to ensure the normal operation of the AP.

In a possible implementation, the electronic device determines that the difference between the time when the image data includes all or part of the face image this time and the time when the image data includes all or part of the face image the previous time is not less than a threshold After the display screen of the electronic device displays the screen-off display interface, it also includes: when the electronic device judges that the display screen is in a non-screen-off state, turning off the face detection function.

In a possible implementation, before the electronic device starts the face detection function, it further includes: the electronic device determines that the electronic device meets a preset condition; wherein the preset condition includes that the display screen of the electronic device is not in a downward posture, and the electronic device The brightness of the environment is not less than the first threshold, and the electronic device does not detect that an object is approaching.

In a possible implementation manner, the electronic device determines that the display screen of the electronic device is not in a downward posture, determines that the ambient light brightness is not less than a first threshold, and further determines that a away event is received, indicating that the display screen of the electronic device is in an upward posture, In an environment with sufficient brightness and no objects approaching, the face detection function can be activated to avoid the waste of power consumption caused by the activation of the face detection function caused by the electronic device not being in an upward posture, or the environment where the brightness is not enough, or being blocked by an object.

In a second aspect, the present application provides an electronic device, including: one or more processors, memory, display screen and front camera; memory and display screen, and front camera coupled with one or more processors, memory It is used to store computer program codes, the computer program codes include computer instructions, and when one or more processors execute the computer instructions, the electronic device executes the screen-off display control method according to any one of the first aspect.

In a third aspect, the present application provides a computer storage medium for storing a computer program. When the computer program is executed, it is specifically used to implement the method for controlling the screen-off display according to any one of the first aspect.

In a fourth aspect, the present application provides a computer program product. When the computer program product is run on a computer, it enables the computer to execute the method for controlling screen-off display according to any one of the first aspect.

Description of drawings

Fig. 1a is a display diagram of an off-screen display interface;

Figure 1b is a diagram of an application scenario provided by this application;

Fig. 2a is the hardware structural diagram of the electronic device provided by the present application;

Figure 2b is a software architecture diagram of the electronic device provided by the present application;

FIG. 3 is a display diagram of an interface of an electronic device provided by the present application;

FIG. 4 is a display diagram of another interface of the electronic device provided by the present application;

FIG. 5 is a signaling diagram of a control method for screen-off display provided in Embodiment 1 of the present application;

FIG. 6 is a signaling diagram of a control method for screen-off display provided in Embodiment 2 of the present application;

FIG. 7 is a signaling diagram of a control method for screen-off display provided in Embodiment 3 of the present application;

FIG. 8 is a signaling diagram of a control method for screen-off display provided in Embodiment 4 of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. The terms used in the following examples are for the purpose of describing particular examples only, and are not intended to limit the application. As used in the specification and appended claims of this application, the singular expressions "a", "an", "said", "above", "the" and "this" are intended to also Expressions such as "one or more" are included unless the context clearly dictates otherwise. It should also be understood that in the embodiments of the present application, "one or more" refers to one, two or more than two; "and/or" describes the association relationship of associated objects, indicating that there may be three types of relationships; for example, A and/or B may mean: A exists alone, A and B exist simultaneously, and B exists alone, wherein A and B may be singular or plural. The character "/" generally indicates that the contextual objects are an "or" relationship.

Reference to "one embodiment" or "some embodiments" or the like in this specification means that a particular feature, structure, or characteristic described in connection with the embodiment is included in one or more embodiments of the present application. Thus, appearances of the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in other embodiments," etc. in various places in this specification are not necessarily All refer to the same embodiment, but mean "one or more but not all embodiments" unless specifically stated otherwise. The terms "including", "comprising", "having" and variations thereof mean "including but not limited to", unless specifically stated otherwise.

A plurality referred to in the embodiment of the present application means greater than or equal to two. It should be noted that in the description of the embodiments of the present application, words such as "first" and "second" are only used to distinguish the purpose of description, and cannot be understood as indicating or implying relative importance, nor can they be understood as indicating or imply order.

In order to clarify the technical solution of the present application, the related concepts involved in the present application are explained below.

1) AOD (Always On Display) display interface, also known as off-screen display interface, is a display interface with low power consumption. This interface generally displays some messages to remind users, such as time, date, mobile phone power, etc. . In addition, the interface can also display animation effects, which are displayed synchronously with messages prompting users.

2) Frame rate (Frames Per Second) refers to: the speed at which the camera captures images.

At present, the screen-off display is a display function when the screen of the electronic device is off. Compared with the conventional display, its power consumption is lower, and it can also meet the user's requirements for information such as date, time, and reminders. The method provided by the electronic device to start the off-screen display is: the electronic device is in the off-screen state, the user touches the display screen, and the electronic device displays the off-screen display interface.

However, if the user is inconvenient to touch the display screen, the electronic device cannot display the AOD display interface. In view of this problem, the embodiment of the present application proposes a control method for screen-off display, as shown in FIG. 1b , which can realize an AOD display interface triggered by a human face when the display screen is in an off-screen state.

The control method for screen-off display provided in the embodiment of the present application can be applied to mobile phones, tablet computers, desktops, laptops, notebook computers, ultra-mobile personal computers (Ultra-mobile Personal Computer, UMPC), handheld computers, netbooks, Personal digital assistant (Personal Digital Assistant, PDA), wearable electronic devices, smart watches and other electronic devices.

Taking a mobile phone as an example, FIG. 2a is a composition example of an electronic device provided in an embodiment of the present application. As shown in FIG. 2 a , the electronic device 100 may include a processor 110 , an internal memory 120 , a camera 130 , a display 140 , an antenna 1 , an antenna 2 , a mobile communication module 150 , a wireless communication module 160 and a sensor module 170 . The sensor module 170 may include an acceleration sensor 170A, a proximity light sensor 170B, an ambient light sensor 170C and the like.

It can be understood that the structure shown in this embodiment does not constitute a specific limitation on the electronic device 100 . In other embodiments, the electronic device 100 may include more or fewer components than shown, or combine certain components, or separate certain components, or arrange different components. The illustrated components can be realized in hardware, software or a combination of software and hardware.

The processor 110 may include one or more processing units, for example: the processor 110 may include an application processor (application processor, AP), a modem processor, a graphics processing unit (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), controller, video codec, digital signal processor (digital signal processor, DSP), baseband processor, intelligent sensor hub (sensor hub) and/or neural network processor (neural- network processing unit, NPU), etc. Wherein, different processing units may be independent devices, or may be integrated in one or more processors.

A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory may hold instructions or data that processor 110 has just used or recycled. If the processor 110 needs to use the instruction or data again, it can be called directly from the memory. Repeated access is avoided, and the waiting time of the processor 110 is reduced, thereby improving the efficiency of the system.

In some embodiments, the intelligent sensor hub (sensor hub) in the processor 110 can provide a solution based on the combination of software and hardware on the low-power MCU and the lightweight RTOS operating system, and its main function is to connect and Process data from various sensor devices.

The sensor hub can include: AON (Always on) module, signal processor and memory (On-chip SRAM). The AON module is an independent low-power module, including: AON_ISP and Face Detect (FD) module. AON_ISP can be understood as a Mini-ISP for processing the data fed back by the camera 130 . In some embodiments, AON_ISP converts the electrical signal converted by the photosensitive element of the camera into image data. The FD module is used to perform face detection on the image data output by AON_ISP to obtain face detection events.

The signal processor can be understood as a sensor chip, such as SSC (Snapdragon Sensing Core), which is used to transmit the face detection events obtained by the FD module.

On-chip SRAM is used to save the image data obtained by AON_ISP.

The internal memory 120 may be used to store computer-executable program code, which includes instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by executing instructions stored in the internal memory 120 . The internal memory 120 may include an area for storing programs and an area for storing data. Wherein, the stored program area can store an operating system, at least one application program required by a function (such as a sound playing function, an image playing function, etc.) and the like. The storage data area can store data created during the use of the electronic device 100 (such as audio data, phonebook, etc.) and the like. In addition, the internal memory 120 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, flash memory device, universal flash storage (universal flash storage, UFS) and the like. The processor 110 executes various functional applications and data processing of the electronic device 100 by executing instructions stored in the internal memory 120 and/or instructions stored in a memory provided in the processor.

The electronic device realizes the display function through the GPU, the display screen 230 , and the application processor. The GPU is a microprocessor for image processing, and is connected to the display screen 230 and the application processor. GPUs are used to perform mathematical and geometric calculations for graphics rendering. Processor 210 may include one or more GPUs that execute program instructions to generate or alter display information.

The display screen 230 is used to display images, videos and the like. The display screen 230 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active matrix organic light emitting diode or an active matrix organic light emitting diode (active-matrix organic light emitting diode, AMOLED), flexible light-emitting diode (flex light-emitting diode, FLED), Miniled, MicroLed, Micro-oled, quantum dot light emitting diodes (quantum dot light emitting diodes, QLED), etc. In some embodiments, the electronic device may include 1 or N display screens 230, where N is a positive integer greater than 1.

The electronic device 100 can realize the shooting function through the ISP, the camera 130 , the video codec, the GPU, the display screen 140 and the application processor.

The ISP is used to process data fed back by the camera 130 . For example, when taking a picture, open the shutter, the light is transmitted to the photosensitive element of the camera through the lens, and the light signal is converted into an electrical signal, and the photosensitive element of the camera transmits the electrical signal to the ISP for processing, and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on image noise, brightness, and skin color. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP may be located in the camera 130 .

Camera 130 is used to capture still images or video. The object generates an optical image through the lens and projects it to the photosensitive element. The photosensitive element may be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then transmits the electrical signal to the ISP to convert it into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. DSP converts digital image signals into standard RGB, YUV and other image signals. In some embodiments, the electronic device 100 may include 1 or N cameras 130 , where N is a positive integer greater than 1.

In some embodiments, the camera 130 can be set as a front camera of the electronic device, and is used to capture a face image located in front of the display screen of the electronic device. It should also be noted that the front camera can be understood as a camera capable of capturing images in front of the display screen during operation, and its installation location is not limited.

In some embodiments, the camera 130 can have three working modes, ambient light detection (Ambient Light Sensor, ALS) mode, ultra low power consumption (Ultra Low Power, ULP) mode and Normal mode.

The display screen is off, and the camera 130 can operate in ALS mode to generate ultra-low resolution images at configurable time intervals. The camera 130 in the ALS mode uses the image data of the ultra-low resolution image to detect the change of light and shadow, and obtain the detection result of the change of light and shadow. Wherein, the camera 130 utilizes the image data of the ultra-low resolution image to detect light and shadow changes, and is used to determine whether there is a light and shadow change in front of the camera.

There are light and shadow changes in front of the camera, and the camera can operate in ULP mode to generate a lower resolution image. Lower resolution images are used for face detection.

When the camera is called to shoot or video, the camera runs in Normal mode.

The wireless communication function of the electronic device 100 can be realized by the antenna 1 , the antenna 2 , the mobile communication module 150 , the wireless communication module 160 , a modem processor, a baseband processor, and the like.

Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 may be used to cover single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization of the antennas. For example: Antenna 1 can be multiplexed as a diversity antenna of a wireless local area network. In other embodiments, the antenna may be used in conjunction with a tuning switch.

The mobile communication module 150 can provide wireless communication solutions including 2G/3G/4G/5G applied on the electronic device 100 . The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (low noise amplifier, LNA) and the like. The mobile communication module 150 can receive electromagnetic waves through the antenna 1, filter and amplify the received electromagnetic waves, and send them to the modem processor for demodulation. The mobile communication module 150 can also amplify the signals modulated by the modem processor, and convert them into electromagnetic waves and radiate them through the antenna 1 . In some embodiments, at least part of the functional modules of the mobile communication module 150 may be set in the processor 110 . In some embodiments, at least part of the functional modules of the mobile communication module 150 and at least part of the modules of the processor 110 may be set in the same device.

The wireless communication module 160 can provide wireless local area networks (wireless local area networks, WLAN) (such as wireless fidelity (Wireless Fidelity, Wi-Fi) network), bluetooth (bluetooth, BT), global navigation satellite, etc. applied on the electronic device 100. System (global navigation satellite system, GNSS), frequency modulation (frequency modulation, FM), near field communication technology (near field communication, NFC), infrared technology (infrared, IR) and other wireless communication solutions. The wireless communication module 160 may be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2 , frequency-modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110 . The wireless communication module 160 can also receive the signal to be sent from the processor 110 , frequency-modulate it, amplify it, and convert it into electromagnetic waves through the antenna 2 for radiation.

In the sensor module 170 , the acceleration sensor 170A can detect the acceleration of the electronic device 100 in various directions (generally three axes). The magnitude and direction of gravity can be detected when the electronic device 100 is stationary. In some embodiments, the acceleration sensor 170A can also be used to identify the posture of the electronic device, and it can be applied to applications such as horizontal and vertical screen switching, pedometer, and screen-off display.

Proximity light sensor 170B may include, for example, a light emitting diode (LED), which may be an infrared light emitting diode, and a photodetector. The electronic device 100 emits infrared light through the light emitting diode. Electronic device 100 uses photodiodes to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it may be determined that there is an object near the electronic device 100 . When insufficient reflected light is detected, the electronic device 100 may determine that there is no object near the electronic device 100 . In some embodiments, the detection result of the proximity light sensor 170B can be provided to the electronic device to determine whether to activate the face detection function.

The ambient light sensor 170C is used for sensing ambient light brightness. The electronic device can adaptively adjust the brightness of the display screen 140 according to the perceived ambient light brightness. The ambient light sensor 170C can also be used to automatically adjust the white balance when taking pictures. In some embodiments, the detection result of the ambient light sensor 170C can be used to provide the electronic device to determine whether to activate the face detection function.

In some embodiments, the acceleration sensor 170A, the proximity light sensor 170B and the ambient light sensor 170C in the sensor module 170 can be in the running state after the electronic device is turned on, or can be triggered by the application program of the electronic device to start and run.

In addition, an operating system runs on top of the above components. Such as iOS operating system, Android operating system, Windows operating system, etc. An application program can be installed and run on the operating system.

Fig. 2b is a block diagram of the software structure of the electronic device according to the embodiment of the present application. The layered architecture divides the software into several layers, and each layer has a clear role and division of labor. Layers communicate through software interfaces. In some embodiments, the Android system is divided into four layers, which are respectively the application program layer, the application program framework layer, the Android runtime (Android runtime) and the system library, and the kernel layer from top to bottom.

The application layer can consist of a series of application packages. As shown in FIG. 2b, the application package may include applications such as camera, gallery, calendar, call, map, setting, AOD application, gaze detection, and face detection.

Among them, setting is mainly responsible for product isolation and product information management. Set and store attribute information of the electronic device. The setting may confirm whether the electronic device has a face detection capability according to the attribute information of the electronic device. Setting an electronic device that is determined to have face detection capabilities will expose the menu where the face detection function is enabled. The "Smart Perception" shown in Figure 3(a) and the "Screen-off display when a face is recognized" shown in Figure 3(b) are examples of menus enabled by the face detection function.

The AOD application is used to control displaying an off-screen display interface on the display screen.

Gaze detection is used to realize the detection of controlling whether the human eye is gazing at the display screen.

Face detection is used to trigger the function of detecting whether a face appears in front of the display screen.

In addition, when the face detection is turned on, the AOD application can have the function of displaying a face in front of the display screen and controlling the display screen to display an off-screen display interface. Moreover, the AOD application can also control the display screen to display an off-screen display interface when it is determined that a human face appears in front of the display screen.

In some embodiments, applications such as the AOD application can initiate a registration process to the processor of the electronic device to activate the proximity light sensor, ambient light sensor and acceleration sensor, and the proximity light sensor detects whether there is an approaching object, and the ambient light sensor Detect the brightness of the environment, and the acceleration sensor detects the acceleration of the electronic device. Proximate objects can be understood as adults or general objects, and objects can also be used instead.

The application framework layer provides an application programming interface (application programming interface, API) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions. As shown in Figure 2b, the application framework layer can include window manager, content provider, phone manager, resource manager, notification manager, view system, AON JNI Interface, etc.

A window manager is used to manage window programs. The window manager can get the size of the display screen, determine whether there is a status bar, lock the screen, capture the screen, etc.

Content providers are used to store and retrieve data and make it accessible to applications. Said data may include video, images, audio, calls made and received, browsing history and bookmarks, phonebook, etc.

The phone manager is used to provide communication functions of electronic devices. For example, the management of call status (including connected, hung up, etc.).

The resource manager provides various resources for the application, such as localized strings, icons, pictures, layout files, video files, and so on.

The notification manager enables the application to display notification information in the status bar, which can be used to convey notification-type messages, and can automatically disappear after a short stay without user interaction. For example, the notification manager is used to notify the download completion, message reminder, etc. The notification manager can also be a notification that appears on the top status bar of the system in the form of a chart or scroll bar text, such as a notification of an application running in the background, or a notification that appears on the screen in the form of a dialog window. For example, prompting text information in the status bar, issuing a prompt sound, vibrating the electronic device, and flashing the indicator light, etc.

The view system includes visual controls, such as controls for displaying text, controls for displaying pictures, and so on. The view system can be used to build applications. A display interface can consist of one or more views. For example, a display interface including a text message notification icon may include a view for displaying text and a view for displaying pictures.

AON JNI Interface is used to complete the upload and release of messages. When the AOD application has a request to enable/disable the off-screen display function, the AON JNI Interface transparently transmits the request to the AON HIDL Interface of the AON Service. When AON HIDL Interface reports a face detection event, AON JNI Interface passes the face detection event to the AOD application.

Moreover, if the electronic device has multiple intelligent sensing functions, AON JNI Interface is also used to realize the coordination and scheduling among multiple intelligent sensing functions. In one example, the electronic device is equipped with two intelligent perception functions of face detection and gaze detection. When the gaze detection is working, AON JNI Interface suspends the face detection; when the gaze detection is completed, it executes the face detection function.

In addition, if the electronic equipment relies on the upper system to do some function switch control, it is also placed in the AON JNI Interface. For example, an electronic device has a sleep detection function, and the face detection function is turned off when the user is sleeping. The trigger and exit of the sleep event are passed through the AON JNI Interface.

AON JNI Interface is also used to provide function switch control for some upper layer services. In an example, the electronic device has a sleep detection service, and the sleep detection process of the electronic device can realize the sleep detection function. When the sleep detection process of the electronic device gets a sleep event, AON JNI Interface can pass the sleep event to turn off the face detection function.

Android Runtime includes core library and virtual machine. The Android runtime is responsible for the scheduling and management of the Android system. In some embodiments of the present application, the cold start of the application will run in the Android runtime, and the Android runtime obtains the optimized file status parameters of the application, and then the Android runtime can judge whether the optimized file is outdated due to system upgrades through the optimized file status parameters , and return the judgment result to the application control module.

The core library consists of two parts: one part is the function function that the java language needs to call, and the other part is the core library of Android.

The application layer and the application framework layer run in virtual machines. The virtual machine executes the java files of the application program layer and the application program framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

A system library can include multiple function modules. For example: surface manager (surface manager), media library (Media Libraries), 3D graphics processing library (eg: OpenGL ES), 2D graphics engine (eg: SGL), etc.

The surface manager is used to manage the display subsystem and provides the fusion of 2D and 3D layers for multiple applications.

The media library supports playback and recording of various commonly used audio and video formats, as well as still image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG2, H.262, MP3, AAC, AMR, JPG, PNG, etc.

The 3D graphics processing library is used to realize 3D graphics drawing, image rendering, compositing and layer processing, etc.

The 2D graphics engine is a drawing engine for 2D drawing.

The kernel layer is the layer between hardware and software. The kernel layer includes at least display driver, camera driver, audio driver, sensor driver, AON service, virtual camera unit Virtual AON Camera and AON Driver, etc.

Wherein: the camera driver can be used to drive the camera 130 to run. In some embodiments, the camera driver controls the camera 130 to operate in the Normal mode.

Sensor drivers may be used to drive the acceleration sensor 170A, the proximity light sensor 170B, and the ambient light sensor 170C to operate.

AON service includes: AON HIDL Interface, AON HIDL Implement and AON Camera HAL. These three parts together constitute the control logic of the camera 130 . The electronic device includes multiple cameras 130, and the AON camera HAL is used to complete the scheduling of the multiple cameras 130. In addition, the switching control of the three working modes of the camera 130 is also realized by AON Camera HAL.

The Virtual AON Camera is used to take over the on/off of the face detection function after the AP sleeps. Specifically, the face detection function can be dynamically turned on/off according to the display state of the display screen, the attitude of the electronic device, the detection result of the proximity light sensor, and the brightness of the ambient light. The Virtual AON Camera is also used to receive face detection events, wake up the AP and report the face detection event; when the face detection event is not received, it does not process or wake up the AP.

AON service is used to drive the AON module to run.

It should be noted that although the embodiment of the present application uses the Android system as an example for illustration, its basic principles are also applicable to electronic devices based on operating systems such as iOS and Windows.

First of all, it needs to be explained that in the embodiment of the present application, the electronic device can be set with a face recognition model. In some embodiments, the face recognition model can be invoked by the sensor hub. Usually, the FD module in the sensor hub calls the face recognition model to perform face detection on the image data, and obtains face detection events. The face recognition model has the function of predicting whether the image data input to the face recognition model contains a human face. The face recognition model can obtain a confidence level indicating whether the image data contains a human face, and the electronic device can determine whether the image data contains a human face using the confidence level.

In some embodiments, the face recognition model can use basic network models such as convolutional neural network (Convolutional Neural Network, CNN), long-short-term memory artificial neural network (Long-Short Term Memory, LSTM).

Convolutional neural networks usually include: input layer, convolution layer (Convolution Layer), pooling layer (Pooling layer), fully connected layer (Fully Connected Layer, FC) and output layer. In general, the first layer of a convolutional neural network is the input layer, and the last layer is the output layer.

Convolution Layer refers to the neuron layer that performs convolution processing on the input signal in the convolutional neural network. In the convolutional layer of a convolutional neural network, a neuron can only be connected to some adjacent neurons. A convolutional layer usually contains several feature planes, and each feature plane can be composed of some rectangularly arranged neural units. Neural units of the same feature plane share weights, and the shared weights here are convolution kernels.

Pooling layer (Pooling layer), usually after the convolutional layer will get a feature with a large dimension, cut the feature into several regions, take its maximum value or average value, and obtain a new feature with a smaller dimension.

Fully-Connected layer, which combines all local features into global features, is used to calculate the final score of each category.

A long-short-term memory artificial neural network (Long-Short Term Memory, LSTM) usually includes an input layer, a hidden layer, and an output layer. Among them, the input layer is composed of at least one input node; when the LSTM network is a unidirectional network, the hidden layer only includes the forward hidden layer; when the LSTM network is a bidirectional network, the hidden layer includes the forward hidden layer and the backward hidden layer to the hidden layer. For each input node is connected to the forward hidden layer node and the backward hidden layer node respectively, and is used to output the input data to the forward hidden layer node and the backward hidden layer node respectively. The hidden nodes are respectively connected to the output nodes, and are used to output their own calculation results to the output nodes, and the output nodes perform calculations according to the output nodes of the hidden layer, and output data.

The face recognition model can be trained in the following ways:

Build the original face recognition model. Among them, the original model of face recognition can choose basic network models such as CNN and LSTM.

A large number of training samples are obtained, and the training samples include: image samples containing human faces and image samples not containing human faces, and the training samples are marked to indicate whether the image samples contain human faces. In some embodiments, a human face may refer to key information of a human face, and the key information of a human face may include at least one of image information indicating contours of eyes, nose, and mouth. Therefore, the image samples containing human faces may include image samples containing all human faces, and image samples containing partial human faces, such as image samples containing side faces of human faces, and image samples containing occluded human faces.

The training samples are input into the original face recognition model, and the original face recognition model detects whether the training samples contain a human face, and obtains the detection result.

Use the loss function to calculate the loss value of the detection result and the labeling result of each training sample to obtain the loss value of the model. In some embodiments, loss functions such as cross-entropy loss function and weighted loss function can be used to calculate the loss value, or a combination of multiple loss functions can be used to calculate multiple loss values.

Determine whether the loss value of the model meets the convergence condition of the model.

In some embodiments, the model convergence condition may be that the loss value of the model is less than or equal to a preset loss threshold. That is to say, the loss value of the model can be compared with the loss threshold. If the loss value of the model is greater than the loss threshold, it can be judged that the loss value of the model does not meet the model convergence conditions. Conversely, if the loss value of the model is less than or equal to the loss threshold, it can be judged that the model loss value meets the model convergence condition.

It should be noted that for multiple training samples, the loss value of the corresponding model can be calculated for each training sample. In this case, only when the model loss value of each training sample meets the model convergence condition will the Execute, on the contrary, as long as there is a model loss value of a training sample that does not meet the model convergence conditions, then execute the subsequent steps.

If the loss value of the model meets the convergence condition of the model, it means that the model training is over. The trained model can be used in the screen-off display control method proposed in the embodiment of this application to detect whether the image data input to the model contains a human face.

If the loss value of the model does not meet the convergence conditions of the model, the parameter update value of the model is calculated according to the loss value of the model, and the original face recognition model is updated with the parameter update value of the model. And use the updated model to continue to process the training samples, get the detection results, and continue to execute the subsequent process until the loss value of the model meets the convergence conditions of the model.

It should also be noted that, in the embodiment of the present application, the electronic device can provide a display interface, and the "face detection" application related to the software framework of the electronic device in FIG. 2b can be set. Figure 3 shows an example of the presentation interface. In Fig. 3(a), the display interface of the auxiliary function includes the "smart perception" application smart perception. The user can click the control of the "Smart Perception" application, and the display interface of the "Smart Perception" application can be displayed on the display screen, as shown in Figure 3(b). The "Smart Perception" application shown in Figure 3(b) includes three functional applications, namely: keep the screen on while watching the screen, reduce the volume of incoming calls while watching the screen, and turn off the screen when the face is recognized. Among them, the display of the screen being turned off when a human face is recognized can be understood as the "face detection" function proposed in the embodiment of this application. In the display interface of the "Smart Perception" application shown in Figure 3(b), the function of "recognizing a face and turning off the screen display" is activated. Correspondingly, the display interface of the screen-off display application of the electronic device will be adjusted.

Fig. 3(c) shows an example of a display interface of an off-screen display application. In the display interface of the off-screen display application shown in FIG. 3( c ), the user can click "display mode" to view the display mode of the off-screen display. When the function of "recognizing the face and turning off the screen display" is activated, the display mode of the off-screen display will be adjusted to the display mode of the smart display, as shown in Figure 3(d).

If the function of "display on screen when face is recognized" is turned off, the display mode of screen-off display will change, see the interface example shown in Figure 4. Similarly, the user controls the display screen to display the display interface of the "Smart Perception" application by clicking the control of the "Smart Perception" application shown in FIG. 4(a). On the display interface of the "Smart Perception" application shown in Figure 4(b), the user clicks the control of "Screen-off display when a face is recognized" to close this function. Correspondingly, the display mode of the screen-off display application will be adjusted to tap display. As shown in FIG. 4(c), the user clicks the “display mode” of the display interface of the screen-off display application, and as shown in FIG. 4(d), the display mode of the screen-off display application is adjusted to touch display.

Embodiment one

As shown in Figure 3(b), the electronic device’s function of “recognizing a human face and turning off the screen display” is activated, and the electronic device can realize that when the display screen is off, the face appears in front of the display screen to trigger the output of the screen off display interface . In order to realize this function, FIG. 5 shows a control method for screen-off display provided by this embodiment. The control method for screen-off display provided in this embodiment includes steps:

S501. The AOD application acquires the display status of the display screen.

Usually, the display state of the display screen includes: screen on state, screen off state, and low power consumption state (doze). The display screen of an electronic device displays the state of the AOD display interface, which can be understood as a low power consumption state. The AOD application can obtain the display status of the display screen by monitoring the events broadcast by the display driver.

In some embodiments, the processor of the electronic device includes a power management module. The power management module can be understood as a logic unit, which is used to monitor the on-off screen request sent by the application program or triggered by the user, and use the on-off screen logic to control the display driver to realize the control of the working state of the display screen.

Taking the screen off as an example, the screen off trigger methods include: 1. The user manually presses the power button to turn off the screen; 2. The system's automatic screen off management, that is, the electronic device automatically turns off the screen if the user does not input an operation for a period of time; 3. The system actively Initiated off-screen, such as blocking off the screen when making a call, off-screen issued by the system when it is automatically restarted, etc.

The power management module judges that any one of the above-mentioned off-screen triggering methods is executed, and then sends an off-screen command to the display driver. The display driver powers off the display after receiving the screen-off command issued by the power management module. At the same time, the screen-off event is broadcast, and the AOD application can monitor the screen-off event to determine that the display state of the display screen is the screen-off state.

Similarly, the AOD application can obtain the bright screen state and low power consumption state in the same way.

S502. The AOD application sends the display status of the display screen to the AON JNI Interface.

As mentioned above in the electronic device software framework, AON JNI Interface belongs to the application framework layer and is used to complete the upload and release of messages. passed down.

S503. The AON JNI Interface receives the display status of the display screen, and sends the display status of the display screen to the AON service.

As mentioned above in the electronic equipment software framework, AON service belongs to the kernel layer, including AON HIDL Interface, AON HIDL Implement and AON Camera HAL. AON JNI Interface sending the display status of the display to AON service includes: AON JNI Interface sending the display status of the display to AON HIDL Interface.

AON HIDL Interface receives the display status of the display and sends it to AON HIDL Implement. AON HIDL Implement receives the display status of the display and sends it to AON Camera HAL.

S504. The AON service receives the display status of the display screen, and sends the display status of the display screen to the Virtual AON Camera.

AON Camera HAL receives the display status of the display screen, and AON Camera HAL sends the display status of the display screen to Virtual AON Camera.

It should be noted that the operation of AON Service depends on the AP being in the wake-up state. If the AP is dormant, AON Service cannot control the enabling/disabling of the face detection function. Based on this, the Sensor hub includes the logic unit Virtual AON Camera, which is used to take over the opening/closing of the face detection function when the AP is sleeping.

Because the Virtual AON Camera takes over the opening/closing of the face detection function, it needs to know the display status of the display screen, so the AON service receives the display status of the display screen. If the status of the display screen is off, considering that the screen is off, the AP It will go to sleep, so before the AP goes to sleep, the AON service sends the display status of the display to the Virtual AON Camera.

S505. The display screen is in an off state, and the AP enters a dormant state.

Among them, the display screen is off, and the AP sleeps. In some embodiments, the AP may determine to turn off the display screen by learning the off-screen instruction issued by the power management module.

S506, Virtual AON Camera records the display status of the display screen.

Before the AP sleeps, the Virtual AON Camera will record the display status of the display screen, and after the AP wakes up, the Virtual AON Camera can send the display status of the display screen to the AP.

It should also be noted that when the display screen is off, the AOD application can initiate a registration request to request to start the face detection function and to request to receive face detection events. When the display screen is not off, the AOD application can initiate a deregistration request to request deregistration, that is, turn off the face detection function and not receive face detection events.

Based on this, in the foregoing step S501 to step S503, what the AOD application sends to the lower layer is an AOD registration message. What Virtual AON Camera records is the registration message of AOD.

S507. The Virtual AON Camera sends an instruction to start the face detection function to the AON Driver.

When the display screen is off, the electronic device can start the face detection function, so that after the face is detected, the electronic device displays the AOD display interface. Among them, the Virtual AON Camera uses the display status of the display screen to determine that the display screen of the electronic device is off, and sends an instruction to the AON Driver to start the face detection function.

AON Driver, like the content of the software framework of the aforementioned electronic equipment, belongs to the kernel layer and is used to drive the operation of the AON module.

S508. The AON Driver sends an instruction to start the face detection function to the AON module.

After receiving the instruction to start the face detection function, the AON Driver can send instructions to the AON module to control the AON module to start the face detection function.

As mentioned above in the hardware structure of the electronic device, the AON module includes an AON_ISP and an FD module, and the FD module is used for face detection on the image data output by the AON_ISP. The AON Driver sends an instruction to the AON module to start the face detection function, which can refer to controlling the AON_ISP to output image data, and controlling the operation of the FD module for face detection.

It should be noted that, as shown in Figure 3(b), after the user manually activates the function of "recognizing a face and turning off the screen display" of the electronic device, the electronic device will not immediately determine whether the face appears in front of the display screen, so as to Trigger the detection of the output of the screen-off display interface. When the electronic device determines that the display screen of the electronic device is in the off-screen state by performing steps S501 to S507, it executes step S508 to activate the face detection function to detect that a face appears in front of the display screen and trigger the display of the screen-off display interface.

It can be seen from this that: for the function of triggering the display of the screen-off display interface when the face appears in front of the display screen, the electronic device is equipped with two activation switches, and one activation switch is provided for the user to manually trigger, as shown in 3(b) The function switch of "turning off the screen display when the face is recognized"; the other start switch belongs to the automatic start switch triggered by the condition, such as the start command of the display screen of the electronic device mentioned in step S508 and the face detection function. After the two start switches configured on the electronic device are both activated, the electronic device can detect that a human face appears in front of the display screen, and trigger the display of an off-screen display interface.

S509. The camera captures image data.

In some embodiments, the display screen enters the off-screen state, and the AON Camera HAL drives the camera to run.

As in the hardware structure of the aforementioned electronic equipment, the camera includes three working modes. The camera operates in ALS mode to obtain the detection result of light and shadow changes. AON Camera HAL uses the detection results of light and shadow changes to determine whether there is light and shadow change in front of the camera. AON Camera HAL switches The camera operates in ULP mode. The camera operates in ULP mode, capturing images at a set frame rate, such as 5fps. After the camera captures a certain number of images, such as 3 frames of images, it can exit ULP mode, and AON Camera HAL switches the camera to run in ALS mode.

In some embodiments, the AON Camera HAL is configured with control logic to switch the camera to operate in ALS mode or ULP mode. AON Camera HAL can control the camera to operate in ALS mode or ULP mode according to the control logic.

It should be noted that what is shown in FIG. 5 is an example of the execution order of the image data captured by the camera, and does not constitute a limitation on the execution order of step S509. In some embodiments, the camera can be activated by the camera driver to capture image data after the display screen enters the off-screen state.

S510. The AON module acquires image data captured by the camera.

The image data obtained by taking images with the camera is in the form of electrical signals. AON_ISP obtains the electrical signal, converts it into image data and provides it to the FD module.

S511. The AON module detects whether the image data includes a human face.

Wherein, the FD module detects whether the image data includes a human face. In some embodiments, the FD module invokes the face recognition model proposed above to process the image data to obtain a confidence level indicating whether the image data includes a face.

It should be noted that the detection by the FD module of whether the image data includes a human face can be understood as: the FD module detects whether the image data includes all or part of the human face, such as whether it contains all or part of the key information of the human face.

If the AON module detects that the image data includes a human face, execute S512 to generate a human face detection event. If the AON module detects that the image data does not include a face, the display screen remains off.

Among them, if the FD module determines that the confidence is greater than the threshold, it means that the image data includes a face, and a face detection event is generated. The face detection event is used to indicate that the image captured by the camera includes a face, indicating that there is someone in front of the display screen of the electronic device. Face. In some embodiments, the face detection event can be a high or low level.

It should also be noted that the camera captures multiple frames of images, and the FD module calls the face recognition model to process the image data of each frame of images to obtain the confidence of each frame of images. When the confidence of a frame image is greater than the threshold, a face detection event is generated. Of course, the FD module can also generate a face detection event after determining that the confidence of each frame of image is greater than the threshold.

The camera shoots multiple frames of images, and the FD module determines that the confidence of each frame of the image is greater than the threshold, and generates a face detection event. It can be seen that: the FD module determines that the image data within a certain period of time includes all or part of the face, and generates Face detection event.

It should also be noted that the image data detected by the AON module includes a face, and before the AON module generates a face detection event, it may further include: the image data detected by the AON module includes a set feature, and the set feature is a pre-configured image feature. The electronic device can take pictures of the legal user of the electronic device, such as the owner of the electronic device, through the camera, and extract key features of the image, such as at least one of the image information of the contours of the eyes, nose, and mouth. The key features of the extracted image can be To set the characteristics.

Based on this, the AON module detects that the image data includes a human face, and then recognizes whether the image data includes pre-configured image features. If the AON module recognizes that the image data includes pre-configured image features, a face detection event is generated. In this way, it can be ensured that the electronic device can be triggered to display the off-screen display interface through the display screen only when the legitimate user of the electronic device appears in front of the display screen of the electronic device.

S513. The AON module sends the face detection event to the signal processor.

Among them, the FD module sends a face detection event to the signal processor.

S514. The signal processor sends a face detection event to the AON Driver.

Among them, the signal processor receives the face detection event and sends the face detection event to the AON Driver.

S515. The AON Driver sends a face detection event to the Virtual AON Camera.

The FD module gets the face detection event, indicating that a face appears in front of the display screen of the electronic device, which can trigger the display screen to output the AOD display interface. Therefore, AON Driver receives the face detection event and sends the face detection event to the Virtual AON Camera.

S516. The Virtual AON Camera wakes up the AP, and sends the recorded display status of the display screen to the AP.

Because the operation of the AON service depends on the AP being in the wake-up state, the Virtual AON Camera needs to wake up the AP before uploading face detection events.

Because the display status of the display screen is recorded by the Virtual AON Camera during the AP sleep process, therefore, after the AP is woken up, the Virtual AON Camera sends the display status of the display screen recorded by itself to the AP.

S517. The Virtual AON Camera sends a face detection event to the AON service.

When the AP is awakened, the Virtual AON Camera and AON service can communicate, and the Virtual AON Camera can send face detection events to the AON service.

In some embodiments, as described above in the software framework of the electronic device, the AON service includes AON HIDL Interface, AON HIDL Implement and AON Camera HAL. Virtual AON Camera sends face detection events to AON Camera HAL, AON Camera HAL sends face detection events to AON HIDL Implement, and AON HIDL Implement sends face detection events to AON HIDL Interface.

S518. The AON service sends a face detection event to the AON JNI Interface.

Among them, AON HIDL Interface receives the face detection event and sends the face detection event to AON JNI Interface.

S519. The AON JNI Interface sends a face detection event to the AOD application.

Among them, AON JNI Interface receives the face detection event and sends the face detection event to the AOD application.

S520. The AOD application control display screen displays an AOD display interface.

Among them, the AOD application receives a face detection event, determines that a face appears in front of the display screen of the electronic device, and then controls the display screen to display the AOD display interface, so that the user can trigger the display screen of the electronic device by placing the face in front of the display screen Display the AOD display interface.

In some embodiments, after the AOD application controls the display screen to display the AOD display interface for a certain period of time, such as 5 seconds, the display screen may exit from displaying the AOD display interface and enter the off-screen state.

Embodiment two

The control method for screen-off display provided in Embodiment 1 can realize the function of displaying an AOD display interface on the display screen when a human face appears in front of the display screen when the display screen of the electronic device is off. When this function is implemented, the face detection function of the electronic device is turned on. When the electronic device does not need to enable the face detection function, the face detection function of the electronic device can be turned off. The screen-off display control method provided in this embodiment can realize the closing of the face detection function.

Figure 6 shows a control method for screen-off display provided by the embodiment of the present application. The control method for screen-off display provided in this embodiment includes steps:

S601. The AOD application acquires the display state of the display screen.

S602. The AOD application sends the display state of the display screen to the AON JNI Interface.

S603. The AON JNI Interface receives the display status of the display screen, and sends the display status of the display screen to the AON service.

S604. The AON service receives the display status of the display screen, and sends the display status of the display screen to the AP.

Wherein, for the specific implementation manners of steps S601 to S604, please refer to the contents of steps S501 to S504 in Embodiment 1, which will not be repeated here.

S605. The AP determines that the display screen is in a bright screen state or a doze state.

Wherein, the AP can determine whether the display screen is in a bright screen state or a doze state according to the received display state.

S606. The Virtual AON Camera sends instruction information to the AP to instruct the face detection function of the electronic device to start.

When the display screen is in the bright screen or doze state, if the face detection function of the electronic device is activated, it will affect the normal use of the electronic device. Therefore, the face detection function should be turned off. In order to disable the face detection function, the AP needs to know whether the face detection function of the electronic device is enabled.

In some embodiments, the Virtual AON Camera can detect whether the face detection function of the electronic device is activated. When the Virtual AON Camera determines that the face detection function of the electronic device is activated, it sends an instruction message to the AP, which is used to represent the activation of the face detection function of the electronic device.

In some other embodiments, the AP can also send an instruction to the Virtual AON Camera, so that after the Virtual AON Camera receives the instruction, it determines whether the face detection function of the electronic device is activated, and sends an instruction message to the AP.

It should also be noted that FIG. 6 shows an example of the execution order of step S606, but step S606 is not limited to this execution order, and the Virtual AON Camera may execute step S606 at any time before step S605.

S607. The AP sends an instruction to disable the face detection function to the AON Driver.

The AP determines that the display screen is in the bright screen state or doze state, and receives the instruction information sent by the Virtual AON Camera to determine that the face detection function of the electronic device is activated, and then sends an instruction to turn off the face detection function to the AON Driver to indicate the shutdown Face detection function.

It should also be noted that both the on-screen state and the doze state belong to the non-off-screen state of the display screen, but the non-off-screen state of the display screen is not limited to the on-screen state and the doze state. When the display screen is not off, the AP receives an instruction message from the Virtual AON Camera to instruct the face detection function of the electronic device to start, and the AP sends an instruction to turn off the face detection function to the AON Driver to instruct to turn off the face detection function. Face detection function.

S608. The AON Driver sends an instruction to disable the face detection function to the AON module.

Among them, the AON Driver receives an instruction from the AON Driver to turn off the face detection function, and sends an instruction to turn off the face detection function to the AON module.

The AO module receives this command and turns off the face detection function. In some embodiments, the AON Driver sends an instruction to the AON module to turn off the face detection function, which may refer to controlling the FD module to turn off the face detection.

If the FD module turns off face detection, it will not perform face detection on the image data captured by the camera, and will not report face detection events. Based on this, the AOD application will not be triggered by face detection events and will not be controlled. The display screen displays the AOD display interface, so that the AOD display interface is not displayed when the display screen of the electronic device is in a non-screen off state.

Embodiment three

In step S507 of the control method for screen-off display provided in Embodiment 1, the Virtual AON Camera sends an instruction to start the face detection function to the AON Driver, which may be when the Virtual AON Camera determines that the electronic device meets certain conditions, and then sends the start face detection function to the AON Driver. Indication of the face detection function. The method for controlling the screen-off display provided in this embodiment will be introduced here.

In this embodiment, the Virtual AON Camera may include: a display state processing module, an acceleration monitoring module, an approach light monitoring module, an ambient light monitoring module, an attitude detection module, a face detection startup condition judgment module, a face The detection stop condition judgment module, the face event reporting module, and the abnormal management module are several logic modules.

Among them: the display status processing module, which is responsible for receiving the display status instructions issued by the upper layer, parsing the messages and recording the current display status. And according to the display status and the start and stop of the face detection module, the sensor information of the electronic device is integrated to make a decision to start or stop the face detection function.

The acceleration monitoring module is responsible for receiving the acceleration change data sent by the acceleration sensor, and forwarding the acceleration change data to the attitude monitoring module.

The proximity light monitoring module is responsible for receiving the proximity light change event sent by the proximity light sensor, and recording the proximity light (approaching or departing) event.

The ambient light monitoring module is used for receiving the ambient light brightness sent by the ambient light sensor and recording the ambient light brightness.

The attitude detection module is responsible for receiving the data transmitted by the acceleration monitoring module, identifying and recording the attitude of the electronic device.

The face detection start condition judging module is used to start the face detection function according to the face detection start condition.

The face detection stop condition judging module is used to stop the face detection function according to the face detection stop condition.

The face event reporting module is responsible for receiving face detection events. If a face is detected, the AP will wake up and report the face detection event; if a non-face is detected, it will not be processed and continue to return to the face detection mode .

The abnormal management module is responsible for identifying process and functional abnormalities, recording and reporting abnormalities.

Figure 7 shows a control method for screen-off display provided by the embodiment of the present application. The control method for screen-off display provided in this embodiment includes steps:

S701. The proximity light sensor detects the proximity of an approaching object, and obtains an approach event or a distance event.

Wherein, the proximity light sensor detects that an approaching object is approaching, and generates an approach event; the proximity light sensor does not detect that an approaching object is approaching, and generates a moving away event. In some embodiments, the proximity event and the distance event can be high and low levels. In one example, the high level refers to the proximity event, and the low level refers to the distance event.

S702. The ambient light sensor detects and obtains ambient light brightness.

Wherein, the ambient light sensor is used to detect the ambient light of the environment where the electronic device is located to obtain the brightness of the ambient light.

S703. The acceleration sensor detects and obtains three-axis detection data.

Wherein, the acceleration sensor detects the acceleration of the electronic device in the three axes of x, y and z, and obtains the detection data of the three axes of x, y and z.

S704. The proximity light sensor sends a proximity event or a distance event to the Virtual AON Camera.

Wherein, the proximity light monitoring module receives the proximity event or the distance event sent by the proximity light sensor.

S705. The ambient light sensor sends ambient light brightness to the Virtual AON Camera.

Wherein, the ambient light monitoring module receives the ambient light brightness sent by the ambient light sensor.

S706. The acceleration sensor sends three-axis detection data to the Virtual AON Camera.

Wherein, the acceleration monitoring module receives the three-axis detection data sent by the acceleration sensor.

It should also be noted that what is shown in FIG. 7 is an example of the execution sequence of steps S701 to S706, but the steps S701 to S706 are not limited to this execution sequence. In some embodiments, the Virtual AON Camera can execute step S704, step S705, step S706 and step S707 in parallel.

S707, Virtual AON Camera receives and records the display status of the display screen.

Among them, the Virtual AON Camera receives the display status of the display screen sent by the AON service. Wherein, the display status of the display screen sent by the AON service can be as the content of step S504 in the first embodiment, and will not be repeated here.

Specifically, the display state processing module receives and records the display state of the display screen.

It should also be noted that step S707 can be understood as step S506 in the first embodiment. Of course, before step S707 is executed, the electronic device needs to execute steps S501 to S505 of the first embodiment, which will not be described in detail in this embodiment, and can be referred to the content of the first embodiment.

S708, Virtual AON Camera judges whether the display state is off screen state.

Wherein, the display state processing module utilizes the display state of the display screen to determine whether the display screen is in an off state.

If the Virtual AON Camera judges that the display state is off-screen state, then execute step S709 to step S711. When the Virtual AON Camera judges that the display state is not the off-screen state, step S712 and step S713 are executed.

S709, Virtual AON Camera judges whether the face detection function is activated.

Wherein, the display state processing module judges whether the face detection function is activated.

It should be noted that what is shown in FIG. 7 is an example of the execution sequence of step S707 , step S708 and step S709 . Step S707, step S708, and step S709 are not limited to the order of execution. In some embodiments, the Virtual AON Camera may execute step S707 and step S709 in parallel, or execute step S709 first, and then execute step S707 and step S708.

Moreover, step S709 may also be an optional step. In some embodiments, the Virtual AON Camera determines that the display state is the off-screen state, and may directly execute step S710.

S710 and Virtual AON Camera use the three-axis detection data to judge whether the electronic device is not in the first posture, judge whether the ambient light brightness is not less than the first threshold, and judge whether to receive a far away event.

Wherein, the first posture can be understood as a downward posture of the display screen of the electronic device, including a posture in which the display screen is completely downward and a posture in which the display screen is tilted downward at a certain angle. Virtual AON Camera can process three-axis detection data to obtain the attitude of electronic equipment. In some embodiments, the Virtual AON Camera can use the following formulas 1 to 3 to process the three-axis detection data to obtain the pitch angle g_pitch. Virtual AON Camera judges whether the calculated pitch angle meets the threshold range, such as [110°-350°]. If the calculated pitch angle is within the threshold range, it means that the electronic device is in the first attitude. If the calculated pitch angle is not within the threshold range, it means that the electronic device is not in the first attitude.

Formula 1

acc_data[PS_DATA_X]=ps_gesture_acc_data.x×SENSOR_DATA_FACTOR

Formula 2

acc_data[PS_DATA_Y]=ps_gesture_acc_data.y×SENSOR_DATA_FACTOR

Formula 3

acc_data[PS_DATA_Z]=ps_gesture_acc_data.z×SENSOR_DATA_FACTOR

In Formula 1, ps_gesture_acc_data.x is the detection data of the x-axis of the acceleration sensor.

In Formula 2, ps_gesture_acc_data.y is the detection data of the y-axis of the acceleration sensor.

In Formula 3, ps_gesture_acc_data.z is the detection data of the z-axis of the acceleration sensor.

In Formula 1 to Formula 3, SENSOR_DATA_FACTOR=100.

In Formula 4, FLAT_ANGLE=180; PEI=3.1415926.

The first threshold is the evaluation value of whether the ambient light brightness of the environment where the electronic device is located is sufficient to support the camera to capture a clear image. If the ambient light brightness is not less than the first threshold, it means that the ambient light brightness of the electronic device is sufficient to support the camera to capture Clear image. In one example, the first threshold may be 6lux.

In some embodiments, the Virtual AON Camera can determine whether an ambient light brightness reported by the ambient light sensor is not less than the first threshold.

In some other embodiments, the ambient light sensor can continuously detect the ambient light brightness, obtain multiple ambient light brightness, and report each ambient light brightness to the Virtual AON Camera. Virtual AON Camera continues to judge whether the brightness of each ambient light is not less than the first threshold. In an example, the Virtual AON Camera judges whether five consecutive ambient light brightnesses are not less than the first threshold.

It should be noted that, in this step, the face detection activation condition judgment module in the Virtual AON Camera can use the three-axis detection data to judge whether the electronic device is in the first posture, judge whether the ambient light brightness is not less than the first threshold, and Steps to determine whether to receive far away events.

Virtual AON Camera uses three-axis detection data to judge that the electronic device is not in the first posture, judges that the ambient light brightness is not less than the first threshold, and judges that it receives a far away event, then executes S711, and Virtual AON Camera starts the face detection function.

In some embodiments, the Virtual AON Camera uses the three-axis detection data to judge that the pitch angle of the electronic device is not within the threshold range, such as [110°-350°]; it judges that the brightness of multiple consecutive environments (such as 5) is not less than the first Threshold, such as 6lux; if it judges that a far away event is received, the face detection function will be activated.

Specifically, the face detection starting condition judging module starts the face detection function. In some embodiments, the Virtual AON Camera sends an instruction to start the face detection function to the AON Driver.

It should be noted that step S711 can be understood as step S507 in the first embodiment. Of course, after step S711 is executed, the electronic device needs to execute steps S508 to S520 of the first embodiment, which will not be described in detail in this embodiment, and can be referred to the content of the first embodiment.

After the AON Driver receives the instruction to start the face detection function, it can execute step S508 provided in the first embodiment. In this way, the face detection function of the electronic device is activated, and the AON module can obtain the image data captured by the camera, detect whether the image data includes a face, and generate a face detection event according to the face detection result.

As described in Embodiment 1, the Virtual AON Camera also receives the face detection event sent by the AON Driver, and executes steps S516 and S517. Wherein, step S516 and step S517 may be performed by the face event reporting module.

It should be noted that the Virtual AON Camera uses the three-axis detection data of the acceleration sensor to determine that the electronic device is not in the first posture, and that the ambient light brightness is not less than the first threshold. Upward posture, the environment with sufficient brightness, and no close objects approaching, the face detection function can be activated.

It should also be noted that if the Virtual AON Camera judges that at least one of the following three conditions is met, the face detection function will not be activated. The three conditions include: condition 1, using the three-axis detection data to determine that the electronic device is in the first attitude; condition 2, determining that the ambient light brightness is less than the first threshold, and condition 3, determining that an approach event is received.

If condition 1 is satisfied, it means that the electronic device is in a posture with the display screen facing downwards, which is a posture in which the user cannot see the display screen, such as the display screen buckled down posture. In this posture, the electronic device activates the face detection function, and the display screen displays the AOD display interface, which cannot be seen by the user. Therefore, the face detection function may not be activated to avoid wasting power consumption.

If condition 2 is satisfied, it means that the brightness of the environment where the electronic device is located is too low, the electronic device activates the face detection function, and the image data captured by the camera is not enough to identify whether the face is included, so the face detection function can also be disabled to avoid Wasted power.

If condition 3 is satisfied, it means that there is an object close to the electronic device. For example, in a scenario where the electronic device is placed in a bag, the face detection function may not be enabled to avoid wasting power consumption.

It should be noted that in condition 2, the ambient light brightness is less than the first threshold may refer to a plurality of consecutive ambient light brightnesses including at least one ambient light brightness being smaller than the first threshold. In one example, five consecutive ambient light brightnesses are less than 6lux. Of course, condition 2 may also be judging that the ambient light brightness is smaller than other values smaller than the first threshold, such as 3lux.

S712, Virtual AON Camera judges whether the face detection function is activated.

It should be noted that, the execution sequence of step S712 and step S708 is not limited to that shown in FIG. 7 . In some embodiments, step S712 may be performed first, and then step S708 may be performed.

The Virtual AON Camera judges that the face detection function is started, and then executes S713, and the Virtual AON Camera sends an instruction message to the AP to instruct the face detection function of the electronic device to start.

Among them, the Virtual AON Camera judges that the face detection function is activated, and the display screen is not in the off-screen state, and then sends an instruction message to the AP to instruct the face detection function of the electronic device to start. It should be noted that step S713 can be understood as step S606 in the second embodiment. Wherein, after the AP receives the instruction information, it can execute step S607 provided in the second embodiment, and the AON Driver executes the step S608 provided in the second embodiment.

It should also be noted that the face detection stop condition judging module of the Virtual AON Camera can execute step S713.

It should also be noted that when the face detection function of the electronic device is activated, the Virtual AON Camera can also use the detection data of the acceleration sensor, the proximity light sensor and the ambient light sensor to judge whether to turn off the face detection function.

Among them, the Virtual AON Camera can use the three-axis detection data of the acceleration sensor to judge whether the electronic device is in the first attitude, use the ambient light sensor to detect whether the ambient light brightness is less than the first threshold, or use the method to judge whether it receives the approach reported by the proximity light sensor. event.

The Virtual AON Camera uses the three-axis detection data to determine that the electronic device is in the first posture, the ambient light brightness is less than the first threshold, or receives an approach event, and then executes steps S712 and S713.

In some embodiments, the Virtual AON Camera uses the three-axis detection data to calculate the pitch angle according to the formula provided in the aforementioned step S710. If it is judged that the calculated pitch angle is within the threshold range, such as [110°-350°], it is determined that the electronic device is in the first attitude.

The ambient light brightness is less than the first threshold, which may mean that one ambient light brightness reported by the ambient light sensor is smaller than the first threshold, or may refer to multiple ambient light brightness reported by the ambient light sensor, all of which are smaller than the first threshold. The first threshold may be the same as or different from the first threshold in step S710. In one example, five consecutive ambient light brightnesses are less than 3 lux.

When the face detection function of the electronic device is activated, the electronic device is in the first posture, which means that the display screen of the electronic device is facing downwards, which is a posture in which the user cannot see the display screen. In this posture, the electronic device activates the face detection function, and the display screen displays the AOD display interface, which cannot be seen by the user. Therefore, the face detection function can be turned off to avoid wasting power consumption.

If the brightness of the ambient light is less than the first threshold, it means that the brightness of the environment where the electronic device is located is too low, the electronic device starts the face detection function, and the image data captured by the camera is not enough to identify whether the face is included, so the face detection function can be turned off , also avoiding waste of power consumption.

When the Virtual AON Camera receives an approach event, it means that something close to the electronic device is close to it. For example, when the electronic device is placed in a bag, the face detection function can also be turned off to avoid wasting power consumption.

Embodiment four

The electronic device can use the screen-off display control method provided in Embodiment 1 to realize face detection when the display screen is in the off-screen state, and after the face is detected, the display screen displays an AOD display interface. However, during the execution of the method for controlling the screen-off display provided in Embodiment 1, there are the following application scenarios.

In this application scenario, the user is always in front of the display screen of the electronic device. The electronic device continues to determine that the face detection conditions are met, causing the electronic device to be in an endless loop of "face detected -> AOD displayed -> AOD disappears after a certain period of time -> face detected". Based on this, the present embodiment provides a control method for turning off the screen display, so as to prevent the electronic device from falling into the endless loop.

In this embodiment, the Virtual AON Camera may include a face continuous detection escape module. The face continuous detection escape module is used to: determine that the difference between the reporting time of this face detection event and the reporting time of the last face detection event is less than the second threshold, and not report the face detection event. The second threshold is 30 seconds, 1 minute. It should be noted that the second threshold is generally greater than the display duration of the AOD display interface.

Figure 8 shows a control method for screen-off display provided by the embodiment of the present application. The control method for screen-off display provided in this embodiment includes steps:

S801. The proximity light sensor detects the proximity of an approaching object, and obtains an approach event or a distance event.

S802. The ambient light sensor detects and obtains ambient light brightness.

S803. The acceleration sensor detects and obtains three-axis detection data.

S804. The proximity light sensor sends a proximity event or a distance event to the Virtual AON Camera.

S805. The ambient light sensor sends ambient light brightness to the Virtual AON Camera.

S806. The acceleration sensor sends three-axis detection data to the Virtual AON Camera.

It should be noted that the specific implementation process of step S801 to step S806 may be the same as the content of step S701 to step S706 provided in the third embodiment, which will not be repeated here.

It should also be noted that what is shown in FIG. 8 is an example of the execution sequence of steps S801 to S806, but the execution sequence of steps S801 to S806 is not limited to this execution sequence. In some embodiments, the Virtual AON Camera can execute step S804, step S805, step S806 and step S807 in parallel.

S807. The Virtual AON Camera receives the face detection event sent by the AON Driver.

Step S807 can be understood as step S515 in the first embodiment. Wherein, as in the content of step S501 to step S514 of the first embodiment, after the face detection function of the electronic device is started, the AON module generates a face detection event when the detection image data includes a face, and passes the signal processor, AON Driver, upload face detection events to Virtual AON Camera.

In some embodiments, after the Virtual AON Camera receives the face detection event sent by the AON Driver, it can record the reporting time of the face detection event.

S808. The Virtual AON Camera judges whether the difference between the reporting time of the face detection event and the reporting time of the last face detection event is smaller than the second threshold.

Wherein, the Virtual AON Camera determines that the difference between the reporting time of the face detection event and the reporting time of the last face detection event is not less than the second threshold, and executes steps S809 and S810. The Virtual AON Camera judges that the difference between the reporting time of the face detection event and the reporting time of the last face detection event is less than the second threshold, and executes steps S811 and S820.

Specifically, the face continuous detection escape module judges whether the difference between the reporting time of the face detection event and the reporting time of the last face detection event is smaller than the second threshold.

It should also be noted that, as in step S512 in the first embodiment, if the AON module detects that the image data includes a face, it can generate a face detection event and report the face detection event to the signal processor. In some embodiments, the AON module can also obtain the reporting time of the face detection event, and report it to the signal processor synchronously, and send it to the Virtual AON Camera through the AON Driver, so that the Virtual AON Camera can obtain the reporting time of the face detection event .

In some other embodiments, when the Virtual AON Camera receives the face detection event sent by the AON Driver, the receiving time is used as the reporting time of the face detection event.

S809. The Virtual AON Camera wakes up the AP and sends the recorded display status to the AP.

S810, Virtual AON Camera sends face detection event to AON service.

Wherein, the specific implementation process of step S809 and step S810 may be the same as the content of step S516 and step S517 in the first embodiment, which will not be repeated here. Moreover, after step S809 and step S810 are executed, the electronic device may continue to execute step S518 to step S520 as proposed in the first embodiment, which will not be described here, and may refer to the content of the first embodiment.

It should be noted that the Virtual AON Camera receives the face detection event, and judges that the difference between the reporting time of the face detection event and the reporting time of the last face detection event is not less than the second threshold, and then executes steps S809 and S810 , AON service can receive face detection events. The AON service can also send the face detection event to the upper layer, so that the AOD application can receive the face detection event and control the display to display the AOD display interface. In this way, the electronic device can be prevented from entering into an infinite loop of "face detected -> display AOD -> AOD disappears after a certain period of time -> face detected".

S811, Virtual AON Camera uses three-axis detection data to judge whether the electronic equipment is still in a static state.

The electronic device remains in a static state, which can be understood as: from the moment when the face detection function of the electronic device is turned on, until the Virtual AON Camera judges the reporting time of the face detection event and the reporting time of the last face detection event through step S808 During the period when the difference is smaller than the second threshold, whether the electronic device is in the same attitude.

In some embodiments, the pitch angle of the electronic device can be calculated through the three-axis detection data, and it can be judged whether the calculated pitch angle maintains a stable value. Maintaining a stable value can be understood as: from the moment when the face detection function of the electronic device is turned on, until the Virtual AON Camera determines that the difference between the reporting time of the face detection event and the reporting time of the last face detection event is less than the first time through step S808 During the period of the second threshold, the acceleration sensor continuously reports the three-axis detection data, and the Virtual AON Camera uses the three-axis detection data reported each time to calculate the difference of the pitch angle within a certain range, such as within a range of 5° .

Among them, the Virtual AON Camera uses the three-axis detection data to determine that the electronic device is not in a static state, and then executes steps S812 and S813. The Virtual AON Camera uses the three-axis detection data to determine that the electronic device is still in the static state, and then executes step S820.

S812. The Virtual AON Camera wakes up the AP and sends the recorded display status to the AP.

S813. The Virtual AON Camera sends a face detection event to the AON service.

Wherein, the specific implementation process of step S812 and step S813 may be the same as the content of step S516 and step S517 in the first embodiment, which will not be repeated here. Moreover, after step S812 and step S813 are executed, the electronic device may continue to execute step S518 to step S520 as proposed in the first embodiment, which will not be described here, and may refer to the content of the first embodiment.

It should be noted that the Virtual AON Camera uses the three-axis detection data to determine that the electronic device is not in a static state, and then executes steps S812 and S813, and the AON service can receive face detection events. The AON service can also send the face detection event to the upper layer, so that the AOD application can receive the face detection event and control the display to display the AOD display interface. In this way, it can be realized that even if the difference between the reporting time of the two face detection events before and after is small, it can be inferred that the user continues to be in front of the display screen, but as long as the posture of the electronic device changes, the display screen will be triggered to display the AOD display interface.

S814, Virtual AON Camera judges whether the ambient light brightness fluctuates greatly.

Wherein, the Virtual AON Camera determines that there is a large fluctuation in the ambient light brightness, and then executes steps S815 and S816. The Virtual AON Camera judges that the ambient light brightness does not fluctuate greatly, and then executes step S820.

Step S814 and step S811 can be understood as being executed in parallel.

There is a large fluctuation in the ambient light brightness, which can be understood as: from the moment when the face detection function of the electronic device is turned on, until the Virtual AON Camera judges the reporting time of the face detection event and the reporting time of the last face detection event through step S808 During the period when the difference is smaller than the second threshold, whether there is a big change in the ambient light brightness reported by the ambient light sensor. In some embodiments, if the ambient light brightness reported by the ambient light sensor changes by at least several times, it can be determined that there is a large fluctuation in the ambient light brightness. In some other embodiments, if the difference of the ambient light brightness reported by the ambient light sensor is not maintained within a certain range, it can be determined that the ambient light brightness fluctuates greatly.

S815. The Virtual AON Camera wakes up the AP and sends the recorded display status to the AP.

S816. The Virtual AON Camera sends a face detection event to the AON service.

Wherein, the specific implementation process of step S815 and step S816 may be the same as the content of step S516 and step S517 in the first embodiment, which will not be repeated here. Moreover, after step S815 and step S816 are executed, the electronic device may continue to execute step S518 to step S520 as proposed in the first embodiment, which will not be described here, and may refer to the content of the first embodiment.

It should be noted that the Virtual AON Camera judges that there is a large fluctuation in the ambient light brightness, and then executes steps S815 and S816, and the AON service can receive face detection events. The AON service can also send the face detection event to the upper layer, so that the AOD application can receive the face detection event and control the display to display the AOD display interface. In this way, it can be realized that even if the difference between the reporting time of the two face detection events before and after is small, it can be inferred that the user continues to be in front of the display screen, but as long as the brightness of the environment where the electronic device is located changes, the display screen can be triggered to display AOD UI.

S817. The Virtual AON Camera judges whether there is a receiving change of an approach event and a far event.

Wherein, the Virtual AON Camera determines that there is a change in the reception of the approach event and the far event, and then executes steps S818 and S819. The Virtual AON Camera judges that there is no change in the reception of the approach event and the far event, and then executes step S820.

Step S817, step S814 and step S811 can be understood as being executed in parallel.

The reception change of approaching event and far away event can be understood as: from the moment when the face detection function of the electronic device is turned on, until the Virtual AON Camera judges the reporting time of the face detection event and the report of the last face detection event through step S808 During the period when the time difference is smaller than the second threshold, the proximity light sensor reports a proximity event and also reports a distance event.

S818. The Virtual AON Camera wakes up the AP and sends the recorded display status to the AP.

S819. The Virtual AON Camera sends a face detection event to the AON service.

Wherein, the specific implementation process of step S818 and step S819 may be the same as the content of step S516 and step S517 in the first embodiment, and will not be repeated here. Moreover, after step S818 and step S819 are executed, the electronic device may continue to execute step S518 to step S520 as proposed in the first embodiment, which will not be described here, and may refer to the content of the first embodiment.

It should be noted that the Virtual AON Camera determines that there is a change in the reception of the approach event and the far event, and then executes steps S818 and S819, and the AON service can receive the face detection event. The AON service can also send the face detection event to the upper layer, so that the AOD application can receive the face detection event and control the display to display the AOD display interface. In this way, it can be realized that even if the difference between the reporting time of the two face detection events before and after is small, it can be inferred that the user continues to be in front of the display screen, but as long as the electronic device has a close object approaching or moving away, the display screen can be triggered to display the AOD display interface .

S820, Virtual AON Camera control does not report face detection events.

Wherein, the face continuous detection escape module performs the step of controlling not to report the face detection event.

It should also be noted that steps S811 to S819 may be optional steps. In some embodiments, the Virtual AON Camera executes step S808 to determine that the difference between the reporting time of the face detection event and the reporting time of the last face detection event is less than the second threshold, and directly executes step S820.

It should also be noted that, when the difference between the reporting time of this face detection event and the reporting time of the last face detection event is determined to be less than the second threshold and the face detection event is not reported, if a human hand is detected touching the display The operation of the screen can make the display screen of the electronic device display the AOD display interface.

Another embodiment of the present application also provides a computer-readable storage medium, where instructions are stored in the computer-readable storage medium, and when the computer-readable storage medium is run on a computer or a processor, the computer or the processor executes any one of the above-mentioned methods. one or more steps.

The computer-readable storage medium may be a non-transitory computer-readable storage medium, for example, the non-transitory computer-readable storage medium may be ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, among others.

Another embodiment of the present application also provides a computer program product including instructions. When the computer program product is run on the computer or the processor, the computer or the processor is made to perform one or more steps in any one of the above methods.

Claims

A control method for screen-off display, characterized in that it comprises:

When the display screen of the electronic device is in the off-screen state, the electronic device starts the face detection function and obtains image data through the front camera;

The electronic device identification image data includes all or part of the face image;

When the electronic device determines that the time when the image data includes all or part of the face image is recognized this time, and the difference between the time when the image data is recognized that the image data includes all or part of the face image is not less than a threshold value, the electronic device's The display shows the screen-off display interface.
The control method for screen-off display according to claim 1, wherein the electronic device determines the time when it recognizes that the image data includes all or part of the face image this time, which is different from the previous recognition that the image data includes all or part of the face image. When the time difference of the facial images is not less than the threshold, the display screen of the electronic device displays an off-screen display interface, including:

The electronic device determines the reporting time of the face detection event, and when the difference between the reporting time of the previous face detection event and the reporting time of the previous face detection event is not less than the threshold, the display screen of the electronic device displays an off-screen display interface, and the face detection event The electronic device recognizes that the image data includes all or part of the face image and is generated.
The control method for screen-off display according to claim 1 or 2, further comprising:

The electronic device determines that the image data includes all or part of the face image this time, and when the difference with the time when the image data was recognized last time includes all or part of the face image is less than a threshold, the display screen of the electronic device maintains Screen off status.
The control method for screen-off display according to claim 3, wherein before the display screen of the electronic device maintains the screen-off state, further comprising:

The electronic device determines that the electronic device is continuously in a static state.
The control method for screen-off display according to claim 4, further comprising:

The electronic device determines that the electronic device is not continuously in a static state, and the display screen of the electronic device displays an off-screen display interface.
The control method for screen-off display according to claim 3 or 4, wherein the electronic device determines that the electronic device is continuously in a static state, comprising:

The electronic device determines that the electronic device is in the same posture within a predetermined period of time, and the predetermined period of time refers to: the moment when the face detection function is turned on, until the time when the image data includes all or part of the face image is recognized. Time, the time period in which the difference from the previous time when the image data includes all or part of the face image is not less than the threshold.
The control method for screen-off display according to claim 6, wherein the electronic device determines that the electronic device is in the same posture within a predetermined period of time, comprising:

The electronic device determines that the electronic device is in the same posture within a predetermined period of time by using the three-axis detection data of the acceleration sensor.
The control method for screen-off display according to any one of claims 3 to 7, characterized in that before the display screen of the electronic device maintains the screen-off state, it further includes:

The electronic device determines that the brightness of the environment where the electronic device is located remains stable.
The control method for screen-off display according to claim 8, further comprising:

The electronic device determines that the light of the environment where the electronic device is located is not stable, and the display screen of the electronic device displays an off-screen display interface.
The control method for screen-off display according to claim 8 or 9, wherein the electronic device determines that the light of the environment where the electronic device is located remains stable, comprising:

The electronic device detects ambient light brightness through an ambient light sensor;

The electronic device determines that the brightness of the ambient light detected by the ambient light sensor remains stable within a predetermined period of time, and the predetermined period of time refers to: the moment when the face detection function is turned on, until this recognition image data includes all or the time of a partial face image, and the time period when the difference between the time when the image data includes all or part of the face image is not less than the threshold value.
The control method for screen-off display according to any one of claims 3 to 10, characterized in that before the display screen of the electronic device maintains the screen-off state, it further includes:

The electronic device determines that the electronic device does not detect changes in object approach and distance.
The control method for screen-off display according to claim 11, further comprising:

The electronic device determines that the electronic device detects the change of the object approaching and moving away, and the display screen of the electronic device displays a screen-off display interface.
The control method for screen-off display according to claim 11 or 12, wherein the electronic device determines that the electronic device does not detect changes in the approach and distance of objects, including:

The electronic device detects an object through a proximity light sensor, and obtains a reported event;

The electronic device determines that the reported event continues to be a far away event within a predetermined time period, or continues to be an approach event within a predetermined time period, and the predetermined time period refers to: the moment when the face detection function is turned on, until this time A time period in which the difference between the time when it is recognized that the image data includes all or part of the face image and the time when the previous time when it was recognized that the image data includes all or part of the face image is not less than a threshold value.
The control method for screen-off display according to any one of claims 1 to 13, wherein the processor of the electronic device includes an intelligent sensor hub sensor hub, and the sensor hub includes: an AON module; When the display screen of the electronic device is in the off-screen state, the electronic device starts the face detection function, including:

When the display screen of the electronic device is in the off-screen state, the electronic device starts the AON module to perform the face detection function.
The control method for screen-off display according to any one of claims 1 to 14, wherein the processor of the electronic device comprises: a virtual camera unit Virtual AON Camera; the display screen of the electronic device is in the screen-off state state, the electronic device starts the face detection function, including:

When the Virtual AON Camera determines that the display screen is in an off-screen state, it starts the face detection function.
The control method for screen-off display according to claim 15, wherein the Virtual AON Camera further comprises:

The Virtual AON Camera receives the display state of the display screen and records the display state of the display screen.
The control method for screen-off display according to claim 15, wherein when the Virtual AON Camera determines that the display screen is in the screen-off state, after starting the face detection function, it also includes:

When the Virtual AON Camera determines that the image data includes part or all of the image data, it wakes up the application processor of the electronic device, and sends the display status of the display screen to the application processor.
The control method for screen-off display according to any one of claims 1 to 17, wherein the electronic device determines that the time when the image data is recognized this time includes all or part of the face image is different from the time when it was recognized that the image data included all or part of the face image in the previous time. When the time difference between the image data including all or part of the facial images is not less than the threshold, after the display screen of the electronic device displays the screen-off display interface, it also includes:

When the electronic device judges that the display screen is in a non-off state, the face detection function is turned off.
The control method for screen-off display according to any one of claims 1 to 17, characterized in that, before the electronic device starts the face detection function, it also includes:

The electronic device determines that the electronic device satisfies a preset condition; wherein the preset condition includes that the display screen of the electronic device is not in a downward posture, and the brightness of the environment where the electronic device is located is not less than a first threshold , and the electronic device does not detect an object approaching.
An electronic device, characterized in that it comprises:

one or more processors, memory, displays and front-facing cameras;

The memory, the display screen, and the front camera are coupled to the one or more processors, the memory is used to store computer program codes, and the computer program codes include computer instructions, when the When one or more processors execute the computer instructions, the electronic device executes the screen-off display control method according to any one of claims 1-19.
A computer storage medium, which is characterized in that it is used to store a computer program, and when the computer program is executed, it is specifically used to implement the control method for screen-off display according to any one of claims 1 to 19.