WO2023071882A1

WO2023071882A1 - Human eye gaze detection method, control method and related device

Info

Publication number: WO2023071882A1
Application number: PCT/CN2022/126122
Authority: WO
Inventors: 龚章泉
Original assignee: Oppo广东移动通信有限公司
Priority date: 2021-10-29
Filing date: 2022-10-19
Publication date: 2023-05-04
Also published as: CN113903078A

Abstract

A human eye gaze detection method, a method for controlling an electronic device (100) by means of eyes, a detection apparatus (10), a control apparatus (30), an electronic device (100), and a non-volatile computer-readable storage medium (300). The human eye gaze detection method comprises: (011) acquiring a human eye area image; (013) determining a weighting of the human eye area image according to missing information of the human eye area image; (015) determining human eye gaze information according to the human eye area image and the weighting of the human eye area image.

Description

Human gaze detection method, control method and related equipment

priority information

This application claims the priority and benefit of the patent application No. 202111274110.3 filed with the State Intellectual Property Office of China on October 29, 2021, which is hereby incorporated by reference in its entirety.

technical field

The present application relates to the technical field of consumer electronics products, and in particular to a human eye gaze detection method, a method for controlling electronic equipment through human eyes, a detection device, a control device, electronic equipment, and a non-volatile computer-readable storage medium.

Background technique

Currently, electronic devices can estimate a user's gaze point by collecting face images.

Contents of the invention

The present application provides a human eye gaze detection method, a method for controlling electronic equipment through human eyes, a detection device, a control device, electronic equipment and a non-volatile computer-readable storage medium.

The human eye gaze detection method in one embodiment of the present application includes acquiring an image of the human eye area; determining the weight of the image of the human eye area according to the missing information of the image of the human eye area; The weight of the eye area image determines the gaze information of the human eye.

A detection device according to an embodiment of the present application includes a first acquisition module, a first determination module, and a second determination module. The first acquisition module is used to acquire the human eye area image; the first determination module is used to determine the weight of the human eye area image according to the missing information of the human eye area image; the second determination module uses The human eye gaze information is determined according to the human eye area image and the weight of the human eye area image.

An electronic device according to an embodiment of the present application includes a processor, and the processor is configured to execute a human gaze detection method. The human eye gaze detection method includes acquiring a human eye area image; determining the weight of the human eye area image according to missing information of the human eye area image; Weight, which determines the gaze information of the human eye.

The human eye gaze detection method according to another embodiment of the present application includes acquiring a human face image; determining the human eye area image and human face pose information according to the human face image; The weight of the eye area image; according to the human eye area image and the weight of the human eye area image, determine the human eye feature information; according to the human eye feature information and the human face posture information, determine the human eye gaze information.

The detection device in another embodiment of the present application includes a first acquisition module, a first determination module, a second determination module, a third determination module and a fourth determination module. The first acquiring module is used to acquire a human face image; the first determining module is used to determine the human eye region image and facial posture information according to the human face image; the second determining module is used to determine the human face image according to the human face The missing information of the eye area image determines the weight of the human eye area image; the third determining module is used to determine the human eye feature information according to the human eye area image and the weight of the human eye area image; The fourth determination module is configured to determine human eye gaze information according to the human eye feature information and the human facial posture information.

An electronic device according to another embodiment of the present application includes a processor, and the processor is configured to execute a human gaze detection method. The human eye gaze detection method includes acquiring a human face image; determining the human eye area image and human face posture information according to the human face image; determining the weight of the human eye area image according to the missing information of the human eye area image ; determining human eye feature information according to the human eye area image and the weight of the human eye area image; determining human eye gaze information according to the human eye feature information and the human face posture information.

According to an embodiment of the present application, the method for controlling an electronic device through human eyes includes acquiring an image of a human eye area; determining human eye gaze information according to the weight of the human eye area image and the human eye area image, wherein the human eye The weight of the area image is determined according to the missing information of the human eye area image; and the electronic device is controlled according to the human eye gaze information.

The control device in the embodiment of the present application includes a first acquisition module, a first determination module and a control module. The first acquiring module is used to acquire an image of the human eye area; the first determination module is used to determine the gaze information of the human eye according to the weight of the image of the human eye area and the image of the human eye area, wherein the human eye The weight of the eye area image is determined according to the missing information of the human eye area image; and the control module is used to control the electronic device according to the human eye gaze information.

An electronic device in still another embodiment of the present application includes a processor, and the processor is configured to execute a method for controlling the electronic device through human eyes. The method for controlling an electronic device through human eyes includes acquiring an image of a human eye region; determining human eye gaze information according to the weight of the human eye region image and the human eye region image, wherein the weight of the human eye region image determining according to the missing information of the human eye area image; and controlling the electronic device according to the human eye gaze information.

A non-transitory computer-readable storage medium containing a computer program according to an embodiment of the present application. When the computer program is executed by one or more processors, the processors execute the human eye gaze detection method or pass the human eye gaze detection method. A method of eye-controlling electronic devices. The human eye gaze detection method includes acquiring a human eye area image; determining the weight of the human eye area image according to missing information of the human eye area image; Weight, which determines the gaze information of the human eye. The method for controlling an electronic device with human eyes includes acquiring an image of a human eye region; determining human eye gaze information according to the weight of the human eye region image and the human eye region image, wherein the weight of the human eye region image is based on Determining missing information of the human eye area image; and controlling the electronic device according to the human eye gaze information.

Additional aspects and advantages of the application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the application.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only These are some implementations of the present application. For those skilled in the art, other drawings can also be obtained according to these drawings without creative work.

FIG. 1 is a schematic flow diagram of a human eye gaze detection method in some embodiments of the present application;

FIG. 2 is a block diagram of a detection device in some embodiments of the present application;

3 is a schematic plan view of an electronic device in some embodiments of the present application;

Fig. 4 is a schematic diagram of connection between an electronic device and a cloud server in some embodiments of the present application;

5 to 8 are schematic flowcharts of the human eye gaze detection method in some embodiments of the present application;

9 is a schematic structural diagram of a human eye detection model in some embodiments of the present application;

Fig. 10 is a schematic flow chart of a human eye gaze detection method in some embodiments of the present application;

Figure 11 is a block diagram of a detection device in some embodiments of the present application;

Fig. 12 is a schematic flow chart of a human eye gaze detection method in some embodiments of the present application;

Fig. 13 is a schematic flowchart of a method for controlling an electronic device through human eyes in some embodiments of the present application;

Fig. 14 is a block diagram of a control device in some embodiments of the present application;

15 to 18 are schematic diagrams of scenarios of a method for controlling an electronic device through human eyes in some embodiments of the present application;

19 to 23 are schematic flowcharts of a method for controlling an electronic device through human eyes in some embodiments of the present application;

FIG. 24 and FIG. 25 are schematic diagrams of scenes of a method for controlling an electronic device through human eyes in some embodiments of the present application;

Fig. 26 is a schematic flowchart of a method for controlling an electronic device through human eyes in some embodiments of the present application;

Fig. 27 is a schematic flowchart of a training method of a human eye detection model in some embodiments of the present application;

Figure 28 is a schematic block diagram of a training device according to some embodiments of the present application; and

Fig. 29 is a schematic diagram of connection between a processor and a computer-readable storage medium in some embodiments of the present application.

Detailed ways

Embodiments of the present application will be further described below in conjunction with the accompanying drawings. The same or similar reference numerals in the drawings represent the same or similar elements or elements having the same or similar functions throughout. In addition, the embodiments of the present application described below in conjunction with the accompanying drawings are exemplary, and are only used to explain the embodiments of the present application, and should not be construed as limiting the present application.

The human eye gaze detection method of the present application includes obtaining the human eye area image; determining the weight of the human eye area image according to the missing information of the human eye area image; determining the human eye gaze information according to the human eye area image and the weight of the human eye area image .

In some implementations, the human eye gaze detection method also includes: acquiring face posture information; determining the human eye gaze information according to the human eye area image and the weight of the human eye area image, including: according to the human face posture information, human eye The weight of the area image and the human eye area image determines the gaze information of the human eye.

In some embodiments, the human eye area image includes a left eye area image and a right eye area image, and determining the weight of the human eye area image according to the missing information of the human eye area image includes: according to the first missing information of the left eye area image The information determines the first weight of the image of the left eye area, and the second weight of the image of the right eye area is determined according to the second missing information of the image of the right eye area.

In some implementations, the human eye gaze detection method also includes: acquiring a face image, and the face image includes a face mask; calculating the position information of the face relative to the electronic device according to the face mask; according to the human eye area image and The weight of the human eye region image determines the human eye gaze information, including: determining the human eye gaze information according to the position information, the human eye region image, and the weight of the human eye region image.

In some embodiments, the human eye gaze detection method also includes: obtaining a training sample set, the training sample set includes a plurality of human eye area images; determining the occlusion area and/or image offset parameters according to the human eye area images; Image, occlusion area, and image offset parameters to train the human eye detection model; according to the weight of the human eye area image and the human eye area image, determine the human eye gaze information, including: based on the human eye detection model, according to the human eye area image and the human eye area image The weight of the eye area image determines the gaze information of the human eye.

In some embodiments, training the human eye detection model according to the human eye area image, the occlusion area, and the image offset parameters includes: inputting the human eye area image, the occlusion area, and the image offset parameters into the human eye detection model to output Training coordinates; based on the preset loss function, calculate the loss value according to the preset coordinates corresponding to the human eye area image and the training coordinates; adjust the human eye detection model according to the loss value until the human eye detection model converges.

In some embodiments, the occlusion area is generated based on replacing at least a part of the pixels in the image of the human eye area with pixels of predetermined pixel values; the image offset parameter is generated based on image offset of the feature points of the human eye area in the image of the human eye area .

The human eye gaze detection method of the present application includes obtaining a human face image; determining the human eye area image and human face posture information according to the human face image; determining the weight of the human eye area image according to the missing information of the human eye area image; The weight of the image and the image of the human eye area determines the characteristic information of the human eye; according to the characteristic information of the human eye and the posture information of the human face, the gaze information of the human eye is determined.

In some embodiments, the face image includes a face mask, and determining the gaze information of the human eye according to the feature information of the human eye and the posture information of the human face includes: calculating the position information of the human face relative to the electronic device according to the face mask; According to position information, human eye feature information, and human face posture information, human eye gaze information is determined.

The method for controlling an electronic device through human eyes of the present application includes acquiring an image of a human eye region; determining human eye gaze information according to the weight of the human eye region image Determining the missing information of ; and controlling electronic equipment according to the gaze information of human eyes.

In some implementations, the method for controlling an electronic device through human eyes further includes: obtaining face posture information; determining human eye gaze information according to the human eye area image and the weight of the human eye area image, including: according to the human face posture information , the human eye area image, and the weight of the human eye area image to determine the gaze information of the human eye.

In some embodiments, the human eye area image includes a left eye area image and a right eye area image, the first weight of the left eye area image is determined according to the first missing information of the left eye area image, and the second weight of the right eye area image Determined according to the second missing information of the right-eye region image.

In some embodiments, the method for controlling an electronic device through human eyes further includes: acquiring a face image, and the face image includes a face mask; calculating the position information of the face relative to the electronic device according to the face mask; The weight of the area image and the human eye area image determines the human eye gaze information, including: determining the human eye gaze information according to the position information, the human eye area image, and the weight of the human eye area image.

In some embodiments, the method for controlling an electronic device through human eyes further includes: obtaining a training sample set, the training sample set includes a plurality of human eye area images; determining the occlusion area and/or image offset parameters according to the human eye area images; Human eye area image, occlusion area, and image offset parameters train the human eye detection model; according to the human eye area image and the weight of the human eye area image, determine the human eye gaze information, including: based on the human eye detection model, according to the human eye area The weight of the image and the image of the human eye area determines the gaze information of the human eye.

In some implementations, the gaze information of the human eye includes the coordinates of the gaze point. Before acquiring the image of the human eye area, the method of controlling the electronic device through the human eye further includes: acquiring the captured image within the first predetermined time period before the screen is off. ; in response to a human face being included in the photographed image; controlling the electronic device according to the gaze information of the human eyes, further comprising, in response to the gaze point coordinates being located in the display area of the display screen, continuing to brighten the screen for a second predetermined duration.

In some implementations, the display area is associated with a preset coordinate range, and the method for controlling an electronic device through human eyes further includes: when the gaze point coordinates are within the preset coordinate range, determining that the gaze point coordinates are located in the display area.

In some implementations, before acquiring the image of the human eye region, the method for controlling the electronic device through the human eye further includes: acquiring the captured image in response to the fact that the electronic device does not receive an input operation; controlling the electronic device according to the gaze information of the human eye The device includes: adjusting the display brightness of the display screen to a first predetermined brightness in response to the captured image containing a human face and the gaze point coordinates are located in the display area; in response to the captured image not containing a human face, or the captured image containing a human face and The gaze point coordinates are located outside the display area, and the display brightness is adjusted to a second predetermined brightness, and the second predetermined brightness is smaller than the first predetermined brightness.

The detection device of the present application includes a first acquisition module, a first determination module and a second determination module. The first acquisition module is used to acquire the image of the human eye region; the first determination module is used to determine the weight of the human eye region image according to the missing information of the human eye region image; the second determination module is used to determine the weight of the human eye region image according to the human eye region image and The weight of the image of the human eye area determines the gaze information of the human eye.

The detection device of the present application includes a first acquisition module, a first determination module, a second determination module, a third determination module and a fourth determination module. The first acquisition module is used to acquire the image of the human eye region; the first determination module is used to determine the image of the human eye region and the facial posture information according to the face image; the second determination module is used to determine the missing information of the human eye region image , to determine the weight of the human eye area image; the third determination module is used to determine the human eye feature information according to the human eye area image and the weight of the human eye area image; the fourth determination module is used to determine the human eye feature information according to the human eye feature information and the face Posture information, to determine the gaze information of human eyes.

The control device of the present application includes a first acquisition module, a first determination module and a control module. The first acquisition module is used to acquire the human eye area image; the first determination module is used to determine the human eye gaze information according to the human eye area image and the weight of the human eye area image, wherein the weight of the human eye area image is based on the human eye area The missing information of the image is determined; and the control module is used to control the electronic equipment according to the gaze information of human eyes.

The electronic device of the present application includes a processor, and the processor is configured to execute the human gaze detection method in any one of the foregoing implementation manners.

The electronic device of the present application includes a processor, and the processor is configured to execute the method for controlling an electronic device through human eyes in any one of the foregoing implementation manners.

The non-transitory computer-readable storage medium of the present application includes a computer program. When the computer program is executed by the processor, the processor executes the human eye gaze detection method of any of the above embodiments; or the method of controlling an electronic device through human eyes of any of the above embodiments.

Please refer to Fig. 1 to Fig. 3, the human eye fixation detection method of the embodiment of the present application comprises the following steps:

011: Obtain the image of the human eye area;

013: According to the missing information of the human eye area image, determine the weight of the human eye area image;

015: According to the weight of the human eye area image and the human eye area image, determine the human eye gaze information.

The detection device 10 in the embodiment of the present application includes a first acquisition module 11 , a first determination module 12 and a second determination module 13 . The first acquiring module 11 is used to acquire the human eye region image; the first determining module 12 is used to determine the weight of the human eye region image according to the missing information of the human eye region image; the second determining module 13 is used to determine the weight of the human eye region image according to the human eye region image and The weight of the image of the human eye area determines the gaze information of the human eye. That is to say, step 011 can be implemented by the first acquisition module 11 , step 013 can be performed by the first determination module 12 and step 015 can be performed by the second determination module 13 .

The electronic device 100 in the embodiment of the present application includes a processor 60 and a collection device 30 . Acquisition device 30 is used for collecting face information by predetermined frame rate (face information comprises people's face image, as the visible light image of people's face, infrared image, depth image etc.); Acquisition device 30 can be visible light camera, infrared camera, depth camera One or more of them, wherein the visible light camera can collect visible light face images, the infrared camera can collect infrared face images, and the depth camera can collect depth face images. In this embodiment, the collection device 30 includes a visible light camera, an infrared face image camera and depth camera, the acquisition device 30 simultaneously visible light face image, infrared face image and depth face image. The processor 60 may include an image processor (Image Signal Processor, ISP), a neural network processor (Neural-Network Processing Unit, NPU) and an application processor (Application Processor, AP), and the detection device 10 is arranged in the electronic device 100, Wherein, the first acquisition module 11 can be arranged in the ISP, and the processor 60 is connected to the acquisition device 30. After the acquisition device 30 acquires the face image, the ISP can process the face image to obtain the human eye area image, the first The determination module 12 can also be set at the ISP, and the second determination module 13 can be set at the NPU. The processor 60 (specifically, it can be an ISP) is used to determine the weight of the human eye region image according to the missing information of the human eye region image; the processor 60 (specifically, it can be an NPU) is also used to The weight of is used to determine the gaze information of human eyes. That is to say, step 011 can be executed by the collection device 30 in cooperation with the processor 60 , and steps 013 and 015 can be executed by the processor 60 .

The electronic device 100 may be a mobile phone, a smart watch, a tablet computer, a display device, a notebook computer, a teller machine, a gate, a head-mounted display device, a game machine, and the like. As shown in FIG. 3 , the embodiment of the present application is described by taking the electronic device 100 as a mobile phone as an example. It can be understood that the specific form of the electronic device 100 is not limited to the mobile phone.

Specifically, when the user uses the electronic device 100, the collection device 30 can collect the user's face information once at a predetermined time interval, and continue to perform gaze detection on the user while ensuring that the power consumption of the electronic device 100 is small, or, when the user When using applications that require gaze detection (such as browser software, post bar software, video software, etc.), collect face information according to a predetermined number of frames (such as 10 frames per second), so that human face information is only performed when there is a gaze detection requirement. Face information collection minimizes the power consumption of gaze detection.

Please refer to Fig. 4, after obtaining the face information (taking the face image as an example), the processor 60 can identify the face image, for example, the processor 60 can compare the face image with the preset face template , so as to determine the image area where the face in the face image and different parts of the face (such as eyes, nose, etc.) are located, to identify the eye area in the face image, thereby obtaining the eye area image, wherein The face template can be stored in the memory of the electronic device 100, and the processor 60 can perform face recognition in a trusted execution environment (Trusted Execution Environment, TEE) to ensure the privacy of the user; or, the preset face template can be stored In the cloud server 200, the electronic device 100 then sends the face image to the cloud server 200 for comparison to determine the image of the human eye area, and the face recognition is handed over to the cloud server 200 for processing, which can reduce the processing capacity of the electronic device 100 and Improving image processing efficiency; then, the processor 60 can identify the image of the human eye area to determine the missing information of the human eye. Specifically, the human eye is relatively smooth and has a high reflectivity to light, compared to other parts of the human face. Therefore, by setting the detection threshold, it can be determined whether the pixel is located in the human eye (for example, when the pixel value of the pixel is greater than the detection threshold, it is determined that the pixel is located in the human eye), so as to determine whether the pixel is located in the human eye. In the regional image, the image part of the human eye, and the shape of the human eye is basically determined as an approximate ellipse. According to the recognized image part of the human eye and the preset shape of the human eye, the missing part of the human eye can be determined. The missing information is thus determined, and the missing information may include the proportion of the missing part of the human eye to the human eye.

The processor 60 can determine the weight of the human eye area image according to the missing information. For example, the larger the proportion of the missing part of the human eye to the human eye, the smaller the weight of the human eye area image; it can be understood that the missing part of the human eye The larger the proportion of the human eye, the greater the degree of occlusion of the human eye, and the worse the accuracy of the human eye area image. Therefore, giving a smaller weight can reduce the impact of the human eye area image on the subsequent calculation of human eye gaze information. The influence of accuracy, improve the accuracy of gaze detection.

In one embodiment, the human eye area image includes a left eye area image and a right eye area image, and the processor 60 can respectively determine the first weight of the left eye area image according to the first missing information of the left eye area image, and the first weight of the left eye area image according to the right eye area image. The second missing information of the region image is used to determine the second weight of the right-eye region image, thereby respectively determining the first weight of the left-eye region image and the second weight of the right-eye region image. For example, the first weight is negatively correlated with the first missing information (such as the proportion of the missing part of the left eye to the left eye), and the second weight is negatively correlated with the second missing information (such as the proportion of the missing part of the right eye to the right eye). Negative correlation, when the first weight of the left-eye region image is small (such as 0.6), it can reduce the impact of the left-eye region image on the calculation of human gaze information (such as by reducing the number of feature points from the left-eye region image to reduce the impact).

After the image of the human eye region is acquired and the weight of the image of the human eye region is determined, the gaze information of the human eye can be calculated according to the image of the human eye region and its weight. For example, the gaze information includes the gaze direction and the coordinates of the gaze point, and the processor 60 performs feature extraction on the left eye area and the right eye area in the human eye area image, thereby determining the gaze direction of the human eye and the display of the human eye on the electronic device 100 The gaze point coordinates of the plane where the screen 40 is located.

After the gaze direction and gaze point coordinates are obtained, the electronic device 100 can be controlled according to the gaze direction and gaze point coordinates. If it is detected that the gaze point coordinates are located in the display area of the display screen 40, keep the screen always on, and when it is detected that the gaze point coordinates are located outside the display area of the display screen 40, a predetermined duration (such as 10S, 20S, etc.), then Turn off the screen.

In the face image collected by the electronic device 100 , the image of the eye area may be incomplete due to reasons such as occlusion, thereby affecting the accuracy of gaze point detection.

The human eye gaze detection method, detection device 10, and electronic device 100 of the present application determine the weight of the human eye area image by acquiring the human eye area image and the missing information of the human eye area image. When the image determines the gaze information of the human eye, the weight of the image of the human eye area is considered as a factor, which is beneficial to reduce the impact of the lack of image of the human eye area caused by the occlusion of the human eye, and the impact on the accuracy of the calculation of the gaze information of the human eye, which can improve the accuracy of human eye gaze information. Accuracy of human eye gaze detection in the absence of eye region images.

Please refer to Fig. 2, Fig. 3 and Fig. 5, in some embodiments, the human eye gaze detection method also includes:

0141: Obtain face pose information.

Step 015 includes:

0151: Determine human eye gaze information based on face posture information, human eye area images, and weights of human eye area images.

In some implementations, the detection device 10 further includes a second acquisition module 14, which can also be set in the ISP to acquire face pose information. The second determining module 13 is further configured to determine human eye gaze information according to human face posture information, human eye region images, and weights of human eye region images. That is to say, step 0141 may be performed by the second obtaining module 14 , and step 0151 may be performed by the second determining module 13 .

In some implementations, the processor 60 is further configured to acquire face pose information, and determine human eye gaze information according to the face pose information, the human eye region image, and the weight of the human eye region image. That is to say, step 0141 and step 0151 can be executed by the processor 60 .

Specifically, before calculating the gaze information of the human eyes, the processor 60 can also obtain face posture information, and the face posture information can be calculated according to the position coordinates of the extracted feature points by performing feature extraction on the face image. Posture information, it can be understood that different postures of the face (such as establishing a three-dimensional coordinate system with the tip of the nose as the origin, and the pitch angle, horizontal rotation angle, and tilt angle of the face represent the rotation of the face relative to the three coordinate axes of the three-dimensional coordinate system Angle, etc.), will affect the user's gaze direction and gaze point coordinates. Therefore, when calculating the gaze information of the human eye, in addition to obtaining the image of the human eye area and its weight, it will also combine the facial posture information to more accurately calculate the gaze direction and gaze point coordinates.

Please refer to Fig. 2, Fig. 3 and Fig. 6, in some embodiments, the human eye gaze detection method also includes:

0142: Obtain a face image, where the face image includes a face mask;

0143: Calculate the position information of the face relative to the electronic device 100 according to the face mask;

Step 015 includes:

0152: Determine the gaze information of the human eye according to the position information, the human eye area image, and the weight of the human eye area image.

In some embodiments, the detection device 10 also includes a third acquisition module 15 and a calculation module 16, both of the third acquisition module 15 and the calculation module 16 can be set at the ISP, and the third acquisition module 15 is used to acquire a face image, calculate The module 16 is used for calculating the position information of the face relative to the electronic device 100 according to the face mask. That is to say, step 0142 can be performed by the third acquisition module 15 , step 0143 can be performed by the calculation module 16 , and step 0152 can be performed by the second determination module 13 .

In some implementations, the processor 60 is also used to acquire a face image, and the face image includes a face mask; calculate the position information of the face relative to the electronic device 100 according to the face mask; The weight of the image and the image of the human eye area determines the gaze information of the human eye. . That is to say, step 0142 , step 0143 and step 0152 may be executed by the processor 60 .

Specifically, before calculating the gaze information of the human eyes, the processor 60 may also obtain a face image, and determine a face mask of the face image, where the face mask is used to characterize the position of the face in the face image, The face mask can be determined by identifying the position of the face in the face image, and the processor 60 can calculate the position information of the face relative to the electronic device 100 according to the face mask (such as according to the ratio of the face mask to the face image) , can calculate the distance between the human face and the electronic device 100), it can be understood that when the distance between the human face and the electronic device 100 changes, even if the gaze direction of the human eye does not change, the gaze point coordinates of the human eye will still change. Therefore, in When calculating the gaze information of the human eye, in addition to obtaining the image of the human eye area and its weight, it will also combine the position information to calculate the coordinates of the gaze point more accurately.

In other implementations, when calculating the gaze information of the human eye, in addition to acquiring the image of the eye region and its weight, it may also be combined with facial posture information and position information, so as to calculate the gaze information of the human eye more accurately.

Please refer to Fig. 2, Fig. 3 and Fig. 7, in some embodiments, the human eye gaze detection method also includes:

0101: Obtain a training sample set, the training sample set includes multiple human eye area images;

0102: Determine the occlusion area and/or image offset parameters according to the human eye area image;

0103: Train the human eye detection model according to the human eye area image, occlusion area, and image offset parameters.

Step 015 includes:

0153: Based on the human eye detection model, determine the human eye gaze information according to the human eye area image and the weight of the human eye area image.

In some embodiments, the detection device 10 further includes a fourth acquisition module 17 , a third determination module 18 and a training module 19 . The fourth acquiring module 17, the third determining module 18 and the training module 19 can all be set in the NPU to train the human eye detection model. The fourth acquisition module 17 is used to obtain the training sample set, the third determination module 18 is used to determine the occlusion area and/or image offset parameters according to the human eye area image; the training module 19 is used to determine the human eye area image, the occlusion area, and Image offset parameters to train the human eye detection model. The second determining module 13 may be configured to determine human eye gaze information based on the human eye detection model and according to the human eye region image and the weight of the human eye region image. That is to say, step 0101 can be performed by the fourth acquisition module 17 , step 0102 can be performed by the third determination module 18 , step 0103 can be performed by the training module 19 , and step 0153 can be performed by the second determination module 13 .

In some embodiments, the processor 60 is further configured to obtain a training sample set, the training sample set includes a plurality of human eye area images; determine the occlusion area and/or image offset parameters according to the human eye area images; according to the human eye area images, The occlusion area and image offset parameters train the human eye detection model; based on the human eye detection model, the human eye gaze information is determined according to the weight of the human eye area image and the human eye area image. That is to say, step 0101 , step 0102 , step 0103 and step 0153 may be executed by the processor 60 .

Specifically, this application can realize the calculation of human eye gaze information through the preset human eye detection model, that is, based on the human eye detection model, determine the human eye gaze information according to the weight of the human eye area image and the human eye area image, in order to ensure that people To ensure the accuracy of eye gaze information, it is necessary to train the human eye detection model first, so that the human eye detection model converges.

During training, in order to enable the human eye detection model to accurately calculate the gaze information of the human eye even when the image of the human eye area is occluded, multiple images of the human eye area can be selected in advance as the training sample set, and the selected human eye area The area image may include an occlusion area and/or image offset parameters, wherein the occlusion area is generated by replacing at least a part of pixels in the human eye area image with pixels of a predetermined pixel value (such as 0), and the occlusion area is used for the human eye area image In the case of being occluded, of course, in order to ensure the accuracy of the calculation of the face pose information, the occluded area should not block the eyebrows, eyes, mouth, nose and other related parts of the face; The image offset generation is performed on the point, and the image offset parameter is used to indicate the deviation of the detection of the feature points of the human eye (such as the deviation of the coordinates of the feature points of the human eye).

In this way, the human eye detection model is trained by using the human eye area image with the occlusion area and/or image offset parameters, so that the human eye detection model trained to convergence can minimize the reduction in the human eye gaze information detection. The impact of the occlusion of the human eye and the deviation of the detection of the feature points of the human eye can ensure the accuracy of gaze detection.

Referring to Fig. 2, Fig. 3 and Fig. 8, in some embodiments, step 0103 includes:

01031: Input the human eye area image, occlusion area, and image offset parameters into the human eye detection model to output training coordinates;

01032: Based on the preset loss function, calculate the loss value according to the preset coordinates and training coordinates corresponding to the human eye area image;

01033: Adjust the eye detection model according to the loss value until the eye detection model converges.

In some embodiments, the training module 19 is also used to input the human eye area image, occlusion area, and image offset parameters into the human eye detection model to output training coordinates; based on the preset loss function, according to the human eye area image Calculate the loss value for the corresponding preset coordinates and training coordinates; adjust the human eye detection model according to the loss value until the human eye detection model converges. That is to say, Step 01031 to Step 01033 can be executed by the training module 19 .

In some embodiments, the processor 60 is further configured to input the human eye region image, occlusion region, and image offset parameters into the human eye detection model to output training coordinates; based on a preset loss function, according to the human eye region image Calculate the loss value for the corresponding preset coordinates and training coordinates; adjust the human eye detection model according to the loss value until the human eye detection model converges. That is to say, Step 01031 to Step 01033 may be executed by the processor 60 .

Specifically, referring to FIG. 9 , the face detection model 50 includes a first feature extraction module 51 , a second feature extraction module 52 , a first weight module 53 , a second weight module 54 and a feature fusion module 55 .

When training, the human eye area image is input into the human eye detection model, and the human eye area image includes a left eye area image and a right eye area image, and the first feature extraction module 51 can use the left eye area image ( Or the feature points of the left eye area image) are extracted, the second feature extraction module 52 can extract the right eye area image (or the feature points of the right eye area image) in the human eye area image, and then the first weight module 53 can be The first missing information is determined according to the occlusion area of the left-eye area image, thereby determining the first weight, and the second weight module 54 can determine the second missing information according to the occlusion area of the right-eye area image, thereby determining the second weight, and then the left-eye The region image, the right-eye region image, the first weight and the second weight are input into the feature fusion module for calculation, and the specific calculation method may be to weight the left-eye region image by the first weight (such as first weight * left-eye region image), weight the right eye area image by the second weight (such as the second weight * right eye area image), and then add the weighted features of the two to obtain the fused human eye feature, and then combine the fused The characteristics of the human eye are input into the feature fusion module 55, thereby outputting the gaze information of the human eye.

In other embodiments, the face detection model 50 also includes a third feature extraction module 56, the third feature extraction module 56 can be used to extract feature points in the face image, so as to output the face pose information to the feature fusion module, thereby Calculating human eye gaze information according to the face posture information, the left-eye area image, the right-eye area image, the first weight and the second weight.

When training, the output human eye gaze information is the training coordinates, and there are preset coordinates as the human eye region image of the training sample, and the preset coordinates are the real human eye gaze information (such as the human eye gaze direction and gaze point coordinates), then processor 60 calculates the loss value based on the preset loss function, the training coordinates and the preset coordinates, and the loss function is as follows:

Among them, loss is the loss value, N is the number of training samples contained in each training sample set, X and Y are the training coordinates, Gx and Gy are the preset coordinates, when the training coordinates are the gaze direction, X and Y represent pitch respectively Angle and yaw angle, when the training coordinates are the gaze point coordinates, X and Y represent the coordinates of the gaze point on the plane where the display screen 40 is located, respectively, thereby calculating the loss value of the gaze direction and the loss value of the gaze point coordinates respectively.

Then, the processor 60 can adjust the human eye detection model according to the loss value of the gaze direction and the loss value of the gaze point coordinates, so that the gradient of the human eye detection model continues to decrease, so that the training coordinates are getting closer and closer to the preset coordinates, and finally the human eye The detection model is trained to convergence. In this way, the human eye detection model is trained through the human eye area image with occlusion area and/or image offset parameters, and the human eye detection model is continuously adjusted to convergence based on the loss value, which can minimize the human eye being occluded and The impact of human eye feature point detection offset on gaze detection improves the accuracy of gaze detection.

Please refer to Figure 3, Figure 10 and Figure 11, the detection method of the embodiment of the present application includes the following steps:

021: Obtain a face image;

022: Determine the human eye area image and face posture information according to the face image;

023: Determine the weight of the human eye area image according to the missing information of the human eye area image;

024: Determine the human eye feature information according to the human eye area image and the weight of the human eye area image;

025: Determine the gaze information of the human eye based on the feature information of the human eye and the posture information of the human face.

The detection device 20 in the embodiment of the present application includes a first acquisition module 21 , a first determination module 22 , a second determination module 23 , a third determination module 24 and a fourth determination module 25 . The first acquiring module 21 is used to acquire the human face image; the first determining module 22 is used to determine the human eye region image and the facial posture information according to the human face image; the second determining module 23 is used to determine the missing information of the human eye region image, Determine the weight of the human eye area image; the third determination module 24 is used to determine the human eye feature information according to the human eye area image and the weight of the human eye area image; Information, to determine the human eye gaze information. That is to say, step 021 can be implemented by the first acquisition module 11, step 022 can be performed by the first determination module 12, step 023 can be performed by the second determination module 23, step 024 can be performed by the third determination module 24, step 025 It may be executed by the fourth determining module 25 .

The electronic device 100 in the embodiment of the present application includes a processor 60 and a collection device 30 . Acquisition device 30 is used for collecting face information by predetermined frame rate (face information comprises people's face image, as the visible light image of people's face, infrared image, depth image etc.); Acquisition device 30 can be visible light camera, infrared camera, depth camera One or more of them, wherein the visible light camera can collect visible light face images, the infrared camera can collect infrared face images, and the depth camera can collect depth face images. In this embodiment, the collection device 30 includes a visible light camera, an infrared face image camera and depth camera, the acquisition device 30 simultaneously visible light face image, infrared face image and depth face image. The detection device 20 is arranged in the electronic equipment 100, and the processor 60 may include an ISP, an NPU, and an AP. For example, the first acquisition module 21 may be arranged in the acquisition device 30 to obtain a face image. The first determination module 22 and the second determination module 23 and the third determination module 24 may be set in the ISP, and the fourth determination module 25 may be set in the NPU. Acquisition device 30 is used for obtaining human face image; Processor 60 (specifically can be ISP) is used for determining human eye area image and human face posture information according to human face image; According to the missing information of human eye area image, determine human eye area image weight; according to the weight of the human eye area image and the human eye area image, determine the human eye feature information; processor 60 (specifically can be NPU) is also used for determining the human eye gaze information according to the human eye feature information and the facial posture information . That is to say, step 021 may be executed by the collection device 30 , and steps 022 to 025 may be executed by the processor 60 .

Specifically, when the user uses the electronic device 100, the collection device 30 can collect the user's face information once at a predetermined time interval, and continue to perform gaze detection on the user while ensuring that the power consumption of the electronic device 100 is small, or, when the user When using applications that require gaze detection (such as browser software, post bar software, video software, etc.), the face information is collected according to the predetermined number of frames, so that face information is collected only when gaze detection is required, maximizing Reduced power consumption for gaze detection.

Please refer to Fig. 4, after obtaining the face information (taking the face image as an example), the processor 60 can identify the face image, for example, the processor 60 can compare the face image with the preset face template , so as to determine the image area where the face and different parts of the face (such as glasses, nose, etc.) The face template can be stored in the memory of the electronic device 100, and the processor 60 can perform face recognition in a trusted execution environment (Trusted Execution Environment, TEE) to ensure the privacy of the user; or, the preset face template can be stored In the cloud server 200, the electronic device 100 then sends the face image to the cloud server 200 for comparison to determine the image of the human eye area, and the face recognition is handed over to the cloud server 200 for processing, which can reduce the processing capacity of the electronic device 100 and Improve image processing efficiency.

The processor 60 can also perform feature extraction on the face image, and calculate the face posture information according to the position coordinates of the extracted feature points. The pitch angle, horizontal rotation angle, and tilt angle represent the rotation angles of the face relative to the three coordinate axes of the three-dimensional coordinate system, etc.), which will affect the user's gaze direction and gaze point coordinates. Therefore, when calculating human eye gaze information When , in addition to obtaining the image of the human eye area, it will also combine the facial posture information to more accurately calculate the gaze direction and gaze point coordinates.

Then, the processor 60 can identify the image of the human eye area to determine the missing information of the human eye. Specifically, the human eye is relatively smooth and has a high reflectivity to light. Compared with other parts of the human face, the human eye is The pixel value in the image will be larger, therefore, by setting the detection threshold, it can be determined whether the pixel is located in the human eye (for example, when the pixel value of the pixel is greater than the detection threshold, it is determined that the pixel is located in the human eye), so as to determine the human eye region image. The image part of the human eye, and the shape of the human eye is basically determined and approximately elliptical. According to the recognized image part of the human eye and the preset shape of the human eye, the missing part of the human eye can be determined, thereby determining the missing information. The missing information may include the ratio of the missing part of the human eye to the human eye.

In one embodiment, the human eye area image includes a left eye area image and a right eye area image, and the processor 60 can respectively determine the first weight of the left eye area image according to the first missing information of the left eye area image, and the first weight of the left eye area image according to the right eye area image. The second missing information of the region image is used to determine the second weight of the right-eye region image, thereby respectively determining the first weight of the left-eye region image and the second weight of the right-eye region image. For example, the first weight is negatively correlated with the first missing information (such as the proportion of the missing part of the left eye to the left eye), and the second weight is negatively correlated with the second missing information (such as the proportion of the missing part of the right eye to the right eye). Negative correlation, when the first weight of the left-eye region image is small (for example, 0.6), the influence of the left-eye region image on the calculation of human gaze information can be reduced (such as by extracting fewer features of the left-eye region image point to reduce the impact).

After the image of the human eye region is obtained and the weight of the image of the human eye region is determined, the feature information of the human eye can be determined according to the image of the human eye region and its weight, for example, the weights corresponding to the image of the face region and the image of the human eye region are directly As the human eye feature information, or first extract the human eye feature points from the human eye area image according to the weight of the human eye area image, and then use the extracted human eye feature points as the human eye feature information.

Finally, the processor 60 determines the gaze information of the human eye according to the characteristic information of the human eye and the posture information of the human face. For example, the gaze information includes gaze direction and gaze point coordinates, processor 60 can perform feature extraction on the left eye area and right eye area in the human eye area image respectively, and then combine the facial posture information to determine the gaze direction of the human eye and The gaze point coordinates of the human eyes on the plane where the display screen 40 of the electronic device 100 is located.

The human eye gaze detection method, detection device 20, and electronic device 100 of the present application determine the weight of the human eye area image by acquiring the human eye area image and the missing information of the human eye area image. When the image determines the gaze information of the human eye, the weight of the image of the human eye area is considered as a factor, which is beneficial to reduce the impact of the lack of image of the human eye area caused by the occlusion of the human eye, and the impact on the accuracy of the calculation of the gaze information of the human eye, which can improve the accuracy of human eye gaze information. Accuracy of human eye gaze detection in the absence of eye region images. And combined with the face posture information to jointly determine the human eye gaze information, even if the human eye area image is missing, the human eye gaze information can be accurately calculated through the face posture information, which can improve the accuracy of the human eye gaze information.

Please refer to Fig. 3, Fig. 11 and Fig. 12, in some embodiments, face image comprises face mask, and step 025 comprises:

0251: Calculate the position information of the face relative to the electronic device 100 according to the face mask;

0252: Determine eye gaze information based on position information, eye feature information, and face posture information.

In some embodiments, the fourth determination module 25 is also used to calculate the position information of the human face relative to the electronic device 100 according to the face mask; determine the gaze information of the human eye according to the position information, human eye feature information and human face posture information . That is to say, step 0251 and step 0252 can be executed by the fourth determination module 25 .

In some implementations, the processor 60 is further configured to calculate the position information of the face relative to the electronic device 100 according to the face mask; and determine the gaze information of the human eye according to the position information, the feature information of the human eyes and the posture information of the human face. That is to say, step 0251 and step 0252 can be executed by the processor 60 .

Specifically, before calculating the gaze information of human eyes, the processor 60 can also determine the face mask of the face image, which is used to represent the position of the face in the face image, and the face mask can be obtained by Identify the position of the human face in the human face image, and the processor 60 can calculate the position information of the human face relative to the electronic device 100 (such as the distance between the human face and the electronic device 100) according to the face mask. It can be understood that the human face and the electronic device 100 When the distance of the device 100 changes, even if the gaze direction of the human eye does not change, the coordinates of the gaze point of the human eye will still change. Therefore, when calculating the gaze information of the human eye, in addition to obtaining the image of the human eye region and its weight, and the human face In addition to attitude information, position information will also be combined to more accurately calculate the coordinates of the gaze point.

Please refer to FIG. 3 , FIG. 13 and FIG. 14 , the method for controlling the electronic device 100 through human eyes according to the embodiment of the present application includes the following steps:

031: Obtain the image of the human eye area;

033: Determine the gaze information of the human eye according to the human eye area image and the weight of the human eye area image, wherein the weight of the human eye area image is determined according to the missing information of the human eye area image; and

035: Control the electronic device 100 according to the gaze information of human eyes.

The control device 30 in the embodiment of the present application includes a first acquisition module 31 , a first determination module 32 and a control module 33 . The first acquisition module 31 is used to obtain the human eye region image; the first determination module 32 is used to determine the human eye gaze information according to the human eye region image and the weight of the human eye region image; the control module 33 is used to control the human eye gaze information according to the human eye region image. Electronic device 100. That is to say, step 031 can be implemented by the first acquisition module 31 , step 033 can be performed by the first determination module 32 and step 035 can be performed by the control module 33 .

The electronic device 100 in the embodiment of the present application includes a processor 60 and a collection device 30 . Acquisition device 30 is used for collecting face information by predetermined frame rate (face information comprises people's face image, as the visible light image of people's face, infrared image, depth image etc.); Acquisition device 30 can be visible light camera, infrared camera, depth camera One or more of them, wherein the visible light camera can collect visible light face images, the infrared camera can collect infrared face images, and the depth camera can collect depth face images. In this embodiment, the collection device 30 includes a visible light camera, an infrared face image camera and depth camera, the acquisition device 30 simultaneously visible light face image, infrared face image and depth face image. Processor 60 may include ISP, NPU and AP, such as control device 30 is arranged in electronic equipment 100, the first acquisition module 11 is arranged in ISP, processor 60 is connected with acquisition device 30, after acquisition device 30 collects the facial image The ISP can process the face image to obtain the image of the human eye area, the first determination module 12 can be set in the NPU, and the control module 13 can be set in the AP. The processor 60 (specifically, it may be an ISP) is used to acquire the image of the human eye area; the processor 60 (specifically, it may be an NPU) is also used to determine the human eye gaze information according to the weight of the human eye area image and the human eye area image. The processor 60 (specifically, it may be an AP) can also be used to control the electronic device 100 according to the gaze information of human eyes. That is to say, step 031 can be executed by the collection device 30 in cooperation with the processor 60 , and steps 032 and 033 can be executed by the processor 60 .

Please refer to Fig. 4, after obtaining the face information (taking the face image as an example), the processor 60 can identify the face image, for example, the processor 60 can compare the face image with the preset face template , so as to determine the image area where the face and different parts of the face (such as glasses, nose, etc.) The face template can be stored in the memory of the electronic device 100, and the processor 60 can perform face recognition in a trusted execution environment (Trusted Execution Environment, TEE) to ensure the privacy of the user; or, the preset face template can be stored In the cloud server 200, the electronic device 100 then sends the face image to the cloud server 200 for comparison to determine the image of the human eye area, and the face recognition is handed over to the cloud server 200 for processing, which can reduce the processing capacity of the electronic device 100 and Improve image processing efficiency;

After the gaze direction and gaze point coordinates are obtained, the electronic device 100 can be controlled according to the gaze direction and gaze point coordinates. Please refer to Figure 15. For example, a three-dimensional coordinate system is established with the midpoint of the eyes as the origin O1, the X1 axis is parallel to the direction of the line connecting the centers of the eyes, the Y1 axis is located on the horizontal plane and perpendicular to the X1 axis, and the Z1 axis is perpendicular to the X1 axis and Y1 axis. The three-axis rotation angle of the line of sight S and the three-dimensional coordinate system indicates the user's gaze direction. For example, the gaze direction includes pitch angle, roll angle and yaw angle respectively. The pitch angle represents the rotation angle around the X1 axis, and the roll angle represents the rotation angle around the Y1 axis. The rotation angle of the axis, the yaw angle represents the rotation angle around the Z1 axis, the processor 60 can realize the page turning or sliding operation of the display content of the electronic device 100 according to the gaze direction, for example, according to the determination of continuous multiple frames of human eye area images (such as 10 consecutive frames) of the gaze direction, the change of the gaze direction can be determined, for example, please combine Figure 15 and Figure 16, when the pitch angle gradually increases (that is, the line of sight S is tilted), it can be determined that the user wants the displayed content to slide up or Turning the page down. For another example, please refer to FIG. 15 and FIG. 17 , the pitch angle gradually decreases (that is, the line of sight S is tilted), then it can be determined that the user wants to slide the displayed content down or turn the page up. Similarly, by detecting the moving direction of the gaze point M, the electronic device 100 can also be slid or page-turned. Please refer to FIG. 18 , the center of the display screen 40 can be used as the coordinate origin O2 to establish a plane coordinate system, the width direction parallel to the electronic device 100 is used as the X2 axis, the length direction parallel to the electronic device 100 is used as the Y2 axis, and the gaze point coordinates include the abscissa (corresponding to the position on the X2 axis) and the ordinate (corresponding to the position on the Y2 axis). If the ordinate gradually increases, it means that the gaze point M moves up. It can be determined that the user wants to slide up or turn the page down, and then For example, if the ordinate gradually decreases, it means that the gaze point M moves down, and it can be determined that the user wants to slide the displayed content down or turn the page up.

In other implementations, the processor 60 can also obtain 10 consecutive frames according to the change speed of the gaze direction (such as the difference between the pitch angles of the first frame and the tenth frame (or the difference between the vertical coordinates of the gaze point M) and The duration is determined), the faster the change speed, the more new display content will be displayed after sliding.

In another example, when it is detected that the coordinates of the gaze point are located in the display area of the display screen 40, it means that the user is always viewing the display screen 40, and the screen is kept on all the time, and when it is detected that the coordinates of the gaze point are located in the display area of the display screen 40 When outside the area, it means that the user has not checked the display screen 40, but in order to prevent the user from looking around outside the display area once in a while, causing a misjudgment, the predetermined time length (such as 10S, 20S, etc.) after the user does not check the display screen 40 can be Turn off the screen again.

The method for controlling the electronic device 100 through human eyes, the control device 30 and the electronic device 100 of the present application determine the weight of the human eye region image by acquiring the human eye region image and the missing information of the human eye region image. When determining the human eye gaze information based on the human eye area image, the weight of the human eye area image is considered as a factor, which is beneficial to reduce the impact of the human eye area image being missing due to the occlusion of the human eye, and the accuracy of the calculation of the human eye gaze information. , can improve the accuracy of human eye gaze detection under the condition that the image of the human eye area is missing, and then control the electronic device 100 through accurate human eye gaze information, thereby ensuring the control accuracy of the electronic device 100 .

Referring to FIG. 3 , FIG. 14 and FIG. 19 , in some embodiments, the method for controlling the electronic device 100 through human eyes further includes:

0321: Obtain face pose information.

Step 033 includes:

0331: Determine the human eye gaze information according to the face posture information, the human eye area image, and the weight of the human eye area image.

In some implementations, the control device 30 further includes a second acquisition module 34, which can also be set in the ISP to acquire facial posture information. The first determining module 32 is further configured to determine human eye gaze information according to the human face posture information, the human eye area image, and the weight of the human eye area image. That is to say, step 0321 can be performed by the second obtaining module 34 , and step 0331 can be performed by the first determining module 32 .

In some implementations, the processor 60 is further configured to acquire face pose information, and determine human eye gaze information according to the face pose information, the human eye region image, and the weight of the human eye region image. That is to say, step 0321 and step 0331 can be executed by the processor 60 .

For the specific description of step 0321, please refer to step 0141, and for the specific description of step 0331, please refer to step 0151, which will not be repeated here.

Please refer to FIG. 3 , FIG. 14 and FIG. 20 , in some embodiments, the method for controlling the electronic device 100 through human eyes further includes:

0322: Obtain a face image, where the face image includes a face mask;

0323: Calculate the position information of the face relative to the electronic device 100 according to the face mask;

Step 033 includes:

0332: Determine the gaze information of the human eye according to the position information, the image of the human eye area, and the weight of the image of the human eye area.

In some embodiments, the control device 30 also includes a third acquisition module 35 and a calculation module 36, both of the third acquisition module 35 and the calculation module 36 can be set at the ISP, and the third acquisition module 35 is used to acquire a face image, calculate The module 36 is used for calculating the position information of the face relative to the electronic device 100 according to the face mask. That is to say, step 0322 can be performed by the third acquisition module 35 , step 0332 can be performed by the calculation module 16 , and step 0332 can be performed by the first determination module 32 .

In some implementations, the processor 60 is further configured to acquire face pose information, and determine human eye gaze information according to the face pose information, the human eye region image, and the weight of the human eye region image. That is to say, step 0322 , step 323 and step 0332 may be executed by the processor 60 .

Specifically, for detailed descriptions of Step 0322, Step 0323, and Step 0332, please refer to Step 0142, Step 0143, and Step 0152, respectively, and details are not repeated here.

Please refer to FIG. 3 , FIG. 14 and FIG. 21 , in some embodiments, the method for controlling the electronic device 100 through human eyes further includes:

0301: Obtain a training sample set, the training sample set includes multiple human eye area images;

0302: Determine the occlusion area and/or image offset parameters according to the human eye area image;

0303: Train the human eye detection model according to the human eye area image, occlusion area, and image offset parameters.

Step 033 includes:

0333: Based on the human eye detection model, determine the human eye gaze information according to the human eye area image and the weight of the human eye area image.

In some implementations, the control device 30 further includes a fourth acquisition module 37 , a second determination module 38 and a training module 39 . The fourth acquiring module 37, the second determining module 38 and the training module 39 can all be set in the NPU to train the human eye detection model. The fourth acquisition module 37 is used to obtain the training sample set, the second determination module 38 is used to determine the occlusion area and/or image offset parameters according to the human eye area image; Image offset parameters to train the human eye detection model. The first determining module 32 may be configured to determine human eye gaze information based on the human eye detection model and according to the human eye region image and the weight of the human eye region image. That is to say, step 0301 can be performed by the fourth acquisition module 37 , step 0302 can be performed by the second determination module 38 , step 0103 can be performed by the training module 39 , and step 0333 can be performed by the first determination module 32 .

In some embodiments, the processor 60 is further configured to obtain a training sample set, the training sample set includes a plurality of human eye area images; determine the occlusion area and/or image offset parameters according to the human eye area images; according to the human eye area images, The occlusion area and image offset parameters train the human eye detection model; based on the human eye detection model, the human eye gaze information is determined according to the weight of the human eye area image and the human eye area image. That is to say, step 0301 , step 0302 , step 0303 and step 0333 may be executed by the processor 60 .

Specifically, for specific descriptions of Step 0301, Step 0302, Step 0303, and Step 0333, please refer to Step 0101, Step 0102, Step 0103, and Step 0153, respectively, and details are not repeated here.

Referring to Fig. 3, Fig. 14 and Fig. 22, in some embodiments, step 0303 includes:

03031: Input the human eye area image, occlusion area, and image offset parameters into the human eye detection model to output training coordinates;

03032: Based on the preset loss function, calculate the loss value according to the preset coordinates and training coordinates corresponding to the human eye area image;

03033: Adjust the eye detection model according to the loss value until the eye detection model converges.

In some embodiments, the training module 39 is also used to input the human eye area image, occlusion area, and image offset parameters into the human eye detection model to output training coordinates; based on a preset loss function, according to the human eye area image Calculate the loss value for the corresponding preset coordinates and training coordinates; adjust the human eye detection model according to the loss value until the human eye detection model converges. That is to say, Step 03031 to Step 03033 can be executed by the training module 39 .

In some embodiments, the processor 60 is further configured to input the human eye region image, occlusion region, and image offset parameters into the human eye detection model to output training coordinates; based on a preset loss function, according to the human eye region image Calculate the loss value for the corresponding preset coordinates and training coordinates; adjust the human eye detection model according to the loss value until the human eye detection model converges. That is to say, Step 03031 to Step 03033 may be executed by the processor 60 .

Specifically, for specific descriptions of Step 03031, Step 03032, and Step 03033, please refer to Step 01031, Step 01032, and Step 01033, respectively, and details are not repeated here.

Please refer to FIG. 3, FIG. 11 and FIG. 23. In some implementations, the gaze information of the human eye includes gaze point coordinates. Before acquiring the image of the human eye area, the method for controlling the electronic device 100 through the human eye further includes:

0301: Obtain the captured image within the first predetermined time period before the screen is off;

0302: Responding to a captured image containing a human face;

Step 035 includes:

0351: In response to the gaze point coordinates being located in the display area of the display screen 40, keep the screen on for a second predetermined duration.

In some implementations, the control module 33 is also configured to acquire a captured image within the first predetermined time before the screen is off; in response to the captured image containing a human face; in response to the gaze point coordinates being located in the display area of the display screen 40, Keep the screen on for a second predetermined duration. That is to say, step 0301 , step 0302 and step 0351 can be executed by the control module 33 .

In some implementations, the processor 60 is further configured to acquire a captured image within the first predetermined time period before the screen turns off; in response to the captured image containing a human face; in response to the gaze point coordinates being located in the display area of the display screen 40, Keep the screen on for a second predetermined duration. That is to say, step 0351 and step 0352 can be executed by the processor 60 .

Specifically, the human eye gaze information can be used to realize off-screen control. Before the screen is off, gaze detection is first performed. For example, the processor 60 first obtains a captured image. If there is a human face in the captured image, the human eye gaze information is performed according to the captured image. Of course, in order to ensure that there is enough time to acquire the captured image and calculate the gaze information before the screen is closed, it is necessary to acquire the captured image within the first predetermined time period (such as 5 seconds, 10 seconds, etc.) before the screen is closed.

Referring to Fig. 24 and Fig. 25, when the gaze point M is located in the display area of the display screen 40, it can be determined that the user is watching the display screen 40, so that the display screen 40 remains on for a second predetermined duration, and the second predetermined duration can be greater than the first predetermined time length, and within the first predetermined time length before the screen is turned off again, the captured image is acquired again, so that when the user looks at the display screen 40, the screen remains bright, and when the user no longer looks at the display screen 40, the screen is turned off again. Screen.

Wherein, please refer to FIG. 18 again, the center of the display area can be used as the coordinate origin O2 to establish a two-dimensional coordinate system parallel to the display screen 40, and the display area is associated with a preset coordinate range, that is, the display area is in the horizontal direction of the two-dimensional coordinate system. The coordinate range and the ordinate range, as the preset coordinate range, can be determined when the gaze point coordinates are within the preset coordinate range (that is, the abscissa of the gaze point coordinates is within the abscissa range and the ordinate is within the ordinate range). The gaze point coordinates are located in the display area, so it is relatively simple to determine whether the user gazes at the display screen 40 .

Moreover, since the acquisition of captured images and the calculation of gaze information of human eyes are performed only within the first predetermined period of time before the screen turns off, it is beneficial to save power consumption.

Please refer to FIG. 3, FIG. 14 and FIG. 26. In some implementations, the gaze information of the human eye includes the gaze point coordinates. Before acquiring the image of the human eye area, the method for controlling the electronic device 100 through the human eye further includes:

0303: Obtain a captured image in response to the fact that the electronic device 100 does not receive an input operation;

Step 035 includes:

0352: Adjust the display brightness of the display screen 40 to a first predetermined brightness in response to the captured image containing a human face and the gaze point coordinates are located in the display area;

0353: In response to the fact that the captured image does not contain a human face, or the captured image contains a human face and the gaze point coordinates are outside the display area, adjust the display brightness to a second predetermined brightness, where the second predetermined brightness is smaller than the first predetermined brightness.

In some implementations, the control module 33 is further configured to acquire a captured image in response to the fact that the electronic device 100 does not receive an input operation, and adjust the display screen 40 in response to the captured image containing a human face and the coordinates of the gaze point located in the display area. Adjust the display brightness to the first predetermined brightness, and adjust the display brightness to the second predetermined brightness in response to the fact that the captured image does not contain a human face, or the captured image contains a human face and the gaze point coordinates are outside the display area, and the second predetermined brightness less than the first predetermined brightness. That is to say, step 0303 , step 0352 and step 0353 can be executed by the control module 33 .

In some implementations, the processor 60 is further configured to acquire a captured image in response to the fact that the electronic device 100 does not receive an input operation, and adjust the display screen 40 in response to the captured image containing a human face and the coordinates of the gaze point located in the display area. Adjust the display brightness to the first predetermined brightness, and adjust the display brightness to the second predetermined brightness in response to the fact that the captured image does not contain a human face, or the captured image contains a human face and the gaze point coordinates are outside the display area, and the second predetermined brightness less than the first predetermined brightness. That is to say, step 0303 , step 0352 and step 0353 can be executed by the processor 60 .

Specifically, please refer to Fig. 24 and Fig. 25 again. The human eye gaze information can also be used to realize intelligent brightening of the screen. In order to save power, the electronic device 100 will generally reduce the display brightness first after a certain period of time when the screen is bright, and then turn on the screen with a low brightness. After a certain period of time, the screen will stop. In this embodiment, when the electronic device 100 does not receive the user's input operation, it can be judged that the user may not be using the electronic device 100 or is only viewing the displayed content, and the processor 60 can obtain the captured image. If the image contains a human face, the gaze information of the human eye is calculated according to the captured image. If the coordinates of the gaze point are located in the display area, it means that although the user is not operating the electronic device 100, but is checking the display content, the display brightness is adjusted to the first predetermined brightness at this time. The first predetermined brightness may be the brightness set by the user when the display screen 40 is normally displayed, or it may be changed in real time according to the brightness of the ambient light to adapt to the brightness of the ambient light, so as to ensure that the user can still brighten the electronic device 100 even if the user does not operate the electronic device 100. screen, so as to prevent the situation that the user does not operate the electronic device 100 but suddenly turns off the screen when viewing the displayed content and affects the user experience.

However, when the electronic device 100 does not receive an input operation, if the captured image does not contain a human face, or although the captured image contains a human face, the gaze point coordinates are outside the display area (that is, the user does not view the display area), then It can be determined that the user does not need to use the electronic device 100 at present, therefore, at this time, the display brightness can be adjusted to a second predetermined brightness, which is smaller than the first predetermined brightness, so as to prevent unnecessary power consumption. When the user looks at the display area again, the display brightness is adjusted to the first predetermined brightness again, so as to ensure the normal viewing experience of the user. In this way, it can be realized that when the user does not operate the electronic device 100, the user looks at the display area, and the display area is displayed at normal brightness; Save battery.

Please refer to Fig. 3, Fig. 27 and Fig. 28, the training method of the human eye detection model of the embodiment of the present application includes:

041: Obtain a training sample set, the training sample set includes multiple human eye area images;

042: Determine the occlusion area and/or image offset parameters according to the human eye area image;

043: Train the human eye detection model according to the human eye area image, occlusion area, and image offset parameters.

The training device 40 in the embodiment of the present application includes an acquisition module 41 , a determination module 42 and a training module 43 . The acquiring module 41, the determining module 42 and the training module 43 can all be set in the NPU to train the human eye detection model. The obtaining module 41 is used to obtain the training sample set, and the determining module 42 is used to determine the occlusion area and/or image offset parameters according to the human eye area image; the training module 43 is used to determine the human eye area image, the occlusion area, and the image offset parameters Train the human eye detection model. That is to say, step 041 may be performed by the acquisition module 41 , step 042 may be performed by the determination module 42 , and step 043 may be performed by the training module 43 .

In some embodiments, the processor 60 is further configured to obtain a training sample set, the training sample set includes a plurality of human eye area images; determine the occlusion area and/or image offset parameters according to the human eye area images; according to the human eye area images, The occlusion area and image offset parameters are used to train the human eye detection model. That is to say, step 041 , step 042 and step 043 may be executed by the processor 60 .

Specifically, please refer to step 0101, step 0102, and step 0103 for specific descriptions of step 041, step 042, and step 043, and details are not repeated here.

Referring to FIG. 29 , one or more non-transitory computer-readable storage media 300 containing a computer program 302 according to an embodiment of the present application, when the computer program 302 is executed by one or more processors 60, the processors 60 can Execute the human eye gaze detection method or the method for controlling the electronic device 100 through human eyes in any one of the above embodiments.

For example, referring to FIG. 1 to FIG. 3 , when the computer program 302 is executed by one or more processors 60, the processors 60 are made to perform the following steps:

011: Obtain the image of the human eye area;

013: Determine the weight of the human eye area image according to the missing information of the human eye area image;

For another example, please refer to FIG. 3 , FIG. 10 and FIG. 11 , when the computer program 302 is executed by one or more processors 60, the processors 60 may also perform the following steps:

021: Obtain a face image;

For another example, please refer to FIG. 3 , FIG. 13 and FIG. 14 , when the computer program 302 is executed by one or more processors 60, the processors 60 may also perform the following steps:

031: Obtain the image of the human eye area;

In the description of this specification, descriptions with reference to the terms "one embodiment", "some embodiments", "exemplary embodiments", "example", "specific examples" or "some examples" mean that a combination of the embodiments or Examples describe specific features, structures, materials, or characteristics that are included in at least one embodiment or example of the present application. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the described specific features, structures, materials or characteristics may be combined in any suitable manner in any one or more embodiments or examples. In addition, those skilled in the art can combine and combine different embodiments or examples and features of different embodiments or examples described in this specification without conflicting with each other.

Any process or method descriptions in flowcharts or otherwise described herein may be understood to represent modules, segments or portions of code comprising one or more executable instructions for implementing specific logical functions or steps of the process , and the scope of preferred embodiments of the present application includes additional implementations in which functions may be performed out of the order shown or discussed, including in substantially simultaneous fashion or in reverse order depending on the functions involved, which shall It should be understood by those skilled in the art to which the embodiments of the present application belong.

Although the implementation of the present application has been shown and described above, it can be understood that the above-mentioned implementation is exemplary and should not be construed as limiting the application, and those skilled in the art can make the above-mentioned The embodiments are subject to changes, modifications, substitutions and variations.

Claims

A human eye gaze detection method, characterized in that, comprising:

Obtain an image of the human eye area;

determining the weight of the human eye region image according to the missing information of the human eye region image;

Determine human eye gaze information according to the human eye area image and the weight of the human eye area image.
The human eye gaze detection method according to claim 1, further comprising:

Obtain face pose information;

According to the weight of the human eye area image and the human eye area image, determining the human eye gaze information includes:

Determine human eye gaze information according to the human face posture information, the human eye area image, and the weight of the human eye area image.
The human eye gaze detection method according to claim 1, wherein the human eye region image comprises a left eye region image and a right eye region image, and the determination of the The weight of the image of the human eye area, including:

determining a first weight of the left-eye region image according to first missing information of the left-eye region image, and

Determining a second weight of the right eye area image according to the second missing information of the right eye area image.
The human eye gaze detection method according to claim 1, further comprising:

Obtain a face image, the face image includes a face mask;

Calculate the position information of the face relative to the electronic device according to the face mask;

According to the weight of the human eye area image and the human eye area image, determining the human eye gaze information includes:

Determine human eye gaze information according to the position information, the human eye area image, and the weight of the human eye area image.
The human eye gaze detection method according to claim 1, further comprising:

Obtain a training sample set, the training sample set includes a plurality of images of the human eye region;

determining an occlusion area and/or an image offset parameter according to the human eye area image;

Training a human eye detection model according to the human eye area image, the occluded area, and the image offset parameters;

According to the weight of the human eye area image and the human eye area image, determining the human eye gaze information includes:

Based on the human eye detection model, the human eye gaze information is determined according to the human eye area image and the weight of the human eye area image.
The human eye gaze detection method according to claim 5, wherein the human eye detection model is trained according to the human eye area image, the occluded area, and the image offset parameters, comprising:

inputting the human eye area image, the occluded area, and the image offset parameters into the human eye detection model to output training coordinates;

Based on a preset loss function, calculate a loss value according to preset coordinates corresponding to the human eye area image and the training coordinates;

Adjusting the human eye detection model according to the loss value until the human eye detection model converges.
The human eye gaze detection method according to claim 6, wherein the occlusion region is generated according to pixels that replace at least a part of pixels in the human eye region image with predetermined pixel values; the image offset parameters are based on Image offset generation is performed on the human eye feature points in the human eye area image.
A human eye gaze detection method, characterized in that, comprising:

Get face image;

Determining the human eye area image and facial posture information according to the human face image;

determining the weight of the human eye region image according to the missing information of the human eye region image;

determining human eye feature information according to the human eye region image and the weight of the human eye region image;

Determine human eye gaze information according to the human eye feature information and the human face posture information.
The human eye gaze detection method according to claim 8, wherein the human face image comprises a human face mask, and the human eye gaze information is determined according to the human eye feature information and the human face posture information ,include:

Calculate the position information of the face relative to the electronic device according to the face mask;

Determine human eye gaze information according to the position information, the human eye feature information, and the human face posture information.
A method for controlling an electronic device through human eyes, comprising:

Obtain an image of the human eye region;

Determine human eye gaze information according to the human eye region image and the weight of the human eye region image, wherein the weight of the human eye region image is determined according to missing information of the human eye region image; and

The electronic device is controlled according to the gaze information of the human eyes.
The method for controlling an electronic device through human eyes according to claim 10, further comprising:

Obtain face pose information;

According to the weight of the human eye area image and the human eye area image, determining the human eye gaze information includes:

Determine human eye gaze information according to the human face posture information, the human eye area image, and the weight of the human eye area image.
The method for controlling an electronic device through human eyes according to claim 10, wherein the human eye region image includes a left eye region image and a right eye region image, and the first weight of the left eye region image is based on the The first missing information of the left-eye area image is determined, and the second weight of the right-eye area image is determined according to the second missing information of the right-eye area image.
The method for controlling an electronic device through human eyes according to claim 10, further comprising:

Obtain a face image, the face image includes a face mask;

Calculate the position information of the face relative to the electronic device according to the face mask;

According to the weight of the human eye area image and the human eye area image, determining the human eye gaze information includes:

Determine human eye gaze information according to the position information, the human eye area image, and the weight of the human eye area image.
The method for controlling an electronic device through human eyes according to claim 10, further comprising:

Obtain a training sample set, the training sample set includes a plurality of images of the human eye region;

determining an occlusion area and/or an image offset parameter according to the human eye area image;

Training a human eye detection model according to the human eye area image, the occluded area, and the image offset parameters;

According to the weight of the human eye area image and the human eye area image, determining the human eye gaze information includes:

Based on the human eye detection model, the human eye gaze information is determined according to the human eye area image and the weight of the human eye area image.
The method for controlling an electronic device through human eyes according to claim 14, wherein the training of the human eye detection model according to the human eye area image, the occluded area, and the image offset parameters includes:

inputting the human eye area image, the occluded area, and the image offset parameters into the human eye detection model to output training coordinates;

Based on a preset loss function, calculate a loss value according to preset coordinates corresponding to the human eye area image and the training coordinates;

Adjusting the human eye detection model according to the loss value until the human eye detection model converges.
The method for controlling an electronic device through human eyes according to claim 15, wherein the occlusion area is generated by replacing at least a part of pixels in the image of the human eye area with pixels of predetermined pixel values; The shift parameter is generated according to performing image shift on the human eye feature points in the human eye area image.
The method for controlling an electronic device through human eyes according to claim 10, wherein the gaze information of the human eyes includes gaze point coordinates, and before acquiring the image of the human eye area, the method for controlling the electronic device through the human eyes, Also includes:

within the first predetermined period of time before the screen is off, acquire the photographed image;

In response to the captured image containing a human face;

The controlling the electronic device according to the gaze information of the human eyes also includes

In response to the gaze point coordinates being located in the display area of the display screen, the screen is kept on for a second predetermined duration.
The method for controlling an electronic device through human eyes according to claim 17, wherein the display area is associated with a preset coordinate range, and the method for controlling an electronic device through human eyes further comprises:

When the gaze point coordinates are within the preset coordinate range, it is determined that the gaze point coordinates are located in the display area.
The method for controlling an electronic device through human eyes according to claim 10, wherein, before acquiring the image of the human eye area, the method for controlling an electronic device through human eyes further comprises:

Acquiring a captured image in response to the fact that the electronic device does not receive an input operation;

The controlling the electronic device according to the gaze information of the human eyes includes:

Adjusting the display brightness of the display screen to a first predetermined brightness in response to the captured image containing a human face and the gaze point coordinates being located in a display area of the display screen;

In response to the captured image not containing a human face, or the captured image containing a human face and the gaze point coordinates are located outside the display area, adjusting the display brightness to a second predetermined brightness, the second The predetermined brightness is smaller than the first predetermined brightness.
A detection device is characterized in that it comprises:

The first acquisition module is used to acquire images of the human eye region;

A first determining module, configured to determine the weight of the human eye region image according to the missing information of the human eye region image;

The second determination module is configured to determine human eye gaze information according to the human eye region image and the weight of the human eye region image.
A detection device is characterized in that it comprises:

The first acquisition module is used to acquire face images;

The first determination module is used to determine the human eye area image and the facial posture information according to the human face image;

The second determination module is used to determine the weight of the human eye area image according to the missing information of the human eye area image;

A third determining module, configured to determine human eye feature information according to the human eye area image and the weight of the human eye area image;

The fourth determination module is configured to determine human eye gaze information according to the human eye feature information and the human facial posture information.
A control device, characterized in that it comprises:

The first acquisition module is used to acquire images of the human eye region;

The first determination module is configured to determine human eye gaze information according to the human eye area image and the weight of the human eye area image, wherein the weight of the human eye area image is based on the missing information of the human eye area image determine; and

A control module, configured to control electronic equipment according to the gaze information of human eyes.
An electronic device, characterized in that it includes a processor, the processor is configured to execute the human gaze detection method according to any one of claims 1-7.
An electronic device, characterized by comprising a processor, the processor is configured to execute the human gaze detection method according to claim 8 or 9.
An electronic device is characterized in that it includes a processor, and the processor is used to execute the method for controlling an electronic device through human eyes according to any one of claims 10-19.
A non-volatile computer-readable storage medium comprising a computer program, when the computer program is executed by a processor, the processor is made to perform the human eye gaze detection method described in any one of claims 1-7, or The human eye gaze detection method described in claim 8 or 9, or the method for controlling electronic equipment through human eyes described in any one of claims 10-19.