CN113936324A

CN113936324A - Gaze detection method, control method of electronic device and related device

Info

Publication number: CN113936324A
Application number: CN202111271397.4A
Authority: CN
Inventors: 龚章泉
Original assignee: Guangdong Oppo Mobile Telecommunications Corp Ltd
Current assignee: Guangdong Oppo Mobile Telecommunications Corp Ltd
Priority date: 2021-10-29
Filing date: 2021-10-29
Publication date: 2022-01-14
Also published as: WO2023071884A1

Abstract

The gazing detection method comprises the steps of determining posture information of a human face according to human face information; determining a correction parameter according to the attitude information in response to the attitude information being greater than a preset threshold; and determining gazing information according to the coordinates of the reference gazing point and the correction parameters. According to the gaze detection method, the control method, the detection device, the control device, the electronic device and the nonvolatile computer readable storage medium, when the gesture information is larger than the preset threshold value and the calculation accuracy of the gaze point coordinate is affected, the reference gaze point coordinate is calculated according to the face information, and then the correction parameter is calculated according to the gesture information, so that the reference gaze point coordinate is corrected to obtain accurate gaze information, the influence of too large face shooting angle on gaze detection in the obtained face information is prevented, and the gaze detection accuracy can be improved. And when the electronic equipment is controlled according to the accurate gazing information, the control accuracy of the electronic equipment can be improved.

Description

Gaze detection method, control method of electronic device and related device

Technical Field

The present application relates to the field of consumer electronics technologies, and in particular, to a gaze detection method, a control method of an electronic device, a detection apparatus, a control apparatus, an electronic device, and a non-volatile computer-readable storage medium.

Background

At present, an electronic device can estimate a gaze point of a user by collecting a face image, but in the face image collected by the electronic device, the obtained face information is not accurate enough due to reasons such as a shooting angle, so that the detection accuracy of the gaze point is affected.

Disclosure of Invention

The application provides a gaze detection method, a control method of an electronic device, a detection apparatus, a control apparatus, an electronic device, and a non-volatile computer-readable storage medium.

The gazing detection method of one embodiment of the application comprises the steps of determining posture information of a human face according to human face information, and determining a reference gazing point coordinate according to the human face information; determining a correction parameter according to the attitude information in response to the attitude information being greater than a preset threshold; and determining gazing information according to the coordinate of the reference gazing point and the correction parameters.

The detection device of one embodiment of the application comprises a first determination module, a second determination module and a third determination module. The first determining module is used for determining the posture information of the face according to the face information and determining the coordinate of the reference fixation point according to the face information; the second determining module is used for responding to the situation information being larger than a preset threshold value, and determining a correction parameter according to the situation information; the third determining module is used for determining gazing information according to the reference gazing point coordinate and the correction parameter.

The electronic equipment of one embodiment of the application comprises a processor, a processing unit and a display unit, wherein the processor is used for determining the posture information of a human face according to human face information and determining the coordinates of a reference fixation point according to the human face information; determining a correction parameter according to the attitude information in response to the attitude information being greater than a preset threshold; and determining gazing information according to the coordinate of the reference gazing point and the correction parameters.

According to the gazing detection method, the detection device and the electronic equipment, after the face information is obtained, the face gesture is calculated through the face information, when the gesture information is larger than a preset threshold value and the calculation accuracy of the gazing point coordinate can be influenced, the reference gazing point coordinate is firstly calculated according to the face information, and then the correction parameter is calculated according to the gesture information, so that the reference gazing point coordinate can be corrected according to the correction parameter, the influence of overlarge face shooting angle on gazing detection in the obtained face information is prevented, and the gazing detection accuracy can be improved.

The control method of the electronic equipment comprises the steps of determining posture information of a human face according to human face information, and determining a reference fixation point coordinate according to the human face information; determining a correction parameter according to the attitude information in response to the attitude information being greater than a preset threshold; determining gazing information according to the coordinates of the reference gazing point and the correction parameters; and controlling the electronic equipment according to the gazing information.

The control device of the embodiment of the application comprises an obtaining module, a first determining module and a second determining module. The acquisition module is used for determining the posture information of the face according to the face information and determining the coordinate of the reference fixation point according to the face information; the first determining module is used for responding to the situation information being larger than a preset threshold value, and determining a correction parameter according to the situation information; the second determination module is used for determining gazing information according to the reference gazing point coordinate and the correction parameter;

the electronic equipment of another embodiment of the application comprises a processor, wherein the processor is used for determining the posture information of a human face according to human face information and determining the coordinates of a reference fixation point according to the human face information; determining a correction parameter according to the attitude information in response to the attitude information being greater than a preset threshold; determining gazing information according to the coordinates of the reference gazing point and the correction parameters; and controlling the electronic equipment according to the gazing information.

A non-transitory computer-readable storage medium embodying a computer program, which when executed by one or more processors, causes the processors to perform a gaze detection method or a control method of embodiments of the present application. The gazing detection method comprises the steps of determining posture information of a human face according to human face information, and determining a reference gazing point coordinate according to the human face information; determining a correction parameter according to the attitude information in response to the attitude information being greater than a preset threshold; and determining gazing information according to the coordinate of the reference gazing point and the correction parameters. The control method of the electronic equipment comprises the steps of determining posture information of a human face according to human face information, and determining a reference fixation point coordinate according to the human face information; determining a correction parameter according to the attitude information in response to the attitude information being greater than a preset threshold; determining gazing information according to the coordinates of the reference gazing point and the correction parameters; and controlling the electronic equipment according to the gazing information.

According to the control method, the control device and the electronic equipment of the electronic equipment, after the face information and the attitude information are obtained, when the attitude information is larger than a preset threshold value and the calculation accuracy of the fixation point coordinate is influenced, the reference fixation point coordinate is firstly calculated according to the face information, then the correction parameter is calculated according to the attitude information, and the reference fixation point coordinate can be corrected according to the correction parameter to obtain accurate fixation information, so that the influence of overlarge face shooting angle on fixation detection in the obtained face information is prevented, and the fixation detection accuracy can be improved. And when the electronic equipment is controlled according to the accurate gazing information, the control accuracy of the electronic equipment can be improved.

Additional aspects and advantages of the present application will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the present application.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative efforts.

FIG. 1 is a schematic flow diagram of a gaze detection method according to certain embodiments of the present application;

FIG. 2 is a block schematic diagram of a detection device according to certain embodiments of the present application;

FIG. 3 is a schematic plan view of an electronic device of some embodiments of the present application;

fig. 4 is a schematic connection diagram of an electronic device and a cloud server according to some embodiments of the present application;

fig. 5-7 are flow diagrams of gaze detection methods according to certain embodiments of the present application;

FIG. 8 is a schematic diagram of the structure of a detection model according to some embodiments of the present application;

FIG. 9 is a schematic flow chart diagram of a method of controlling an electronic device according to some embodiments of the present application;

FIG. 10 is a block schematic diagram of a control device according to certain embodiments of the present application;

FIGS. 11-14 are schematic diagrams of scenarios of control methods according to certain embodiments of the present application;

FIGS. 15 and 16 are schematic flow charts of control methods according to certain embodiments of the present application;

FIGS. 17 and 18 are schematic diagrams of a scenario of a control method according to some embodiments of the present application;

FIG. 19 is a schematic flow chart diagram of a control method according to certain embodiments of the present application; and

FIG. 20 is a schematic diagram of a connection between a processor and a computer-readable storage medium according to some embodiments of the present application.

Detailed Description

Embodiments of the present application will be further described below with reference to the accompanying drawings. The same or similar reference numbers in the drawings identify the same or similar elements or elements having the same or similar functionality throughout. In addition, the embodiments of the present application described below in conjunction with the accompanying drawings are exemplary and are only for the purpose of explaining the embodiments of the present application, and are not to be construed as limiting the present application.

Referring to fig. 1 to 3, a gaze detection method according to an embodiment of the present application includes the following steps:

011: determining the posture information of the face according to the face information, and determining the coordinates of the reference fixation point according to the face information;

013: determining a correction parameter according to the attitude information in response to the attitude information being greater than a preset threshold; and

015: and determining gazing information according to the coordinates of the reference gazing point and the correction parameters.

The detection device 10 of the embodiment of the present application includes a first determination module 11, a second determination module 12, and a third determination module 13. The first determining module 11 is configured to determine pose information of a face according to face information, and determine coordinates of a reference fixation point according to the face information; the second determining module 12 is configured to determine a correction parameter according to the attitude information in response to the attitude information being greater than a preset threshold; the third determining module 13 is configured to determine gazing information according to the coordinates of the reference gazing point and the correction parameters. That is, step 011 may be implemented by the first determination module 11, step 013 may be performed by the second determination module 12, and step 015 may be performed by the third determination module 13.

The electronic device 100 of the embodiment of the present application includes a processor 20 and an acquisition apparatus 30. The collecting device 30 is used for collecting face information (the face information may include face images, such as visible light images, infrared images, depth images, etc. of a face) at a predetermined frame rate; the collecting device 30 may be one or more of a visible light camera, an infrared camera, and a depth camera, wherein the visible light camera may collect a visible light face image, the infrared camera may collect an infrared face image, and the depth camera may collect a depth face image, in this embodiment, the collecting device 30 includes a visible light camera, an infrared camera, and a depth camera, and the collecting device 30 may simultaneously collect a visible light face image, an infrared face image, and a depth face image. The Processor 20 may include an Image Processor 20 (ISP), a Neural-Network Processing Unit 20 (NPU), and an Application Processor 20 (AP), the detecting device 10 is disposed in the electronic device 100, wherein the first determining module 11 may be disposed at the ISP and the NPU, the Processor 20 is connected to the collecting device 30, after the collecting device 30 collects the face Image, the ISP may process the face Image to obtain the pose information of the face, the NPU may determine the reference gazing point coordinate according to the face information, and the second determining module 12 and the third determining module 13 may be disposed at the NPU. The processor 20 (specifically, the ISP and the NPU) is configured to determine pose information of the face according to the face information; the processor 20 (which may be specifically the NPU) is further configured to determine a correction parameter based on the pose information and determine gaze information based on the fiducial gaze point coordinates and the correction parameter in response to the pose information being greater than a preset threshold. That is, step 011, step 013, and step 015 may be performed by processor 20.

The electronic device 100 may be a cell phone, smart watch, tablet computer, display device, laptop computer, teller machine, gate, head display device, game console, and the like. As shown in fig. 3, in the embodiment of the present application, the electronic device 100 is a mobile phone as an example, and it is understood that the specific form of the electronic device 100 is not limited to the mobile phone.

Specifically, when the user uses the electronic device 100, the collecting device 30 may collect the face information of the user once at intervals of a predetermined time, and continuously perform the gaze detection on the user under the condition that the power consumption of the electronic device 100 is small, or when the user uses an application program (such as browser software, sticker software, video software, and the like) which needs to perform the gaze detection, collect the face information according to a predetermined frame number (such as 10 frames per second), so that the face information collection is performed only when there is a gaze detection requirement, and the power consumption of the gaze detection is maximally reduced.

Referring to fig. 4, after obtaining face information (taking a face image as an example), the processor 20 may identify the face image, for example, the processor 20 may compare the face image with a preset face template, so as to determine a face in the face image and an image area where different parts (such as eyes and a nose) of the face are located, so as to obtain a face area image, where the preset face template may be stored in a memory of the electronic device 100, and the processor 20 may perform face identification in a Trusted Execution Environment (TEE) to ensure privacy of a user; or, the preset face template may be stored in the cloud server 200, and then the electronic device 100 sends the face image to the cloud server 200 for comparison to determine a face region image, and hands face recognition to the cloud server 200 for processing, so that the processing amount of the electronic device 100 can be reduced and the image processing efficiency can be improved; the processor 20 may then identify the face region image to determine pose information for the face. More specifically, the human face and the different parts of the human face can be identified according to the shape characteristics of the human face and the different parts of the human face, so as to obtain human face information, wherein the human face information can include images of different parts such as a human face area image and a human eye area image.

After the face information is obtained, the pose information of the face can be calculated according to the face information, the pose information can be obtained by extracting the features of the face image, calculating attitude information from the position coordinates of the extracted feature points, for example, using the tip of the nose, the centers of the left and right eyes, and the left and right corners of the mouth as feature points, the position coordinates of the feature points are changed continuously with the change of the pose of the face, for example, a three-dimensional coordinate system is established with the nose tip as the origin, the pitch angle, the horizontal rotation angle and the inclination angle of the face respectively represent the rotation angles of the face relative to three coordinate axes of the three-dimensional coordinate system, and the like), taking the horizontal rotation angle of the human face when facing the display screen 40 of the electronic device 100 as an example, the larger the deflection angle (i.e., horizontal rotation angle) of the human face, the closer the distance between the two feature points corresponding to the left and right eyes is, the more accurate the pose information of the face can be calculated from the position coordinates of the feature points.

It can be understood that different poses of the face all affect the gaze direction and gaze point coordinates of the user. Therefore, after the coordinates of the reference fixation point are calculated according to the face information, the correction parameters can be determined according to the posture information, so that the detection error of the fixation point caused by the posture change is corrected, and the fixation information obtained according to the coordinates of the reference fixation point and the correction parameters is relatively accurate.

When the coordinates of the reference fixation point are calculated according to the face information, the processor 20 can directly calculate the coordinates of the reference fixation point according to the face region image, or perform feature point identification on the face region image, and calculate the coordinates of the reference fixation point through the feature points, so that the calculation amount is small; or, the processor 20 may acquire the face region image and the eye region image, perform feature point identification on the face region image, and calculate the reference fixation point coordinate by combining the feature point of the face region image and the eye region image, so as to further improve the calculation accuracy of the reference fixation point coordinate on the basis of ensuring a small calculation amount.

Then, it can be understood that, when the posture of the user is directly opposite to the display screen 40 of the electronic device 100, the obtained face information is generally the most accurate, and therefore, the processor 20 may first determine whether the posture information is greater than a preset threshold, where the posture information includes a pitch angle, a roll angle, and a yaw angle of the face, and certainly, since a change in the roll angle (rotation of the face parallel to the display screen 40) does not cause a change in a position of a feature point of the face on the face, it may only determine whether the pitch angle or the yaw angle is greater than 0 degree.

If the posture of the user is directly opposite to the display screen 40 of the electronic device 100, the pitch angle, the roll angle and the yaw angle of the human face are all 0 degrees, and the preset threshold value is 0 degrees, it can be determined that the reference fixation point coordinate needs to be corrected when the posture information is greater than 0 degrees (for example, the pitch angle or the yaw angle is greater than 0 degrees), wherein the pitch angle, the roll angle and the yaw angle have directionality and may be negative values, which affects the accuracy of the determination, and therefore, when determining whether the posture information is greater than the preset threshold value, it can be determined whether the absolute value of the posture information is greater than the preset threshold value.

At this time, the processor 20 calculates a correction parameter according to the attitude information, where the correction parameter includes a coordinate correction coefficient if the reference gazing point coordinate only includes a two-dimensional coordinate of the line of sight on the display screen 40, and the gazing information can be obtained according to the reference gazing point coordinate and the coordinate correction coefficient, where the gazing information is (ax, by) if the reference gazing point coordinate is (x, y), and the coordinate correction coefficients are (a) and (b); or, when the reference gazing point coordinate includes a two-dimensional coordinate of the line of sight on the display screen 40 and a direction of the line of sight, the correction parameter includes a coordinate correction coefficient and a direction correction coefficient, and the gazing information can be obtained according to the reference gazing point coordinate and the coordinate correction coefficient, where if the reference gazing point coordinate is (x, y), the direction of the line of sight is (α, β, γ), the coordinate correction coefficients are a and b, and the direction correction coefficient includes c, d, and e, the gazing information is (ax, by, ca, d β, e γ).

When the posture information is less than or equal to the preset threshold, it is indicated that the user is facing the display screen 40 or the deflection angle of the user relative to the display screen 40 is small at this time, it can be determined that the reference fixation point coordinate does not need to be corrected, and the processor 20 directly determines that the reference fixation point coordinate is the final fixation information after calculating the reference fixation point coordinate, so that the calculation amount for calculating the correction parameter is saved.

Of course, in order to further reduce the calculation amount, the preset threshold may be set to be larger, for example, when the preset threshold is 5 degrees, the deflection of the face is smaller, and at this time, the detection accuracy of the gazing information is not substantially affected, and the calculation of the correction parameter may not be performed. Or, a preset threshold is set according to the requirement of the gazing information, if the gazing information only includes the gazing direction and does not need accurate coordinates of the gazing point, the preset threshold can be set to be larger at this time, and when the gazing information includes the coordinates of the gazing point on the display screen 40, the preset threshold can be set to be smaller, so that the detection accuracy of the gazing point is ensured.

After the gaze information is obtained, control of the electronic device 100 may be achieved based on the gaze information (gaze direction and/or gaze point coordinates). For example, when it is detected that the gazing point coordinate is located in the display area of the display screen 40, the screen is kept on all the time, and when it is detected that the gazing point coordinate is located outside the display area of the display screen 40 for a predetermined time period (e.g., 10S, 20S, etc.), the screen is turned off. Alternatively, an operation such as page turning is performed in accordance with the change of the gaze direction.

According to the gazing detection method, the detection device 10 and the electronic equipment 100, after the face information is obtained, the face pose is calculated through the face information, when the pose information is larger than a preset threshold value and the calculation accuracy of the gazing point coordinate can be influenced, the reference gazing point coordinate is firstly calculated according to the face information, and then the correction parameter is calculated according to the pose information, so that the reference gazing point coordinate can be corrected according to the correction parameter, the influence of overlarge face shooting angle on gazing detection in the obtained face information is prevented, and the accuracy of gazing detection can be improved.

Referring to fig. 2, fig. 3 and fig. 5, in some embodiments, the face information includes a face mask, a left-eye image and a right-eye image, the face mask is used to indicate the position of the face in the image, and step 011: calculating the coordinates of the reference fixation point according to the face information, comprising the following steps:

0111: calculating the position information of the face relative to the electronic equipment 100 according to the face mask;

0112: and calculating the reference fixation point coordinate according to the position information, the left eye image and the right eye image.

In some embodiments, the first determining module 11 is configured to calculate the position information of the face relative to the electronic device 100 according to the face mask; and calculating the reference fixation point coordinate according to the position information, the left eye image and the right eye image. That is, step 0111 and step 0112 may be performed by the first determination module 11.

In some embodiments, the processor 20 is further configured to calculate location information of a face relative to the electronic device 100 based on the face mask; and calculating the reference fixation point coordinate according to the position information, the left eye image and the right eye image. That is, step 0111 and step 0112 may be performed by processor 20.

Specifically, before the attention information is obtained through calculation, the processor 20 may further determine a face mask of the face image, where the face mask is used to characterize the position of the face in the face image, the face mask may be determined by identifying the position of the face in the face image, the processor 20 may calculate the position information of the face relative to the electronic device 100 according to the face mask (for example, the distance between the face and the electronic device 100 may be calculated according to the proportion of the face mask in the face image), and it may be understood that, when the distance between the face and the electronic device 100 changes, even if the gaze direction of the human eye is not changed, the gaze point coordinates of the human eye still change, and therefore, when calculating the fixation information, the fixation information can be calculated according to the human face image and/or the human eye image (such as the left eye image and the right eye image), and the fixation point coordinate can be calculated more accurately by combining the position information.

Referring again to fig. 2, 3 and 5, in some embodiments, the face information includes face feature points, the pose information includes pose angles and three-dimensional coordinate offsets, the correction parameters include rotation matrices and translation matrices, and step 011: determining the pose information of the face according to the face information, and further comprising:

0113: calculating a posture angle and three-dimensional coordinate offset according to the face characteristic points;

step 013: calculating correction parameters based on the pose information, including

0131: and calculating a rotation matrix according to the attitude angle, and calculating a translation matrix according to the three-dimensional coordinate offset.

In some embodiments, the first determining module 11 is further configured to calculate a pose angle and a three-dimensional coordinate offset from the face feature point; the second determination module 12 is further configured to calculate a rotation matrix according to the attitude angle, and calculate a translation matrix according to the three-dimensional coordinate offset. That is, step 0113 may be performed by the first determining module 11, and step 0131 may be performed by the second determining module 12.

In some embodiments, the processor 20 is further configured to calculate a pose angle and a three-dimensional coordinate offset from the face feature points; and calculating a rotation matrix according to the attitude angle, and calculating a translation matrix according to the three-dimensional coordinate offset. That is, step 0133 and step 0134 may be executed by processor 20.

Specifically, the correction parameters may include a rotation matrix and a translation matrix to represent the position change and the attitude change of the face, respectively, and when calculating the correction parameters, an attitude angle and a three-dimensional coordinate offset may be first calculated according to the feature points of the face, where the attitude angle is used to represent the attitude (such as a pitch angle, a roll angle, and a yaw angle) of the face, and the three-dimensional coordinate offset may represent the position of the face, and then the rotation matrix is calculated according to the attitude angle, and the offset matrix is calculated according to the three-dimensional coordinate offset, so as to determine the correction parameters of the reference fixation point coordinates, and the fixation information may be accurately calculated according to the reference fixation point coordinates, the rotation matrix, and the translation matrix.

Referring to fig. 2, 3 and 6, in some embodiments, the gaze detection method further includes:

0101: acquiring a training sample set, wherein the training sample set comprises a first type sample of which the posture information of the face is smaller than a preset threshold value and a second type sample of which the posture information of the face is larger than the preset threshold value;

0102: training a preset detection model according to the first type of sample and the second type of sample;

step 013 comprises:

0132: based on the detection model, correction parameters are determined according to the attitude information.

In certain embodiments, the detection device 10 further comprises an acquisition module 14 and a training module 15. The acquisition module 14 and the training module 15 may be both provided in the NPU for training of the detection model. The obtaining module 14 is configured to obtain a training sample set; the training module 16 is used for training a preset detection model according to the first type of sample and the second type of sample; the second determination module 12 is further configured to determine a correction parameter according to the pose information based on the detection model. That is, step 0101 may be performed by the obtaining module 14, step 0102 may be performed by the training module 15, and step 0132 may be performed by the second determining module 12.

In some embodiments, the processor 20 is further configured to obtain a training sample set; training a preset detection model according to the first type of sample and the second type of sample; based on the detection model, correction parameters are determined according to the attitude information. That is, step 0101, step 0102 and step 0132 may be executed by processor 20.

Specifically, the calculation of gazing information can be realized through a preset detection model, and in order to guarantee the accuracy of gazing information, the detection model needs to be trained first, so that the detection model converges.

During training, in order to enable the detection model to accurately calculate the gazing information when the face deflects relative to the display screen 40, a plurality of first samples of which the face pose information is smaller than a preset threshold value and a plurality of second samples of which the face pose information is larger than the preset threshold value can be selected in advance to serve as a training sample set; the first type of samples are face images of which the posture information is smaller than a preset threshold value; the second type of sample is a face image with the posture information larger than a preset threshold value; therefore, the detection model is trained through the first type of samples with the posture information smaller than the preset threshold value and the second type of samples with the posture information larger than the preset threshold value, and after the training is completed to convergence, the influence caused by deflection of the human face relative to the display screen 40 can be reduced to the maximum degree when the detection model detects the gazing information, and the accuracy of gazing detection can be guaranteed.

Referring to fig. 2, 3 and 7, in some embodiments, step 0102 includes:

01021: inputting the first type of sample into a fixation point detection module to output a first training coordinate;

01022: inputting the second type of sample into the fixation point detection module and the correction module to output a second training coordinate;

01023: calculating a first loss value according to a first preset coordinate and a first training coordinate corresponding to the first type of sample based on a preset loss function, and calculating a second loss value according to a second preset coordinate and a second training coordinate corresponding to the second type of sample;

01024: and adjusting the detection model according to the first loss value and the second loss value until the detection model converges.

In some embodiments, the training module 15 is further configured to input the first type of sample into the gaze point detection module to output a first training coordinate; inputting the second type of sample into the fixation point detection module and the correction module to output a second training coordinate; calculating a first loss value according to a first preset coordinate and a first training coordinate corresponding to the first type of sample based on a preset loss function, and calculating a second loss value according to a second preset coordinate and a second training coordinate corresponding to the second type of sample; and adjusting the detection model according to the first loss value and the second loss value until the detection model converges. That is, steps 01021 through 01024 may be performed by training module 15.

In some embodiments, the processor 20 is further configured to input the first type of sample to the point of regard detection module to output a first training coordinate; inputting the second type of sample into the fixation point detection module and the correction module to output a second training coordinate; calculating a first loss value according to a first preset coordinate and a first training coordinate corresponding to the first type of sample based on a preset loss function, and calculating a second loss value according to a second preset coordinate and a second training coordinate corresponding to the second type of sample; and adjusting the detection model according to the first loss value and the second loss value until the detection model converges. That is, steps 01021 to 01024 may be performed by processor 20.

Specifically, referring to fig. 8, the face detection model 50 includes a fixation point detection module 51 and a correction module 52. When training, inputting a training sample set to the detection model, wherein a first type of sample is input to the fixation point detection module 51 to output a first training coordinate; because the posture information of the first type training sample is smaller than a preset threshold value, a first training coordinate is directly output; the second type of training samples are input to the gaze point detection module 51 and the calibration module 52 at the same time, the gaze point detection module 51 outputs the reference training coordinates, and then the calibration module 52 outputs the calibration parameters and calibrates the reference training coordinates according to the calibration parameters to output the second training coordinates.

It can be understood that each training sample has a corresponding preset coordinate, and the preset coordinate represents actual gazing information of the training sample, wherein the first type of training sample corresponds to a first preset coordinate, and the second type of training sample corresponds to a second preset coordinate, so that the processor 20 can calculate a first loss value based on a preset loss function, the first training coordinate and the first preset coordinate; then, the processor 20 adjusts the gaze point detection module 51 based on the first loss value, so that the first training coordinate output by the gaze point detection module 51 gradually approaches the first preset coordinate until convergence; the processor 20 may calculate a second loss value based on the preset loss function, the second training coordinate and the second preset coordinate; the processor 20 then simultaneously adjusts the gaze point detection module 51 and the correction module 52 based on the second loss value such that the second training coordinates output by the detection model gradually approach the second preset coordinates until convergence.

For example, the loss function is as follows:

wherein, loss is a loss value, N is the number of training samples contained in each training sample set, X and Y are training coordinates (such as a first training coordinate or a second training coordinate), Gx and Gy are preset coordinates (such as a first preset coordinate and a second preset coordinate), when the training coordinates are gazing directions, X and Y respectively represent a pitch angle and a yaw angle, and when the training coordinates are gazing point coordinates, X and Y respectively represent that a gazing point is at the gazing pointAnd displaying the coordinates of the plane where the screen 40 is located, so as to quickly calculate the first loss value and the second loss value.

Then, the processor 20 may adjust the detection model according to the first loss value and the second loss value, so that the gradient of the detection model is continuously decreased, the training coordinate is closer to the preset coordinate, and finally the detection model is trained to converge. For example, in N consecutive batches of samples for training, when a first difference of first loss values corresponding to any two first samples and a second difference of second loss values corresponding to any two second samples are both smaller than a predetermined difference threshold, determining that the detection model converges, where N is a positive integer greater than 1; that is, in the training process of N consecutive batches, the first loss value is not substantially changed, which indicates that the first loss value and the second loss value reach the limit, and it is determined that the detection model has converged; or when the first loss value and the second loss value are both smaller than the preset loss threshold value, determining that the detection model is converged, and when the first loss value and the second loss value are smaller than the preset loss threshold value, indicating that the training coordinates are close to the preset coordinates, determining that the detection model is converged.

Therefore, the detection model is trained to be convergent through the first class training sample and the second class training sample, so that the detection model can still output accurate gazing information according to the face information when the face deflects.

Referring to fig. 3, 9 and 10, a method for controlling an electronic device 100 according to an embodiment of the present disclosure includes the following steps:

021: determining the posture information of the face according to the face information, and determining the coordinates of the reference fixation point according to the face information;

023: determining a correction parameter according to the attitude information in response to the attitude information being greater than a preset threshold;

025: determining gazing information according to the coordinates of the reference gazing point and the correction parameters; and

027: the electronic device 100 is controlled according to the gaze information.

The control device 20 of the embodiment of the present application includes an acquisition module 21, a first determination module 22, a second determination module 23, and a control module 24. The acquisition module 21 is configured to determine pose information of a face according to the face information, and determine coordinates of a reference fixation point according to the face information; the first determining module 22 is configured to determine a correction parameter according to the attitude information in response to the attitude information being greater than a preset threshold; the second determining module 23 is configured to determine gazing information according to the coordinates of the reference gazing point and the correction parameters; the control module 24 is configured to control the electronic device 100 according to the gaze information. That is, step 021 may be performed by the acquisition module 21, step 023 may be performed by the first determination module 22, step 025 may be performed by the second determination module 23, and step 027 may be performed by the control module 24.

The electronic device 100 of the embodiment of the present application includes a processor 20 and an acquisition apparatus 30. The collecting device 30 is used for collecting face information (the face information includes face images, such as visible light images, infrared images, depth images, etc. of a face) at a predetermined frame rate; the collecting device 30 may be one or more of a visible light camera, an infrared camera, and a depth camera, wherein the visible light camera may collect a visible light face image, the infrared camera may collect an infrared face image, and the depth camera may collect a depth face image, in this embodiment, the collecting device 30 includes a visible light camera, an infrared camera, and a depth camera, and the collecting device 30 may simultaneously collect a visible light face image, an infrared face image, and a depth face image. The processor 20 may include an ISP, an NPU, and an AP, for example, the control device 20 is disposed in the electronic device 100, the obtaining module 21 is disposed in the ISP and the NPU, the processor 20 is connected to the collecting device 30, after the collecting device 30 collects the face image, the ISP may process the face image to determine pose information of the face according to the face information, the NPU may determine the reference gazing point coordinate according to the face information, the first determining module 22 and the second determining module 23 may be disposed in the NPU, and the control module 24 may be disposed in the AP. The processor 20 (specifically, the ISP and the NPU) is configured to obtain face information and pose information; the processor 20 (specifically, the NPU) is further configured to determine a correction parameter according to the attitude information in response to the attitude information being greater than a preset threshold; determining gazing information according to the coordinates of the reference gazing point and the correction parameters; the processor 20, which may specifically be an AP, may also be used to control the electronic device 100 according to the gaze information. That is, step 021 may be executed by acquisition device 30 in cooperation with processor 20, and steps 023, 025, and 027 may be executed by processor 20.

Specifically, please refer to the descriptions of step 011, step 013, and step 015 respectively for the manner of determining the gazing information, i.e., step 021, step 023, and step 025, which is not described herein again.

After the gazing information (such as gazing direction and gazing point coordinates) is obtained, the control of the electronic device 100 can be realized according to the gazing direction and the gazing point coordinates. Referring to fig. 11, for example, a three-dimensional coordinate system is established with a midpoint of the two eyes as an origin O1, an X1 axis is parallel to a line connecting centers of the two eyes, a Y1 axis is located on a horizontal plane and is perpendicular to the X1 axis, a Z1 axis is perpendicular to the X1 axis and the Y1 axis, a gaze direction of the user is represented by rotation angles of a user' S sight line S and three axes of the three-dimensional coordinate system, such as a gaze direction including a pitch angle, a roll angle and a yaw angle, respectively, the pitch angle represents a rotation angle around the X1 axis, the roll angle represents a rotation angle around the Y1 axis, the yaw angle represents a rotation angle around the Z1 axis, the processor 20 may implement a page-turning or sliding operation on display contents of the electronic device 100 according to the gaze direction, for example, a change of the gaze direction may be determined according to the gaze direction of a plurality of consecutive frames of human eye region images (such as consecutive 10 frames), for example, referring to fig. 11 and fig. 12, when the pitch angle is gradually increased (i.e.g., it may be determined that the user wants the display content to slide or page down, and for example, please refer to fig. 11 and 13, the pitch angle gradually decreases (i.e., the line of sight S bows), it may be determined that the user wants the display content to slide or page down. Similarly, by detecting the moving direction of the gaze point M, a sliding or page-turning operation may also be performed on the electronic device 100. Referring to fig. 14, a plane coordinate system may be established with the center of the display screen 40 as a coordinate origin O2, the width direction of the parallel electronic device 100 is taken as an X2 axis, the length direction of the parallel electronic device 100 is taken as a Y2 axis, the gaze point coordinate includes an abscissa (corresponding to the position of the X2 axis) and an ordinate (corresponding to the position of the Y2 axis), if the ordinate is gradually increased, the gaze point M is moved upwards, it may be determined that the user wants the display content to slide upwards or turn down, and for example, if the ordinate is gradually decreased, the gaze point M is moved downwards, it may be determined that the user wants the display content to slide downwards or turn up.

In other embodiments, the processor 20 may also display more new display content after the sliding according to the change speed of the gaze direction (as determined by the difference between the pitch angles of the 1 st and 10 th frames (or the difference between the ordinate of the gaze point M) and the duration of acquiring the consecutive 10 frames).

In another example, when it is detected that the gazing point coordinate is located in the display area of the display screen 40, it indicates that the user is always viewing the display screen 40, and the screen is kept always on, and when it is detected that the gazing point coordinate is located outside the display area of the display screen 40, it indicates that the user is not viewing the display screen 40, but in order to prevent the user from accidentally looking outside the display area, which may cause erroneous determination, the screen may be closed after the user does not view the display screen 40 for a predetermined time (e.g., 10S, 20S, etc.).

According to the control method of the electronic device 100, the control device 20 and the electronic device 100, after the face information and the posture information are obtained, when the posture information is larger than a preset threshold value and the calculation accuracy of the fixation point coordinate is affected, the reference fixation point coordinate is firstly calculated according to the face information, then the correction parameter is calculated according to the posture information, and the reference fixation point coordinate can be corrected according to the correction parameter to obtain accurate fixation information, so that the influence of too large face shooting angle on fixation detection in the obtained face information is prevented, and the fixation detection accuracy can be improved. And the control accuracy of the electronic apparatus 100 can be improved when the electronic apparatus 100 is controlled based on the accurate gazing information.

Referring to fig. 3, 10 and 15, in some embodiments, the face information includes a face mask, a left-eye image and a right-eye image, the face mask is used to indicate the position of the face in the image, and step 021: calculating the coordinates of the reference fixation point according to the face information, comprising the following steps:

0211: calculating the position information of the face relative to the electronic equipment 100 according to the face mask;

0212: and calculating the reference fixation point coordinate according to the position information, the left eye image and the right eye image.

In some embodiments, the first determining module 22 is further configured to calculate location information of a face relative to the electronic device 100 according to the face mask; and calculating the reference fixation point coordinate according to the position information, the left eye image and the right eye image. That is, step 0211 and step 0212 may be performed by the first determining module 22.

In some embodiments, the processor 20 is further configured to calculate location information of a face relative to the electronic device 100 based on the face mask; and calculating the reference fixation point coordinate according to the position information, the left eye image and the right eye image. That is, step 0211 and step 0212 may be executed by processor 20.

Specifically, the detailed descriptions of step 0231 and step 0232 refer to step 0131 and step 0132, respectively, and are not repeated here.

Referring to fig. 3, 10 and 15, in some embodiments, the face information includes face feature points, the pose information includes pose angles and three-dimensional coordinate offsets, the correction parameters include rotation matrices and translation matrices, and the step 021: determining pose information of the face according to the face information, comprising:

0213: calculating a posture angle and three-dimensional coordinate offset according to the face characteristic points;

step 023 includes:

0231: and calculating a rotation matrix according to the attitude angle, and calculating a translation matrix according to the three-dimensional coordinate offset.

In some embodiments, the first determining module 22 is further configured to calculate a pose angle and a three-dimensional coordinate offset from the face feature points; and calculating a rotation matrix according to the attitude angle, and calculating a translation matrix according to the three-dimensional coordinate offset. That is, steps 0233 and 0234 may be performed by the first determining module 22.

In some embodiments, the processor 20 is further configured to calculate a pose angle and a three-dimensional coordinate offset from the face feature points; and calculating a rotation matrix according to the attitude angle, and calculating a translation matrix according to the three-dimensional coordinate offset. That is, steps 0233 and 0234 may be executed by processor 20.

Specifically, the detailed descriptions of step 0233 and step 0234 refer to step 0133 and step 0134, respectively, and are not repeated here.

Referring to fig. 3, 10 and 16, in some embodiments, the gaze information includes gaze point coordinates, and before step 021, the control method further includes:

0201: acquiring a shot image within a first preset time before screen turning;

0202: responding to the shot image containing the human face;

step 027: controlling the electronic device 100 according to the gaze information, comprising:

0271: the screen continues to be lit for a second predetermined length of time in response to the gaze point coordinates being located in the display area of the display screen 40.

In some embodiments, the control module 24 is further configured to obtain the captured image within a first predetermined time period before the screen is turned off; responding to the shot image containing the human face; the screen is continuously lit for a second predetermined length of time in response to when the gaze point coordinates are located in the display area of the display screen 40. That is, step 0201, step 0202, and step 0271 can be performed by control module 24.

In some embodiments, the processor 20 is further configured to obtain the captured image within a first predetermined time period before the screen is turned off; in response to the captured image including a human face, the screen is continuously lit for a second predetermined length of time in response to the gaze point coordinates being located in the display area of the display screen 40. That is, step 0201, step 0202, and step 0271 can be performed by processor 20.

Specifically, the gazing information may be used to implement the screen-off control, before the screen-off control is performed, the gazing detection is performed first, for example, the processor 20 obtains the shot image first, and if a human face exists in the shot image, the gazing information is determined according to the shot image, and certainly, in order to ensure that there is enough time to obtain the shot image and calculate the gazing information before the screen-off control, the shot image needs to be obtained within a first predetermined time period (e.g., 5 seconds, 10 seconds, etc.) before the screen-off control is performed.

Referring to fig. 17 and 18, when the gaze point M is located in the display area of the display screen 40, it may be determined that the user is gazing at the display screen 40, so that the display screen 40 is continuously bright for a second predetermined time period, where the second predetermined time period may be greater than the first predetermined time period, and the captured image is obtained again within the first predetermined time period before the user turns off the screen again, so that the user keeps bright when gazing at the display screen 40, and turns off the screen again when the user no longer gazes at the display screen 40.

Referring to fig. 17 again, a two-dimensional coordinate system parallel to the display screen 40 may be established by using the center of the display area as the origin O2, the display area is associated with a preset coordinate range, that is, the display area is in the abscissa range and the ordinate range of the two-dimensional coordinate system, as the preset coordinate range, and when the gazing point coordinate is located in the preset coordinate range (that is, the abscissa of the gazing point coordinate is located in the abscissa range and the ordinate is located in the ordinate range), it may be determined that the gazing point coordinate is located in the display area, so as to more easily determine whether the user gazes at the display screen 40.

And because the acquisition of the shot image and the calculation of the watching information are only carried out within the first preset time before the screen is turned off, the power consumption is saved.

Referring to fig. 3, 10 and 19, in some embodiments, the gaze information includes gaze point coordinates, and before step 021, the control method further includes:

0203: acquiring a photographed image in response to a case where the electronic apparatus 100 does not receive an input operation;

step 027 comprises:

0272: in response to the shot image containing the face and the fixation point coordinate being located in the display area, adjusting the display brightness of the display screen 40 to a first predetermined brightness;

0273: and adjusting the display brightness to a second preset brightness in response to the fact that the shot image does not contain the human face or the shot image contains the human face and the fixation point coordinate is located outside the display area, wherein the second preset brightness is smaller than the first preset brightness.

In some embodiments, the control module 24 is also used to acquire the captured image; in response to a situation in which the electronic apparatus 100 does not receive an input operation; in response to the shot image containing the face and the fixation point coordinate being located in the display area, adjusting the display brightness of the display screen 40 to a first predetermined brightness; and adjusting the display brightness to a second preset brightness in response to the fact that the shot image does not contain the human face or the shot image contains the human face and the fixation point coordinate is located outside the display area, wherein the second preset brightness is smaller than the first preset brightness. That is, step 0203, step 0204, step 0272 and step 0273 may be performed by control module 24.

In some embodiments, the processor 20 is also used to acquire the captured image; in response to a situation in which the electronic apparatus 100 does not receive an input operation; in response to the shot image containing the face and the fixation point coordinate being located in the display area, adjusting the display brightness of the display screen 40 to a first predetermined brightness; and adjusting the display brightness to a second preset brightness in response to the fact that the shot image does not contain the human face or the shot image contains the human face and the fixation point coordinate is located outside the display area, wherein the second preset brightness is smaller than the first preset brightness. That is, step 0203, step 0204, step 0272 and step 0273 can be performed by processor 20.

Specifically, referring to fig. 17 and 18 again, the gazing information may also be used to realize intelligent screen brightening, and in order to save power, the electronic device 100 generally reduces the display brightness after the screen is brightened for a certain period of time, and then turns off the screen after the screen is brightened for a certain period of time at low brightness. In this embodiment, in a case where the electronic device 100 does not receive the input operation of the user, and it may be determined that the user may not use the electronic device 100 or only view the display content at this time, the processor 20 may obtain the captured image, and if the captured image includes a human face, the gaze information is calculated from the captured image, and if the gaze point coordinates are located in the display area, it indicates that the user has not operated the electronic apparatus 100, but adjusts the display brightness to a first predetermined brightness when viewing the display content, which may be a brightness custom set by the user when the display screen 40 is normally displayed, or changes in real time according to the brightness of the ambient light to adapt to the brightness of the ambient light, thereby ensuring that the user can turn on the screen even if the user does not operate the electronic device 100, to prevent a situation where the user experience is affected by suddenly turning off the screen while the user is not operating the electronic device 100 but viewing the display content.

However, under the condition that the electronic device 100 does not receive the input operation, if the photographed image does not include a human face, or the photographed image includes a human face, and the gaze point coordinate is outside the display area (i.e., the user does not view the display area), it may be determined that the user does not need to use the electronic device 100 currently, and therefore, the display brightness may be adjusted to the second predetermined brightness, which is smaller than the first predetermined brightness, at this time, thereby preventing unnecessary power consumption. When the user watches the display area again, the display brightness is adjusted to the first preset brightness, and the normal watching experience of the user is guaranteed. Therefore, under the condition that the user does not operate the electronic equipment 100, the user can watch the display area, the display area is displayed at normal brightness, the user does not watch the display area, the display area is displayed at low brightness, and on the basis of ensuring the watching experience of the user, the electric quantity is saved to the maximum extent.

Referring to fig. 20, one or more non-transitory computer-readable storage media 300 containing a computer program 302 according to an embodiment of the present disclosure may enable a processor 20 to execute a gaze detection method or a control method of the electronic device 100 according to any of the above embodiments when the computer program 302 is executed by one or more processors 20.

For example, referring to fig. 1, the computer program 302, when executed by the one or more processors 20, causes the processors 20 to perform the steps of:

For another example, referring to fig. 9, when the computer program 302 is executed by the one or more processors 20, the processors 20 may further perform the steps of:

027: the electronic device 100 is controlled according to the gaze information.

In the description herein, references to the description of the terms "one embodiment," "some embodiments," "an illustrative embodiment," "an example," "a specific example" or "some examples" or the like mean that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the present application. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, the various embodiments or examples and features of the various embodiments or examples described in this specification can be combined and combined by those skilled in the art without contradiction.

Any process or method descriptions in flow charts or otherwise described herein may be understood as representing modules, segments, or portions of code which include one or more executable instructions for implementing specific logical functions or steps of the process, and the scope of the preferred embodiments of the present application includes other implementations in which functions may be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved, as would be understood by those reasonably skilled in the art of the embodiments of the present application.

Although embodiments of the present application have been shown and described above, it is to be understood that the above embodiments are exemplary and not to be construed as limiting the present application, and that changes, modifications, substitutions and alterations can be made to the above embodiments by those of ordinary skill in the art within the scope of the present application.

Claims

1. A gaze detection method, comprising:

determining the posture information of a human face according to the human face information, and determining the coordinates of a reference fixation point according to the human face information;

determining a correction parameter according to the attitude information in response to the attitude information being greater than a preset threshold; and

and determining gazing information according to the coordinate of the reference gazing point and the correction parameters.

2. The gaze detection method of claim 1, further comprising:

and responding to the gesture information being smaller than the preset threshold value, calculating the coordinate of the reference fixation point according to the face information to serve as the fixation information.

3. The gaze detection method of claim 1, wherein the pose information comprises pose angles including a pitch angle and a yaw angle, and wherein determining whether the pose information of the face is greater than a predetermined threshold based on the face information comprises:

and judging whether the pitch angle or the yaw angle is larger than the preset threshold value or not according to the face information.

4. The gaze detection method of claim 1, further comprising:

acquiring a training sample set, wherein the training sample set comprises a first type sample of which the posture information of the face is smaller than the preset threshold value and a second type sample of which the posture information of the face is larger than the preset threshold value;

training a preset detection model according to the first type of sample and the second type of sample;

the determining a correction parameter according to the attitude information includes:

determining the correction parameters according to the attitude information based on the detection model.

5. The gaze detection method of claim 4, wherein the detection model comprises a gaze point detection module and a correction module, the training of the detection model from the first type of samples and the second type of samples comprising:

inputting a first type of sample into the fixation point detection module to output a first training coordinate;

inputting a second type of sample into the fixation point detection module and the correction module to output a second training coordinate;

based on a preset loss function, calculating a first loss value according to a first preset coordinate and the first training coordinate corresponding to the first type of sample, and calculating a second loss value according to a second preset coordinate and the second training coordinate corresponding to the second type of sample;

and adjusting the detection model according to the first loss value and the second loss value until the detection model converges.

6. The gaze detection method of claim 5, wherein the detection model is determined to converge when a first difference between the first loss values corresponding to any two first samples and a second difference between the second loss values corresponding to any two second samples in N consecutive batches of samples under training are both smaller than a predetermined difference threshold, wherein N is a positive integer greater than 1; or when the first loss value and the second loss value are both smaller than a preset loss threshold value, determining that the detection model converges.

7. A gaze detection method according to claim 1, wherein the face information comprises a face mask indicating the position of a face in an image, a left eye image and a right eye image, the calculating of the reference gaze point coordinates from the face information comprises:

calculating the position information of the face relative to the electronic equipment according to the face mask;

and calculating the coordinates of the reference fixation point according to the position information, the left eye image and the right eye image.

8. The gaze detection method of claim 1, wherein the face information comprises face feature points, the pose information comprises pose angles and three-dimensional coordinate offsets, the correction parameters comprise rotation matrices and translation matrices,

the determining the pose information of the face according to the face information comprises the following steps:

calculating the attitude angle and the three-dimensional coordinate offset according to the face feature point;

the calculating of the correction parameters according to the attitude information comprises:

and calculating the rotation matrix according to the attitude angle, and calculating the translation matrix according to the three-dimensional coordinate offset.

9. A method of controlling an electronic device, comprising:

determining a correction parameter according to the attitude information in response to the attitude information being greater than a preset threshold;

determining gazing information according to the coordinates of the reference gazing point and the correction parameters; and

and controlling the electronic equipment according to the gazing information.

10. The control method according to claim 9, characterized by further comprising:

11. The control method according to claim 9, wherein the face information includes a face mask, a left-eye image, and a right-eye image, the face mask indicating a position of a face in an image, and the calculating the reference gazing point coordinate based on the face information includes:

12. The control method according to claim 9, wherein the face information includes face feature points, the pose information includes a pose angle and a three-dimensional coordinate offset, the correction parameters include a rotation matrix and a translation matrix, and the calculating the correction parameters from the pose information includes:

13. The control method according to claim 9, wherein the gaze information includes gaze point coordinates,

before determining pose information of a human face according to human face information and determining a reference fixation point coordinate according to the human face information, the control method further comprises the following steps:

acquiring a shot image within a first preset time before screen turning;

responding to the shot image containing face information;

the controlling the electronic device according to the gazing information comprises:

and responding to the fixation point coordinate positioned in the display area of the display screen, and continuously lightening the screen for a second preset time length.

14. The control method according to claim 13, wherein the display area is associated with a preset coordinate range, the control method further comprising:

and when the fixation point coordinate is located in the preset coordinate range, determining that the fixation point coordinate is located in the display area.

15. The control method according to claim 9, wherein before determining pose information of a face from face information from which the reference gaze point coordinates are determined, the control method further comprises:

acquiring a shot image in response to the situation that the electronic equipment does not receive input operation;

responding to the fact that the shot image contains a human face and the fixation point coordinate is located in the display area, and adjusting the display brightness of the display screen to a first preset brightness;

and adjusting the display brightness to a second preset brightness in response to the fact that the shot image does not contain the human face or the shot image contains the human face and the fixation point coordinate is located outside the display area, wherein the second preset brightness is smaller than the first preset brightness.

16. A detection device, comprising:

the first determining module is used for determining the posture information of the face according to the face information and determining the coordinate of the reference fixation point according to the face information;

the second determination module is used for responding to the situation that the posture information is larger than a preset threshold value, and determining correction parameters according to the posture information;

and the third determining module is used for determining gazing information according to the reference gazing point coordinate and the correction parameter.

17. A control device, comprising:

the acquisition module is used for determining the posture information of the face according to the face information and determining the coordinate of the reference fixation point according to the face information;

the first determining module is used for responding to the situation that the posture information is larger than a preset threshold value, and determining a correction parameter according to the posture information;

the second determination module is used for determining gazing information according to the reference gazing point coordinate and the correction parameter;

and the control module is used for controlling the electronic equipment according to the gazing information.

18. An electronic device is characterized by comprising a processor, wherein the processor is used for determining the posture information of a human face according to human face information and determining a reference fixation point coordinate according to the human face information; determining a correction parameter according to the attitude information in response to the attitude information being greater than a preset threshold; and determining gazing information according to the coordinate of the reference gazing point and the correction parameters.

19. An electronic device is characterized by comprising a processor, wherein the processor is used for determining the posture information of a human face according to human face information and determining a reference fixation point coordinate according to the human face information; determining a correction parameter according to the attitude information in response to the attitude information being greater than a preset threshold; determining gazing information according to the coordinates of the reference gazing point and the correction parameters; and controlling the electronic equipment according to the gazing information.

20. A non-transitory computer-readable storage medium comprising a computer program which, when executed by a processor, causes the processor to perform the gaze detection method of any one of claims 1-8, or the control method of the electronic device of any one of claims 9-15.