WO2023071882A1 - Procédé de détection de regard humain, procédé de commande et dispositif associé - Google Patents

Procédé de détection de regard humain, procédé de commande et dispositif associé Download PDF

Info

Publication number
WO2023071882A1
WO2023071882A1 PCT/CN2022/126122 CN2022126122W WO2023071882A1 WO 2023071882 A1 WO2023071882 A1 WO 2023071882A1 CN 2022126122 W CN2022126122 W CN 2022126122W WO 2023071882 A1 WO2023071882 A1 WO 2023071882A1
Authority
WO
WIPO (PCT)
Prior art keywords
human eye
image
human
information
gaze
Prior art date
Application number
PCT/CN2022/126122
Other languages
English (en)
Chinese (zh)
Inventor
龚章泉
Original Assignee
Oppo广东移动通信有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Oppo广东移动通信有限公司 filed Critical Oppo广东移动通信有限公司
Publication of WO2023071882A1 publication Critical patent/WO2023071882A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting

Definitions

  • the present application relates to the technical field of consumer electronics products, and in particular to a human eye gaze detection method, a method for controlling electronic equipment through human eyes, a detection device, a control device, electronic equipment, and a non-volatile computer-readable storage medium.
  • electronic devices can estimate a user's gaze point by collecting face images.
  • the present application provides a human eye gaze detection method, a method for controlling electronic equipment through human eyes, a detection device, a control device, electronic equipment and a non-volatile computer-readable storage medium.
  • the human eye gaze detection method in one embodiment of the present application includes acquiring an image of the human eye area; determining the weight of the image of the human eye area according to the missing information of the image of the human eye area; The weight of the eye area image determines the gaze information of the human eye.
  • a detection device includes a first acquisition module, a first determination module, and a second determination module.
  • the first acquisition module is used to acquire the human eye area image;
  • the first determination module is used to determine the weight of the human eye area image according to the missing information of the human eye area image;
  • the second determination module uses The human eye gaze information is determined according to the human eye area image and the weight of the human eye area image.
  • An electronic device includes a processor, and the processor is configured to execute a human gaze detection method.
  • the human eye gaze detection method includes acquiring a human eye area image; determining the weight of the human eye area image according to missing information of the human eye area image; Weight, which determines the gaze information of the human eye.
  • the human eye gaze detection method includes acquiring a human face image; determining the human eye area image and human face pose information according to the human face image; The weight of the eye area image; according to the human eye area image and the weight of the human eye area image, determine the human eye feature information; according to the human eye feature information and the human face posture information, determine the human eye gaze information.
  • the detection device in another embodiment of the present application includes a first acquisition module, a first determination module, a second determination module, a third determination module and a fourth determination module.
  • the first acquiring module is used to acquire a human face image;
  • the first determining module is used to determine the human eye region image and facial posture information according to the human face image;
  • the second determining module is used to determine the human face image according to the human face
  • the missing information of the eye area image determines the weight of the human eye area image;
  • the third determining module is used to determine the human eye feature information according to the human eye area image and the weight of the human eye area image;
  • the fourth determination module is configured to determine human eye gaze information according to the human eye feature information and the human facial posture information.
  • An electronic device includes a processor, and the processor is configured to execute a human gaze detection method.
  • the human eye gaze detection method includes acquiring a human face image; determining the human eye area image and human face posture information according to the human face image; determining the weight of the human eye area image according to the missing information of the human eye area image ; determining human eye feature information according to the human eye area image and the weight of the human eye area image; determining human eye gaze information according to the human eye feature information and the human face posture information.
  • the method for controlling an electronic device through human eyes includes acquiring an image of a human eye area; determining human eye gaze information according to the weight of the human eye area image and the human eye area image, wherein the human eye The weight of the area image is determined according to the missing information of the human eye area image; and the electronic device is controlled according to the human eye gaze information.
  • the control device in the embodiment of the present application includes a first acquisition module, a first determination module and a control module.
  • the first acquiring module is used to acquire an image of the human eye area;
  • the first determination module is used to determine the gaze information of the human eye according to the weight of the image of the human eye area and the image of the human eye area, wherein the human eye
  • the weight of the eye area image is determined according to the missing information of the human eye area image;
  • the control module is used to control the electronic device according to the human eye gaze information.
  • An electronic device in still another embodiment of the present application includes a processor, and the processor is configured to execute a method for controlling the electronic device through human eyes.
  • the method for controlling an electronic device through human eyes includes acquiring an image of a human eye region; determining human eye gaze information according to the weight of the human eye region image and the human eye region image, wherein the weight of the human eye region image determining according to the missing information of the human eye area image; and controlling the electronic device according to the human eye gaze information.
  • the processors execute the human eye gaze detection method or pass the human eye gaze detection method.
  • the human eye gaze detection method includes acquiring a human eye area image; determining the weight of the human eye area image according to missing information of the human eye area image; Weight, which determines the gaze information of the human eye.
  • the method for controlling an electronic device with human eyes includes acquiring an image of a human eye region; determining human eye gaze information according to the weight of the human eye region image and the human eye region image, wherein the weight of the human eye region image is based on Determining missing information of the human eye area image; and controlling the electronic device according to the human eye gaze information.
  • FIG. 1 is a schematic flow diagram of a human eye gaze detection method in some embodiments of the present application
  • FIG. 2 is a block diagram of a detection device in some embodiments of the present application.
  • FIG. 3 is a schematic plan view of an electronic device in some embodiments of the present application.
  • Fig. 4 is a schematic diagram of connection between an electronic device and a cloud server in some embodiments of the present application
  • 5 to 8 are schematic flowcharts of the human eye gaze detection method in some embodiments of the present application.
  • FIG. 9 is a schematic structural diagram of a human eye detection model in some embodiments of the present application.
  • Fig. 10 is a schematic flow chart of a human eye gaze detection method in some embodiments of the present application.
  • Figure 11 is a block diagram of a detection device in some embodiments of the present application.
  • Fig. 12 is a schematic flow chart of a human eye gaze detection method in some embodiments of the present application.
  • Fig. 13 is a schematic flowchart of a method for controlling an electronic device through human eyes in some embodiments of the present application
  • Fig. 14 is a block diagram of a control device in some embodiments of the present application.
  • 15 to 18 are schematic diagrams of scenarios of a method for controlling an electronic device through human eyes in some embodiments of the present application.
  • 19 to 23 are schematic flowcharts of a method for controlling an electronic device through human eyes in some embodiments of the present application.
  • FIG. 24 and FIG. 25 are schematic diagrams of scenes of a method for controlling an electronic device through human eyes in some embodiments of the present application.
  • Fig. 26 is a schematic flowchart of a method for controlling an electronic device through human eyes in some embodiments of the present application.
  • Fig. 27 is a schematic flowchart of a training method of a human eye detection model in some embodiments of the present application.
  • Figure 28 is a schematic block diagram of a training device according to some embodiments of the present application.
  • Fig. 29 is a schematic diagram of connection between a processor and a computer-readable storage medium in some embodiments of the present application.
  • the human eye gaze detection method of the present application includes obtaining the human eye area image; determining the weight of the human eye area image according to the missing information of the human eye area image; determining the human eye gaze information according to the human eye area image and the weight of the human eye area image .
  • the human eye gaze detection method also includes: acquiring face posture information; determining the human eye gaze information according to the human eye area image and the weight of the human eye area image, including: according to the human face posture information, human eye The weight of the area image and the human eye area image determines the gaze information of the human eye.
  • the human eye area image includes a left eye area image and a right eye area image
  • determining the weight of the human eye area image according to the missing information of the human eye area image includes: according to the first missing information of the left eye area image The information determines the first weight of the image of the left eye area, and the second weight of the image of the right eye area is determined according to the second missing information of the image of the right eye area.
  • the human eye gaze detection method also includes: acquiring a face image, and the face image includes a face mask; calculating the position information of the face relative to the electronic device according to the face mask; according to the human eye area image and The weight of the human eye region image determines the human eye gaze information, including: determining the human eye gaze information according to the position information, the human eye region image, and the weight of the human eye region image.
  • the human eye gaze detection method also includes: obtaining a training sample set, the training sample set includes a plurality of human eye area images; determining the occlusion area and/or image offset parameters according to the human eye area images; Image, occlusion area, and image offset parameters to train the human eye detection model; according to the weight of the human eye area image and the human eye area image, determine the human eye gaze information, including: based on the human eye detection model, according to the human eye area image and the human eye area image The weight of the eye area image determines the gaze information of the human eye.
  • training the human eye detection model according to the human eye area image, the occlusion area, and the image offset parameters includes: inputting the human eye area image, the occlusion area, and the image offset parameters into the human eye detection model to output Training coordinates; based on the preset loss function, calculate the loss value according to the preset coordinates corresponding to the human eye area image and the training coordinates; adjust the human eye detection model according to the loss value until the human eye detection model converges.
  • the occlusion area is generated based on replacing at least a part of the pixels in the image of the human eye area with pixels of predetermined pixel values; the image offset parameter is generated based on image offset of the feature points of the human eye area in the image of the human eye area .
  • the human eye gaze detection method of the present application includes obtaining a human face image; determining the human eye area image and human face posture information according to the human face image; determining the weight of the human eye area image according to the missing information of the human eye area image; The weight of the image and the image of the human eye area determines the characteristic information of the human eye; according to the characteristic information of the human eye and the posture information of the human face, the gaze information of the human eye is determined.
  • the face image includes a face mask
  • determining the gaze information of the human eye according to the feature information of the human eye and the posture information of the human face includes: calculating the position information of the human face relative to the electronic device according to the face mask; According to position information, human eye feature information, and human face posture information, human eye gaze information is determined.
  • the method for controlling an electronic device through human eyes of the present application includes acquiring an image of a human eye region; determining human eye gaze information according to the weight of the human eye region image Determining the missing information of ; and controlling electronic equipment according to the gaze information of human eyes.
  • the method for controlling an electronic device through human eyes further includes: obtaining face posture information; determining human eye gaze information according to the human eye area image and the weight of the human eye area image, including: according to the human face posture information , the human eye area image, and the weight of the human eye area image to determine the gaze information of the human eye.
  • the human eye area image includes a left eye area image and a right eye area image
  • the first weight of the left eye area image is determined according to the first missing information of the left eye area image
  • the second weight of the right eye area image Determined according to the second missing information of the right-eye region image.
  • the method for controlling an electronic device through human eyes further includes: acquiring a face image, and the face image includes a face mask; calculating the position information of the face relative to the electronic device according to the face mask; The weight of the area image and the human eye area image determines the human eye gaze information, including: determining the human eye gaze information according to the position information, the human eye area image, and the weight of the human eye area image.
  • the method for controlling an electronic device through human eyes further includes: obtaining a training sample set, the training sample set includes a plurality of human eye area images; determining the occlusion area and/or image offset parameters according to the human eye area images; Human eye area image, occlusion area, and image offset parameters train the human eye detection model; according to the human eye area image and the weight of the human eye area image, determine the human eye gaze information, including: based on the human eye detection model, according to the human eye area The weight of the image and the image of the human eye area determines the gaze information of the human eye.
  • training the human eye detection model according to the human eye area image, the occlusion area, and the image offset parameters includes: inputting the human eye area image, the occlusion area, and the image offset parameters into the human eye detection model to output Training coordinates; based on the preset loss function, calculate the loss value according to the preset coordinates corresponding to the human eye area image and the training coordinates; adjust the human eye detection model according to the loss value until the human eye detection model converges.
  • the occlusion area is generated based on replacing at least a part of the pixels in the image of the human eye area with pixels of predetermined pixel values; the image offset parameter is generated based on image offset of the feature points of the human eye area in the image of the human eye area .
  • the gaze information of the human eye includes the coordinates of the gaze point.
  • the method of controlling the electronic device through the human eye further includes: acquiring the captured image within the first predetermined time period before the screen is off. ; in response to a human face being included in the photographed image; controlling the electronic device according to the gaze information of the human eyes, further comprising, in response to the gaze point coordinates being located in the display area of the display screen, continuing to brighten the screen for a second predetermined duration.
  • the display area is associated with a preset coordinate range
  • the method for controlling an electronic device through human eyes further includes: when the gaze point coordinates are within the preset coordinate range, determining that the gaze point coordinates are located in the display area.
  • the method for controlling the electronic device through the human eye further includes: acquiring the captured image in response to the fact that the electronic device does not receive an input operation; controlling the electronic device according to the gaze information of the human eye
  • the device includes: adjusting the display brightness of the display screen to a first predetermined brightness in response to the captured image containing a human face and the gaze point coordinates are located in the display area; in response to the captured image not containing a human face, or the captured image containing a human face and The gaze point coordinates are located outside the display area, and the display brightness is adjusted to a second predetermined brightness, and the second predetermined brightness is smaller than the first predetermined brightness.
  • the detection device of the present application includes a first acquisition module, a first determination module and a second determination module.
  • the first acquisition module is used to acquire the image of the human eye region;
  • the first determination module is used to determine the weight of the human eye region image according to the missing information of the human eye region image;
  • the second determination module is used to determine the weight of the human eye region image according to the human eye region image and
  • the weight of the image of the human eye area determines the gaze information of the human eye.
  • the detection device of the present application includes a first acquisition module, a first determination module, a second determination module, a third determination module and a fourth determination module.
  • the first acquisition module is used to acquire the image of the human eye region;
  • the first determination module is used to determine the image of the human eye region and the facial posture information according to the face image;
  • the second determination module is used to determine the missing information of the human eye region image , to determine the weight of the human eye area image;
  • the third determination module is used to determine the human eye feature information according to the human eye area image and the weight of the human eye area image;
  • the fourth determination module is used to determine the human eye feature information according to the human eye feature information and the face Posture information, to determine the gaze information of human eyes.
  • the control device of the present application includes a first acquisition module, a first determination module and a control module.
  • the first acquisition module is used to acquire the human eye area image;
  • the first determination module is used to determine the human eye gaze information according to the human eye area image and the weight of the human eye area image, wherein the weight of the human eye area image is based on the human eye area The missing information of the image is determined; and
  • the control module is used to control the electronic equipment according to the gaze information of human eyes.
  • the electronic device of the present application includes a processor, and the processor is configured to execute the human gaze detection method in any one of the foregoing implementation manners.
  • the electronic device of the present application includes a processor, and the processor is configured to execute the method for controlling an electronic device through human eyes in any one of the foregoing implementation manners.
  • the non-transitory computer-readable storage medium of the present application includes a computer program.
  • the processor executes the human eye gaze detection method of any of the above embodiments; or the method of controlling an electronic device through human eyes of any of the above embodiments.
  • the human eye fixation detection method of the embodiment of the present application comprises the following steps:
  • 015 According to the weight of the human eye area image and the human eye area image, determine the human eye gaze information.
  • the detection device 10 in the embodiment of the present application includes a first acquisition module 11 , a first determination module 12 and a second determination module 13 .
  • the first acquiring module 11 is used to acquire the human eye region image;
  • the first determining module 12 is used to determine the weight of the human eye region image according to the missing information of the human eye region image;
  • the second determining module 13 is used to determine the weight of the human eye region image according to the human eye region image and
  • the weight of the image of the human eye area determines the gaze information of the human eye. That is to say, step 011 can be implemented by the first acquisition module 11 , step 013 can be performed by the first determination module 12 and step 015 can be performed by the second determination module 13 .
  • the electronic device 100 in the embodiment of the present application includes a processor 60 and a collection device 30 .
  • Acquisition device 30 is used for collecting face information by predetermined frame rate (face information comprises people's face image, as the visible light image of people's face, infrared image, depth image etc.);
  • Acquisition device 30 can be visible light camera, infrared camera, depth camera One or more of them, wherein the visible light camera can collect visible light face images, the infrared camera can collect infrared face images, and the depth camera can collect depth face images.
  • the collection device 30 includes a visible light camera, an infrared face image camera and depth camera, the acquisition device 30 simultaneously visible light face image, infrared face image and depth face image.
  • the processor 60 may include an image processor (Image Signal Processor, ISP), a neural network processor (Neural-Network Processing Unit, NPU) and an application processor (Application Processor, AP), and the detection device 10 is arranged in the electronic device 100,
  • the first acquisition module 11 can be arranged in the ISP, and the processor 60 is connected to the acquisition device 30.
  • the ISP can process the face image to obtain the human eye area image
  • the first The determination module 12 can also be set at the ISP
  • the second determination module 13 can be set at the NPU.
  • the processor 60 (specifically, it can be an ISP) is used to determine the weight of the human eye region image according to the missing information of the human eye region image; the processor 60 (specifically, it can be an NPU) is also used to The weight of is used to determine the gaze information of human eyes. That is to say, step 011 can be executed by the collection device 30 in cooperation with the processor 60 , and steps 013 and 015 can be executed by the processor 60 .
  • the electronic device 100 may be a mobile phone, a smart watch, a tablet computer, a display device, a notebook computer, a teller machine, a gate, a head-mounted display device, a game machine, and the like. As shown in FIG. 3 , the embodiment of the present application is described by taking the electronic device 100 as a mobile phone as an example. It can be understood that the specific form of the electronic device 100 is not limited to the mobile phone.
  • the collection device 30 can collect the user's face information once at a predetermined time interval, and continue to perform gaze detection on the user while ensuring that the power consumption of the electronic device 100 is small, or, when the user When using applications that require gaze detection (such as browser software, post bar software, video software, etc.), collect face information according to a predetermined number of frames (such as 10 frames per second), so that human face information is only performed when there is a gaze detection requirement. Face information collection minimizes the power consumption of gaze detection.
  • applications that require gaze detection such as browser software, post bar software, video software, etc.
  • the processor 60 can identify the face image, for example, the processor 60 can compare the face image with the preset face template , so as to determine the image area where the face in the face image and different parts of the face (such as eyes, nose, etc.) are located, to identify the eye area in the face image, thereby obtaining the eye area image, wherein
  • the face template can be stored in the memory of the electronic device 100, and the processor 60 can perform face recognition in a trusted execution environment (Trusted Execution Environment, TEE) to ensure the privacy of the user; or, the preset face template can be stored
  • the electronic device 100 then sends the face image to the cloud server 200 for comparison to determine the image of the human eye area, and the face recognition is handed over to the cloud server 200 for processing, which can reduce the processing capacity of the electronic device 100 and Improving image processing efficiency; then, the processor 60 can identify the image of the human eye area to determine the missing information of the human eye.
  • TEE Truste Execution Environment
  • the human eye is relatively smooth and has a high reflectivity to light, compared to other parts of the human face. Therefore, by setting the detection threshold, it can be determined whether the pixel is located in the human eye (for example, when the pixel value of the pixel is greater than the detection threshold, it is determined that the pixel is located in the human eye), so as to determine whether the pixel is located in the human eye.
  • the image part of the human eye, and the shape of the human eye is basically determined as an approximate ellipse. According to the recognized image part of the human eye and the preset shape of the human eye, the missing part of the human eye can be determined. The missing information is thus determined, and the missing information may include the proportion of the missing part of the human eye to the human eye.
  • the processor 60 can determine the weight of the human eye area image according to the missing information. For example, the larger the proportion of the missing part of the human eye to the human eye, the smaller the weight of the human eye area image; it can be understood that the missing part of the human eye. The larger the proportion of the human eye, the greater the degree of occlusion of the human eye, and the worse the accuracy of the human eye area image. Therefore, giving a smaller weight can reduce the impact of the human eye area image on the subsequent calculation of human eye gaze information. The influence of accuracy, improve the accuracy of gaze detection.
  • the human eye area image includes a left eye area image and a right eye area image
  • the processor 60 can respectively determine the first weight of the left eye area image according to the first missing information of the left eye area image, and the first weight of the left eye area image according to the right eye area image.
  • the second missing information of the region image is used to determine the second weight of the right-eye region image, thereby respectively determining the first weight of the left-eye region image and the second weight of the right-eye region image.
  • the first weight is negatively correlated with the first missing information (such as the proportion of the missing part of the left eye to the left eye)
  • the second weight is negatively correlated with the second missing information (such as the proportion of the missing part of the right eye to the right eye).
  • Negative correlation when the first weight of the left-eye region image is small (such as 0.6), it can reduce the impact of the left-eye region image on the calculation of human gaze information (such as by reducing the number of feature points from the left-eye region image to reduce the impact).
  • the gaze information of the human eye can be calculated according to the image of the human eye region and its weight.
  • the gaze information includes the gaze direction and the coordinates of the gaze point
  • the processor 60 performs feature extraction on the left eye area and the right eye area in the human eye area image, thereby determining the gaze direction of the human eye and the display of the human eye on the electronic device 100
  • the electronic device 100 can be controlled according to the gaze direction and gaze point coordinates. If it is detected that the gaze point coordinates are located in the display area of the display screen 40, keep the screen always on, and when it is detected that the gaze point coordinates are located outside the display area of the display screen 40, a predetermined duration (such as 10S, 20S, etc.), then Turn off the screen.
  • a predetermined duration such as 10S, 20S, etc.
  • the image of the eye area may be incomplete due to reasons such as occlusion, thereby affecting the accuracy of gaze point detection.
  • the human eye gaze detection method, detection device 10, and electronic device 100 of the present application determine the weight of the human eye area image by acquiring the human eye area image and the missing information of the human eye area image.
  • the weight of the image of the human eye area is considered as a factor, which is beneficial to reduce the impact of the lack of image of the human eye area caused by the occlusion of the human eye, and the impact on the accuracy of the calculation of the gaze information of the human eye, which can improve the accuracy of human eye gaze information.
  • the human eye gaze detection method also includes:
  • Step 015 includes:
  • 0151 Determine human eye gaze information based on face posture information, human eye area images, and weights of human eye area images.
  • the detection device 10 further includes a second acquisition module 14, which can also be set in the ISP to acquire face pose information.
  • the second determining module 13 is further configured to determine human eye gaze information according to human face posture information, human eye region images, and weights of human eye region images. That is to say, step 0141 may be performed by the second obtaining module 14 , and step 0151 may be performed by the second determining module 13 .
  • the processor 60 is further configured to acquire face pose information, and determine human eye gaze information according to the face pose information, the human eye region image, and the weight of the human eye region image. That is to say, step 0141 and step 0151 can be executed by the processor 60 .
  • the processor 60 can also obtain face posture information, and the face posture information can be calculated according to the position coordinates of the extracted feature points by performing feature extraction on the face image.
  • Posture information it can be understood that different postures of the face (such as establishing a three-dimensional coordinate system with the tip of the nose as the origin, and the pitch angle, horizontal rotation angle, and tilt angle of the face represent the rotation of the face relative to the three coordinate axes of the three-dimensional coordinate system Angle, etc.), will affect the user's gaze direction and gaze point coordinates. Therefore, when calculating the gaze information of the human eye, in addition to obtaining the image of the human eye area and its weight, it will also combine the facial posture information to more accurately calculate the gaze direction and gaze point coordinates.
  • the human eye gaze detection method also includes:
  • 0142 Obtain a face image, where the face image includes a face mask
  • 0143 Calculate the position information of the face relative to the electronic device 100 according to the face mask
  • Step 015 includes:
  • 0152 Determine the gaze information of the human eye according to the position information, the human eye area image, and the weight of the human eye area image.
  • the detection device 10 also includes a third acquisition module 15 and a calculation module 16, both of the third acquisition module 15 and the calculation module 16 can be set at the ISP, and the third acquisition module 15 is used to acquire a face image, calculate The module 16 is used for calculating the position information of the face relative to the electronic device 100 according to the face mask. That is to say, step 0142 can be performed by the third acquisition module 15 , step 0143 can be performed by the calculation module 16 , and step 0152 can be performed by the second determination module 13 .
  • the processor 60 is also used to acquire a face image, and the face image includes a face mask; calculate the position information of the face relative to the electronic device 100 according to the face mask; The weight of the image and the image of the human eye area determines the gaze information of the human eye. . That is to say, step 0142 , step 0143 and step 0152 may be executed by the processor 60 .
  • the processor 60 may also obtain a face image, and determine a face mask of the face image, where the face mask is used to characterize the position of the face in the face image,
  • the face mask can be determined by identifying the position of the face in the face image, and the processor 60 can calculate the position information of the face relative to the electronic device 100 according to the face mask (such as according to the ratio of the face mask to the face image) , can calculate the distance between the human face and the electronic device 100), it can be understood that when the distance between the human face and the electronic device 100 changes, even if the gaze direction of the human eye does not change, the gaze point coordinates of the human eye will still change. Therefore, in When calculating the gaze information of the human eye, in addition to obtaining the image of the human eye area and its weight, it will also combine the position information to calculate the coordinates of the gaze point more accurately.
  • the gaze information of the human eye in addition to acquiring the image of the eye region and its weight, it may also be combined with facial posture information and position information, so as to calculate the gaze information of the human eye more accurately.
  • the human eye gaze detection method also includes:
  • 0101 Obtain a training sample set, the training sample set includes multiple human eye area images;
  • 0102 Determine the occlusion area and/or image offset parameters according to the human eye area image
  • 0103 Train the human eye detection model according to the human eye area image, occlusion area, and image offset parameters.
  • Step 015 includes:
  • 0153 Based on the human eye detection model, determine the human eye gaze information according to the human eye area image and the weight of the human eye area image.
  • the detection device 10 further includes a fourth acquisition module 17 , a third determination module 18 and a training module 19 .
  • the fourth acquiring module 17, the third determining module 18 and the training module 19 can all be set in the NPU to train the human eye detection model.
  • the fourth acquisition module 17 is used to obtain the training sample set
  • the third determination module 18 is used to determine the occlusion area and/or image offset parameters according to the human eye area image
  • the training module 19 is used to determine the human eye area image, the occlusion area, and Image offset parameters to train the human eye detection model.
  • the second determining module 13 may be configured to determine human eye gaze information based on the human eye detection model and according to the human eye region image and the weight of the human eye region image. That is to say, step 0101 can be performed by the fourth acquisition module 17 , step 0102 can be performed by the third determination module 18 , step 0103 can be performed by the training module 19 , and step 0153 can be performed by the second determination module 13 .
  • the processor 60 is further configured to obtain a training sample set, the training sample set includes a plurality of human eye area images; determine the occlusion area and/or image offset parameters according to the human eye area images; according to the human eye area images, The occlusion area and image offset parameters train the human eye detection model; based on the human eye detection model, the human eye gaze information is determined according to the weight of the human eye area image and the human eye area image. That is to say, step 0101 , step 0102 , step 0103 and step 0153 may be executed by the processor 60 .
  • this application can realize the calculation of human eye gaze information through the preset human eye detection model, that is, based on the human eye detection model, determine the human eye gaze information according to the weight of the human eye area image and the human eye area image, in order to ensure that people To ensure the accuracy of eye gaze information, it is necessary to train the human eye detection model first, so that the human eye detection model converges.
  • the area image may include an occlusion area and/or image offset parameters, wherein the occlusion area is generated by replacing at least a part of pixels in the human eye area image with pixels of a predetermined pixel value (such as 0), and the occlusion area is used for the human eye area image
  • the occluded area should not block the eyebrows, eyes, mouth, nose and other related parts of the face;
  • the image offset generation is performed on the point, and the image offset parameter is used to indicate the deviation of the detection of the feature points of the human eye (such as the deviation of the coordinates of the feature points of the human eye).
  • the human eye detection model is trained by using the human eye area image with the occlusion area and/or image offset parameters, so that the human eye detection model trained to convergence can minimize the reduction in the human eye gaze information detection.
  • the impact of the occlusion of the human eye and the deviation of the detection of the feature points of the human eye can ensure the accuracy of gaze detection.
  • step 0103 includes:
  • 01031 Input the human eye area image, occlusion area, and image offset parameters into the human eye detection model to output training coordinates;
  • 01032 Based on the preset loss function, calculate the loss value according to the preset coordinates and training coordinates corresponding to the human eye area image;
  • 01033 Adjust the eye detection model according to the loss value until the eye detection model converges.
  • the training module 19 is also used to input the human eye area image, occlusion area, and image offset parameters into the human eye detection model to output training coordinates; based on the preset loss function, according to the human eye area image Calculate the loss value for the corresponding preset coordinates and training coordinates; adjust the human eye detection model according to the loss value until the human eye detection model converges. That is to say, Step 01031 to Step 01033 can be executed by the training module 19 .
  • the processor 60 is further configured to input the human eye region image, occlusion region, and image offset parameters into the human eye detection model to output training coordinates; based on a preset loss function, according to the human eye region image Calculate the loss value for the corresponding preset coordinates and training coordinates; adjust the human eye detection model according to the loss value until the human eye detection model converges. That is to say, Step 01031 to Step 01033 may be executed by the processor 60 .
  • the face detection model 50 includes a first feature extraction module 51 , a second feature extraction module 52 , a first weight module 53 , a second weight module 54 and a feature fusion module 55 .
  • the human eye area image is input into the human eye detection model, and the human eye area image includes a left eye area image and a right eye area image, and the first feature extraction module 51 can use the left eye area image ( Or the feature points of the left eye area image) are extracted, the second feature extraction module 52 can extract the right eye area image (or the feature points of the right eye area image) in the human eye area image, and then the first weight module 53 can be The first missing information is determined according to the occlusion area of the left-eye area image, thereby determining the first weight, and the second weight module 54 can determine the second missing information according to the occlusion area of the right-eye area image, thereby determining the second weight, and then the left-eye
  • the region image, the right-eye region image, the first weight and the second weight are input into the feature fusion module for calculation, and the specific calculation method may be to weight the left-eye region image by the first weight (such as first weight * left-eye region image), weight the right eye area image by the second weight (such as the second
  • the face detection model 50 also includes a third feature extraction module 56, the third feature extraction module 56 can be used to extract feature points in the face image, so as to output the face pose information to the feature fusion module, thereby Calculating human eye gaze information according to the face posture information, the left-eye area image, the right-eye area image, the first weight and the second weight.
  • the output human eye gaze information is the training coordinates, and there are preset coordinates as the human eye region image of the training sample, and the preset coordinates are the real human eye gaze information (such as the human eye gaze direction and gaze point coordinates)
  • processor 60 calculates the loss value based on the preset loss function, the training coordinates and the preset coordinates, and the loss function is as follows: Among them, loss is the loss value, N is the number of training samples contained in each training sample set, X and Y are the training coordinates, Gx and Gy are the preset coordinates, when the training coordinates are the gaze direction, X and Y represent pitch respectively Angle and yaw angle, when the training coordinates are the gaze point coordinates, X and Y represent the coordinates of the gaze point on the plane where the display screen 40 is located, respectively, thereby calculating the loss value of the gaze direction and the loss value of the gaze point coordinates respectively.
  • the processor 60 can adjust the human eye detection model according to the loss value of the gaze direction and the loss value of the gaze point coordinates, so that the gradient of the human eye detection model continues to decrease, so that the training coordinates are getting closer and closer to the preset coordinates, and finally the human eye
  • the detection model is trained to convergence.
  • the human eye detection model is trained through the human eye area image with occlusion area and/or image offset parameters, and the human eye detection model is continuously adjusted to convergence based on the loss value, which can minimize the human eye being occluded and The impact of human eye feature point detection offset on gaze detection improves the accuracy of gaze detection.
  • the detection method of the embodiment of the present application includes the following steps:
  • the detection device 20 in the embodiment of the present application includes a first acquisition module 21 , a first determination module 22 , a second determination module 23 , a third determination module 24 and a fourth determination module 25 .
  • the first acquiring module 21 is used to acquire the human face image;
  • the first determining module 22 is used to determine the human eye region image and the facial posture information according to the human face image;
  • the second determining module 23 is used to determine the missing information of the human eye region image, Determine the weight of the human eye area image;
  • the third determination module 24 is used to determine the human eye feature information according to the human eye area image and the weight of the human eye area image; Information, to determine the human eye gaze information.
  • step 021 can be implemented by the first acquisition module 11, step 022 can be performed by the first determination module 12, step 023 can be performed by the second determination module 23, step 024 can be performed by the third determination module 24, step 025 It may be executed by the fourth determining module 25 .
  • the electronic device 100 in the embodiment of the present application includes a processor 60 and a collection device 30 .
  • Acquisition device 30 is used for collecting face information by predetermined frame rate (face information comprises people's face image, as the visible light image of people's face, infrared image, depth image etc.);
  • Acquisition device 30 can be visible light camera, infrared camera, depth camera One or more of them, wherein the visible light camera can collect visible light face images, the infrared camera can collect infrared face images, and the depth camera can collect depth face images.
  • the collection device 30 includes a visible light camera, an infrared face image camera and depth camera, the acquisition device 30 simultaneously visible light face image, infrared face image and depth face image.
  • the detection device 20 is arranged in the electronic equipment 100, and the processor 60 may include an ISP, an NPU, and an AP.
  • the first acquisition module 21 may be arranged in the acquisition device 30 to obtain a face image.
  • the first determination module 22 and the second determination module 23 and the third determination module 24 may be set in the ISP, and the fourth determination module 25 may be set in the NPU.
  • Acquisition device 30 is used for obtaining human face image;
  • Processor 60 (specifically can be ISP) is used for determining human eye area image and human face posture information according to human face image; According to the missing information of human eye area image, determine human eye area image weight; according to the weight of the human eye area image and the human eye area image, determine the human eye feature information;
  • processor 60 (specifically can be NPU) is also used for determining the human eye gaze information according to the human eye feature information and the facial posture information . That is to say, step 021 may be executed by the collection device 30 , and steps 022 to 025 may be executed by the processor 60 .
  • the collection device 30 can collect the user's face information once at a predetermined time interval, and continue to perform gaze detection on the user while ensuring that the power consumption of the electronic device 100 is small, or, when the user When using applications that require gaze detection (such as browser software, post bar software, video software, etc.), the face information is collected according to the predetermined number of frames, so that face information is collected only when gaze detection is required, maximizing Reduced power consumption for gaze detection.
  • applications that require gaze detection such as browser software, post bar software, video software, etc.
  • the processor 60 can identify the face image, for example, the processor 60 can compare the face image with the preset face template , so as to determine the image area where the face and different parts of the face (such as glasses, nose, etc.)
  • the face template can be stored in the memory of the electronic device 100, and the processor 60 can perform face recognition in a trusted execution environment (Trusted Execution Environment, TEE) to ensure the privacy of the user; or, the preset face template can be stored
  • TEE Truste Execution Environment
  • the electronic device 100 then sends the face image to the cloud server 200 for comparison to determine the image of the human eye area, and the face recognition is handed over to the cloud server 200 for processing, which can reduce the processing capacity of the electronic device 100 and Improve image processing efficiency.
  • the processor 60 can also perform feature extraction on the face image, and calculate the face posture information according to the position coordinates of the extracted feature points.
  • the pitch angle, horizontal rotation angle, and tilt angle represent the rotation angles of the face relative to the three coordinate axes of the three-dimensional coordinate system, etc.), which will affect the user's gaze direction and gaze point coordinates. Therefore, when calculating human eye gaze information When , in addition to obtaining the image of the human eye area, it will also combine the facial posture information to more accurately calculate the gaze direction and gaze point coordinates.
  • the processor 60 can identify the image of the human eye area to determine the missing information of the human eye.
  • the human eye is relatively smooth and has a high reflectivity to light. Compared with other parts of the human face, the human eye is The pixel value in the image will be larger, therefore, by setting the detection threshold, it can be determined whether the pixel is located in the human eye (for example, when the pixel value of the pixel is greater than the detection threshold, it is determined that the pixel is located in the human eye), so as to determine the human eye region image.
  • the image part of the human eye, and the shape of the human eye is basically determined and approximately elliptical. According to the recognized image part of the human eye and the preset shape of the human eye, the missing part of the human eye can be determined, thereby determining the missing information.
  • the missing information may include the ratio of the missing part of the human eye to the human eye.
  • the processor 60 can determine the weight of the human eye area image according to the missing information. For example, the larger the proportion of the missing part of the human eye to the human eye, the smaller the weight of the human eye area image; it can be understood that the missing part of the human eye. The larger the proportion of the human eye, the greater the degree of occlusion of the human eye, and the worse the accuracy of the human eye area image. Therefore, giving a smaller weight can reduce the impact of the human eye area image on the subsequent calculation of human eye gaze information. The influence of accuracy, improve the accuracy of gaze detection.
  • the human eye area image includes a left eye area image and a right eye area image
  • the processor 60 can respectively determine the first weight of the left eye area image according to the first missing information of the left eye area image, and the first weight of the left eye area image according to the right eye area image.
  • the second missing information of the region image is used to determine the second weight of the right-eye region image, thereby respectively determining the first weight of the left-eye region image and the second weight of the right-eye region image.
  • the first weight is negatively correlated with the first missing information (such as the proportion of the missing part of the left eye to the left eye)
  • the second weight is negatively correlated with the second missing information (such as the proportion of the missing part of the right eye to the right eye).
  • Negative correlation when the first weight of the left-eye region image is small (for example, 0.6), the influence of the left-eye region image on the calculation of human gaze information can be reduced (such as by extracting fewer features of the left-eye region image point to reduce the impact).
  • the feature information of the human eye can be determined according to the image of the human eye region and its weight, for example, the weights corresponding to the image of the face region and the image of the human eye region are directly As the human eye feature information, or first extract the human eye feature points from the human eye area image according to the weight of the human eye area image, and then use the extracted human eye feature points as the human eye feature information.
  • the processor 60 determines the gaze information of the human eye according to the characteristic information of the human eye and the posture information of the human face.
  • the gaze information includes gaze direction and gaze point coordinates
  • processor 60 can perform feature extraction on the left eye area and right eye area in the human eye area image respectively, and then combine the facial posture information to determine the gaze direction of the human eye and The gaze point coordinates of the human eyes on the plane where the display screen 40 of the electronic device 100 is located.
  • the electronic device 100 can be controlled according to the gaze direction and gaze point coordinates. If it is detected that the gaze point coordinates are located in the display area of the display screen 40, keep the screen always on, and when it is detected that the gaze point coordinates are located outside the display area of the display screen 40, a predetermined duration (such as 10S, 20S, etc.), then Turn off the screen.
  • a predetermined duration such as 10S, 20S, etc.
  • the human eye gaze detection method, detection device 20, and electronic device 100 of the present application determine the weight of the human eye area image by acquiring the human eye area image and the missing information of the human eye area image.
  • the weight of the image of the human eye area is considered as a factor, which is beneficial to reduce the impact of the lack of image of the human eye area caused by the occlusion of the human eye, and the impact on the accuracy of the calculation of the gaze information of the human eye, which can improve the accuracy of human eye gaze information.
  • the human eye gaze information can be accurately calculated through the face posture information, which can improve the accuracy of the human eye gaze information.
  • face image comprises face mask
  • step 025 comprises:
  • the fourth determination module 25 is also used to calculate the position information of the human face relative to the electronic device 100 according to the face mask; determine the gaze information of the human eye according to the position information, human eye feature information and human face posture information . That is to say, step 0251 and step 0252 can be executed by the fourth determination module 25 .
  • the processor 60 is further configured to calculate the position information of the face relative to the electronic device 100 according to the face mask; and determine the gaze information of the human eye according to the position information, the feature information of the human eyes and the posture information of the human face. That is to say, step 0251 and step 0252 can be executed by the processor 60 .
  • the processor 60 can also determine the face mask of the face image, which is used to represent the position of the face in the face image, and the face mask can be obtained by Identify the position of the human face in the human face image, and the processor 60 can calculate the position information of the human face relative to the electronic device 100 (such as the distance between the human face and the electronic device 100) according to the face mask.
  • the human face and the electronic device 100 When the distance of the device 100 changes, even if the gaze direction of the human eye does not change, the coordinates of the gaze point of the human eye will still change. Therefore, when calculating the gaze information of the human eye, in addition to obtaining the image of the human eye region and its weight, and the human face In addition to attitude information, position information will also be combined to more accurately calculate the coordinates of the gaze point.
  • the method for controlling the electronic device 100 through human eyes includes the following steps:
  • the control device 30 in the embodiment of the present application includes a first acquisition module 31 , a first determination module 32 and a control module 33 .
  • the first acquisition module 31 is used to obtain the human eye region image;
  • the first determination module 32 is used to determine the human eye gaze information according to the human eye region image and the weight of the human eye region image;
  • the control module 33 is used to control the human eye gaze information according to the human eye region image.
  • Electronic device 100 That is to say, step 031 can be implemented by the first acquisition module 31 , step 033 can be performed by the first determination module 32 and step 035 can be performed by the control module 33 .
  • the electronic device 100 in the embodiment of the present application includes a processor 60 and a collection device 30 .
  • Acquisition device 30 is used for collecting face information by predetermined frame rate (face information comprises people's face image, as the visible light image of people's face, infrared image, depth image etc.);
  • Acquisition device 30 can be visible light camera, infrared camera, depth camera One or more of them, wherein the visible light camera can collect visible light face images, the infrared camera can collect infrared face images, and the depth camera can collect depth face images.
  • the collection device 30 includes a visible light camera, an infrared face image camera and depth camera, the acquisition device 30 simultaneously visible light face image, infrared face image and depth face image.
  • Processor 60 may include ISP, NPU and AP, such as control device 30 is arranged in electronic equipment 100, the first acquisition module 11 is arranged in ISP, processor 60 is connected with acquisition device 30, after acquisition device 30 collects the facial image
  • the ISP can process the face image to obtain the image of the human eye area
  • the first determination module 12 can be set in the NPU
  • the control module 13 can be set in the AP.
  • the processor 60 (specifically, it may be an ISP) is used to acquire the image of the human eye area; the processor 60 (specifically, it may be an NPU) is also used to determine the human eye gaze information according to the weight of the human eye area image and the human eye area image.
  • the processor 60 (specifically, it may be an AP) can also be used to control the electronic device 100 according to the gaze information of human eyes. That is to say, step 031 can be executed by the collection device 30 in cooperation with the processor 60 , and steps 032 and 033 can be executed by the processor 60 .
  • the collection device 30 can collect the user's face information once at a predetermined time interval, and continue to perform gaze detection on the user while ensuring that the power consumption of the electronic device 100 is small, or, when the user When using applications that require gaze detection (such as browser software, post bar software, video software, etc.), the face information is collected according to the predetermined number of frames, so that face information is collected only when gaze detection is required, maximizing Reduced power consumption for gaze detection.
  • applications that require gaze detection such as browser software, post bar software, video software, etc.
  • the processor 60 can identify the face image, for example, the processor 60 can compare the face image with the preset face template , so as to determine the image area where the face and different parts of the face (such as glasses, nose, etc.)
  • the face template can be stored in the memory of the electronic device 100, and the processor 60 can perform face recognition in a trusted execution environment (Trusted Execution Environment, TEE) to ensure the privacy of the user; or, the preset face template can be stored In the cloud server 200, the electronic device 100 then sends the face image to the cloud server 200 for comparison to determine the image of the human eye area, and the face recognition is handed over to the cloud server 200 for processing, which can reduce the processing capacity of the electronic device 100 and Improve image processing efficiency;
  • TEE Truste Execution Environment
  • the processor 60 can identify the image of the human eye area to determine the missing information of the human eye.
  • the human eye is relatively smooth and has a high reflectivity to light. Compared with other parts of the human face, the human eye is The pixel value in the image will be larger, therefore, by setting the detection threshold, it can be determined whether the pixel is located in the human eye (for example, when the pixel value of the pixel is greater than the detection threshold, it is determined that the pixel is located in the human eye), so as to determine the human eye region image.
  • the image part of the human eye, and the shape of the human eye is basically determined and approximately elliptical. According to the recognized image part of the human eye and the preset shape of the human eye, the missing part of the human eye can be determined, thereby determining the missing information.
  • the missing information may include the ratio of the missing part of the human eye to the human eye.
  • the processor 60 can determine the weight of the human eye area image according to the missing information. For example, the larger the proportion of the missing part of the human eye to the human eye, the smaller the weight of the human eye area image; it can be understood that the missing part of the human eye. The larger the proportion of the human eye, the greater the degree of occlusion of the human eye, and the worse the accuracy of the human eye area image. Therefore, giving a smaller weight can reduce the impact of the human eye area image on the subsequent calculation of human eye gaze information. The influence of accuracy, improve the accuracy of gaze detection.
  • the human eye area image includes a left eye area image and a right eye area image
  • the processor 60 can respectively determine the first weight of the left eye area image according to the first missing information of the left eye area image, and the first weight of the left eye area image according to the right eye area image.
  • the second missing information of the region image is used to determine the second weight of the right-eye region image, thereby respectively determining the first weight of the left-eye region image and the second weight of the right-eye region image.
  • the first weight is negatively correlated with the first missing information (such as the proportion of the missing part of the left eye to the left eye)
  • the second weight is negatively correlated with the second missing information (such as the proportion of the missing part of the right eye to the right eye).
  • Negative correlation when the first weight of the left-eye region image is small (for example, 0.6), the influence of the left-eye region image on the calculation of human gaze information can be reduced (such as by extracting fewer features of the left-eye region image point to reduce the impact).
  • the gaze information of the human eye can be calculated according to the image of the human eye region and its weight.
  • the gaze information includes the gaze direction and the coordinates of the gaze point
  • the processor 60 performs feature extraction on the left eye area and the right eye area in the human eye area image, thereby determining the gaze direction of the human eye and the display of the human eye on the electronic device 100
  • the electronic device 100 can be controlled according to the gaze direction and gaze point coordinates.
  • a three-dimensional coordinate system is established with the midpoint of the eyes as the origin O1
  • the X1 axis is parallel to the direction of the line connecting the centers of the eyes
  • the Y1 axis is located on the horizontal plane and perpendicular to the X1 axis
  • the Z1 axis is perpendicular to the X1 axis and Y1 axis.
  • the three-axis rotation angle of the line of sight S and the three-dimensional coordinate system indicates the user's gaze direction.
  • the gaze direction includes pitch angle, roll angle and yaw angle respectively.
  • the pitch angle represents the rotation angle around the X1 axis
  • the roll angle represents the rotation angle around the Y1 axis.
  • the rotation angle of the axis, the yaw angle represents the rotation angle around the Z1 axis
  • the processor 60 can realize the page turning or sliding operation of the display content of the electronic device 100 according to the gaze direction, for example, according to the determination of continuous multiple frames of human eye area images (such as 10 consecutive frames) of the gaze direction, the change of the gaze direction can be determined, for example, please combine Figure 15 and Figure 16, when the pitch angle gradually increases (that is, the line of sight S is tilted), it can be determined that the user wants the displayed content to slide up or Turning the page down. For another example, please refer to FIG. 15 and FIG.
  • the pitch angle gradually decreases (that is, the line of sight S is tilted), then it can be determined that the user wants to slide the displayed content down or turn the page up.
  • the electronic device 100 can also be slid or page-turned.
  • the center of the display screen 40 can be used as the coordinate origin O2 to establish a plane coordinate system
  • the width direction parallel to the electronic device 100 is used as the X2 axis
  • the length direction parallel to the electronic device 100 is used as the Y2 axis
  • the gaze point coordinates include the abscissa (corresponding to the position on the X2 axis) and the ordinate (corresponding to the position on the Y2 axis).
  • the ordinate gradually increases, it means that the gaze point M moves up. It can be determined that the user wants to slide up or turn the page down, and then For example, if the ordinate gradually decreases, it means that the gaze point M moves down, and it can be determined that the user wants to slide the displayed content down or turn the page up.
  • the processor 60 can also obtain 10 consecutive frames according to the change speed of the gaze direction (such as the difference between the pitch angles of the first frame and the tenth frame (or the difference between the vertical coordinates of the gaze point M) and The duration is determined), the faster the change speed, the more new display content will be displayed after sliding.
  • the predetermined time length (such as 10S, 20S, etc.) after the user does not check the display screen 40 can be Turn off the screen again.
  • the method for controlling the electronic device 100 through human eyes, the control device 30 and the electronic device 100 of the present application determine the weight of the human eye region image by acquiring the human eye region image and the missing information of the human eye region image.
  • the weight of the human eye area image is considered as a factor, which is beneficial to reduce the impact of the human eye area image being missing due to the occlusion of the human eye, and the accuracy of the calculation of the human eye gaze information.
  • the method for controlling the electronic device 100 through human eyes further includes:
  • Step 033 includes:
  • 0331 Determine the human eye gaze information according to the face posture information, the human eye area image, and the weight of the human eye area image.
  • control device 30 further includes a second acquisition module 34, which can also be set in the ISP to acquire facial posture information.
  • the first determining module 32 is further configured to determine human eye gaze information according to the human face posture information, the human eye area image, and the weight of the human eye area image. That is to say, step 0321 can be performed by the second obtaining module 34 , and step 0331 can be performed by the first determining module 32 .
  • the processor 60 is further configured to acquire face pose information, and determine human eye gaze information according to the face pose information, the human eye region image, and the weight of the human eye region image. That is to say, step 0321 and step 0331 can be executed by the processor 60 .
  • step 0321 For the specific description of step 0321, please refer to step 0141, and for the specific description of step 0331, please refer to step 0151, which will not be repeated here.
  • the method for controlling the electronic device 100 through human eyes further includes:
  • Step 033 includes:
  • 0332 Determine the gaze information of the human eye according to the position information, the image of the human eye area, and the weight of the image of the human eye area.
  • control device 30 also includes a third acquisition module 35 and a calculation module 36, both of the third acquisition module 35 and the calculation module 36 can be set at the ISP, and the third acquisition module 35 is used to acquire a face image, calculate The module 36 is used for calculating the position information of the face relative to the electronic device 100 according to the face mask. That is to say, step 0322 can be performed by the third acquisition module 35 , step 0332 can be performed by the calculation module 16 , and step 0332 can be performed by the first determination module 32 .
  • the processor 60 is further configured to acquire face pose information, and determine human eye gaze information according to the face pose information, the human eye region image, and the weight of the human eye region image. That is to say, step 0322 , step 323 and step 0332 may be executed by the processor 60 .
  • Step 0322, Step 0323, and Step 0332 please refer to Step 0142, Step 0143, and Step 0152, respectively, and details are not repeated here.
  • the method for controlling the electronic device 100 through human eyes further includes:
  • 0303 Train the human eye detection model according to the human eye area image, occlusion area, and image offset parameters.
  • Step 033 includes:
  • 0333 Based on the human eye detection model, determine the human eye gaze information according to the human eye area image and the weight of the human eye area image.
  • the control device 30 further includes a fourth acquisition module 37 , a second determination module 38 and a training module 39 .
  • the fourth acquiring module 37, the second determining module 38 and the training module 39 can all be set in the NPU to train the human eye detection model.
  • the fourth acquisition module 37 is used to obtain the training sample set
  • the second determination module 38 is used to determine the occlusion area and/or image offset parameters according to the human eye area image; Image offset parameters to train the human eye detection model.
  • the first determining module 32 may be configured to determine human eye gaze information based on the human eye detection model and according to the human eye region image and the weight of the human eye region image. That is to say, step 0301 can be performed by the fourth acquisition module 37 , step 0302 can be performed by the second determination module 38 , step 0103 can be performed by the training module 39 , and step 0333 can be performed by the first determination module 32 .
  • the processor 60 is further configured to obtain a training sample set, the training sample set includes a plurality of human eye area images; determine the occlusion area and/or image offset parameters according to the human eye area images; according to the human eye area images, The occlusion area and image offset parameters train the human eye detection model; based on the human eye detection model, the human eye gaze information is determined according to the weight of the human eye area image and the human eye area image. That is to say, step 0301 , step 0302 , step 0303 and step 0333 may be executed by the processor 60 .
  • Step 0301 for specific descriptions of Step 0301, Step 0302, Step 0303, and Step 0333, please refer to Step 0101, Step 0102, Step 0103, and Step 0153, respectively, and details are not repeated here.
  • step 0303 includes:
  • 03031 Input the human eye area image, occlusion area, and image offset parameters into the human eye detection model to output training coordinates;
  • 03032 Based on the preset loss function, calculate the loss value according to the preset coordinates and training coordinates corresponding to the human eye area image;
  • the training module 39 is also used to input the human eye area image, occlusion area, and image offset parameters into the human eye detection model to output training coordinates; based on a preset loss function, according to the human eye area image Calculate the loss value for the corresponding preset coordinates and training coordinates; adjust the human eye detection model according to the loss value until the human eye detection model converges. That is to say, Step 03031 to Step 03033 can be executed by the training module 39 .
  • the processor 60 is further configured to input the human eye region image, occlusion region, and image offset parameters into the human eye detection model to output training coordinates; based on a preset loss function, according to the human eye region image Calculate the loss value for the corresponding preset coordinates and training coordinates; adjust the human eye detection model according to the loss value until the human eye detection model converges. That is to say, Step 03031 to Step 03033 may be executed by the processor 60 .
  • Step 03031, Step 03032, and Step 03033 please refer to Step 01031, Step 01032, and Step 01033, respectively, and details are not repeated here.
  • the gaze information of the human eye includes gaze point coordinates.
  • the method for controlling the electronic device 100 through the human eye further includes:
  • Step 035 includes:
  • control module 33 is also configured to acquire a captured image within the first predetermined time before the screen is off; in response to the captured image containing a human face; in response to the gaze point coordinates being located in the display area of the display screen 40, Keep the screen on for a second predetermined duration. That is to say, step 0301 , step 0302 and step 0351 can be executed by the control module 33 .
  • the processor 60 is further configured to acquire a captured image within the first predetermined time period before the screen turns off; in response to the captured image containing a human face; in response to the gaze point coordinates being located in the display area of the display screen 40, Keep the screen on for a second predetermined duration. That is to say, step 0351 and step 0352 can be executed by the processor 60 .
  • the human eye gaze information can be used to realize off-screen control. Before the screen is off, gaze detection is first performed. For example, the processor 60 first obtains a captured image. If there is a human face in the captured image, the human eye gaze information is performed according to the captured image. Of course, in order to ensure that there is enough time to acquire the captured image and calculate the gaze information before the screen is closed, it is necessary to acquire the captured image within the first predetermined time period (such as 5 seconds, 10 seconds, etc.) before the screen is closed.
  • the first predetermined time period such as 5 seconds, 10 seconds, etc.
  • the gaze point M when the gaze point M is located in the display area of the display screen 40, it can be determined that the user is watching the display screen 40, so that the display screen 40 remains on for a second predetermined duration, and the second predetermined duration can be greater than the first predetermined time length, and within the first predetermined time length before the screen is turned off again, the captured image is acquired again, so that when the user looks at the display screen 40, the screen remains bright, and when the user no longer looks at the display screen 40, the screen is turned off again. Screen.
  • the center of the display area can be used as the coordinate origin O2 to establish a two-dimensional coordinate system parallel to the display screen 40, and the display area is associated with a preset coordinate range, that is, the display area is in the horizontal direction of the two-dimensional coordinate system.
  • the coordinate range and the ordinate range can be determined when the gaze point coordinates are within the preset coordinate range (that is, the abscissa of the gaze point coordinates is within the abscissa range and the ordinate is within the ordinate range).
  • the gaze point coordinates are located in the display area, so it is relatively simple to determine whether the user gazes at the display screen 40 .
  • the gaze information of the human eye includes the gaze point coordinates.
  • the method for controlling the electronic device 100 through the human eye further includes:
  • Step 035 includes:
  • control module 33 is further configured to acquire a captured image in response to the fact that the electronic device 100 does not receive an input operation, and adjust the display screen 40 in response to the captured image containing a human face and the coordinates of the gaze point located in the display area. Adjust the display brightness to the first predetermined brightness, and adjust the display brightness to the second predetermined brightness in response to the fact that the captured image does not contain a human face, or the captured image contains a human face and the gaze point coordinates are outside the display area, and the second predetermined brightness less than the first predetermined brightness. That is to say, step 0303 , step 0352 and step 0353 can be executed by the control module 33 .
  • the processor 60 is further configured to acquire a captured image in response to the fact that the electronic device 100 does not receive an input operation, and adjust the display screen 40 in response to the captured image containing a human face and the coordinates of the gaze point located in the display area. Adjust the display brightness to the first predetermined brightness, and adjust the display brightness to the second predetermined brightness in response to the fact that the captured image does not contain a human face, or the captured image contains a human face and the gaze point coordinates are outside the display area, and the second predetermined brightness less than the first predetermined brightness. That is to say, step 0303 , step 0352 and step 0353 can be executed by the processor 60 .
  • the human eye gaze information can also be used to realize intelligent brightening of the screen.
  • the electronic device 100 will generally reduce the display brightness first after a certain period of time when the screen is bright, and then turn on the screen with a low brightness. After a certain period of time, the screen will stop.
  • the processor 60 can obtain the captured image. If the image contains a human face, the gaze information of the human eye is calculated according to the captured image.
  • the display brightness is adjusted to the first predetermined brightness at this time.
  • the first predetermined brightness may be the brightness set by the user when the display screen 40 is normally displayed, or it may be changed in real time according to the brightness of the ambient light to adapt to the brightness of the ambient light, so as to ensure that the user can still brighten the electronic device 100 even if the user does not operate the electronic device 100. screen, so as to prevent the situation that the user does not operate the electronic device 100 but suddenly turns off the screen when viewing the displayed content and affects the user experience.
  • the display brightness can be adjusted to a second predetermined brightness, which is smaller than the first predetermined brightness, so as to prevent unnecessary power consumption.
  • the display brightness is adjusted to the first predetermined brightness again, so as to ensure the normal viewing experience of the user. In this way, it can be realized that when the user does not operate the electronic device 100, the user looks at the display area, and the display area is displayed at normal brightness; Save battery.
  • the training method of the human eye detection model of the embodiment of the present application includes:
  • the training device 40 in the embodiment of the present application includes an acquisition module 41 , a determination module 42 and a training module 43 .
  • the acquiring module 41, the determining module 42 and the training module 43 can all be set in the NPU to train the human eye detection model.
  • the obtaining module 41 is used to obtain the training sample set, and the determining module 42 is used to determine the occlusion area and/or image offset parameters according to the human eye area image;
  • the training module 43 is used to determine the human eye area image, the occlusion area, and the image offset parameters Train the human eye detection model. That is to say, step 041 may be performed by the acquisition module 41 , step 042 may be performed by the determination module 42 , and step 043 may be performed by the training module 43 .
  • the processor 60 is further configured to obtain a training sample set, the training sample set includes a plurality of human eye area images; determine the occlusion area and/or image offset parameters according to the human eye area images; according to the human eye area images, The occlusion area and image offset parameters are used to train the human eye detection model. That is to say, step 041 , step 042 and step 043 may be executed by the processor 60 .
  • step 0101 please refer to step 0101, step 0102, and step 0103 for specific descriptions of step 041, step 042, and step 043, and details are not repeated here.
  • one or more non-transitory computer-readable storage media 300 containing a computer program 302 when the computer program 302 is executed by one or more processors 60, the processors 60 can Execute the human eye gaze detection method or the method for controlling the electronic device 100 through human eyes in any one of the above embodiments.
  • the processors 60 are made to perform the following steps:
  • 013 Determine the weight of the human eye area image according to the missing information of the human eye area image
  • 015 According to the weight of the human eye area image and the human eye area image, determine the human eye gaze information.
  • processors 60 when the computer program 302 is executed by one or more processors 60, the processors 60 may also perform the following steps:
  • processors 60 when the computer program 302 is executed by one or more processors 60, the processors 60 may also perform the following steps:

Abstract

L'invention concerne un procédé de détection du regard humain, un procédé de commande d'un dispositif électronique (100) au moyen des yeux, un appareil de détection (10), un appareil de commande (30), un dispositif électronique (100), et un support de stockage lisible par ordinateur non volatil (300). Le procédé de détection du regard humain consiste à : (011) acquérir une image de zone d'œil humain ; (013) déterminer une pondération de l'image de la zone de l'œil humain en fonction des informations manquantes de l'image de la zone de l'œil humain ; (015) déterminer des informations de regard humain en fonction de l'image de la zone de l'œil humain et de la pondération de l'image de la zone de l'œil humain.
PCT/CN2022/126122 2021-10-29 2022-10-19 Procédé de détection de regard humain, procédé de commande et dispositif associé WO2023071882A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202111274110.3 2021-10-29
CN202111274110.3A CN113903078A (zh) 2021-10-29 2021-10-29 人眼注视检测方法、控制方法及相关设备

Publications (1)

Publication Number Publication Date
WO2023071882A1 true WO2023071882A1 (fr) 2023-05-04

Family

ID=79026982

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/126122 WO2023071882A1 (fr) 2021-10-29 2022-10-19 Procédé de détection de regard humain, procédé de commande et dispositif associé

Country Status (2)

Country Link
CN (1) CN113903078A (fr)
WO (1) WO2023071882A1 (fr)

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113903078A (zh) * 2021-10-29 2022-01-07 Oppo广东移动通信有限公司 人眼注视检测方法、控制方法及相关设备
CN116704589A (zh) * 2022-12-01 2023-09-05 荣耀终端有限公司 一种注视点估计方法、电子设备和计算机可读存储介质
CN117133043A (zh) * 2023-03-31 2023-11-28 荣耀终端有限公司 注视点估计方法、电子设备及计算机可读存储介质

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101489467A (zh) * 2006-07-14 2009-07-22 松下电器产业株式会社 视线方向检测装置和视线方向检测方法
CN104766059A (zh) * 2015-04-01 2015-07-08 上海交通大学 快速精确的人眼定位方法及基于人眼定位的视线估计方法
CN110825220A (zh) * 2019-09-29 2020-02-21 深圳市火乐科技发展有限公司 眼球追踪控制方法、装置、智能投影仪及存储介质
CN113903078A (zh) * 2021-10-29 2022-01-07 Oppo广东移动通信有限公司 人眼注视检测方法、控制方法及相关设备

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101489467A (zh) * 2006-07-14 2009-07-22 松下电器产业株式会社 视线方向检测装置和视线方向检测方法
CN104766059A (zh) * 2015-04-01 2015-07-08 上海交通大学 快速精确的人眼定位方法及基于人眼定位的视线估计方法
CN110825220A (zh) * 2019-09-29 2020-02-21 深圳市火乐科技发展有限公司 眼球追踪控制方法、装置、智能投影仪及存储介质
CN113903078A (zh) * 2021-10-29 2022-01-07 Oppo广东移动通信有限公司 人眼注视检测方法、控制方法及相关设备

Also Published As

Publication number Publication date
CN113903078A (zh) 2022-01-07

Similar Documents

Publication Publication Date Title
WO2023071884A1 (fr) Procédé de détection de regard, procédé de commande pour dispositif électronique et dispositifs associés
WO2023071882A1 (fr) Procédé de détection de regard humain, procédé de commande et dispositif associé
US9373156B2 (en) Method for controlling rotation of screen picture of terminal, and terminal
US9696859B1 (en) Detecting tap-based user input on a mobile device based on motion sensor data
US8942434B1 (en) Conflict resolution for pupil detection
EP2634727B1 (fr) Terminal portable et procédé permettant de corriger la direction du regard d'un utilisateur dans une image
US20220301218A1 (en) Head pose estimation from local eye region
WO2017113668A1 (fr) Procédé et dispositif de commande d'un terminal conformément au mouvement oculaire
CN104679225B (zh) 移动终端的屏幕调节方法、屏幕调节装置及移动终端
CN109375765B (zh) 眼球追踪交互方法和装置
TWI704501B (zh) 可由頭部操控的電子裝置與其操作方法
US20150109204A1 (en) Human-machine interaction method and apparatus
KR20190129947A (ko) 얼굴 이미지 품질을 결정하는 방법과 장치, 전자 기기 및 컴퓨터 저장 매체
US20160078680A1 (en) Technologies for adjusting a perspective of a captured image for display
WO2020042542A1 (fr) Procédé et appareil d'acquisition de données d'étalonnage de commande de mouvement oculaire
TWI631506B (zh) 螢幕旋轉控制方法及系統
WO2019062056A1 (fr) Procédé et système de projection intelligente et terminal intelligent
CN104574321A (zh) 图像修正方法、图像修正装置和视频系统
CN108090463B (zh) 对象控制方法、装置、存储介质和计算机设备
WO2018078857A1 (fr) Dispositif d'estimation de ligne de visée, procédé d'estimation de ligne de visée et support d'enregistrement de programme
WO2021197466A1 (fr) Procédé, appareil et dispositif de détection de globe oculaire, et support de stockage
WO2024021742A1 (fr) Procédé d'estimation de point de fixation et dispositif associé
WO2018076172A1 (fr) Procédé et terminal d'affichage d'image
WO2021135639A1 (fr) Procédé et appareil de détection de corps vivant
TWI768852B (zh) 人體方向之偵測裝置及人體方向之偵測方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22885760

Country of ref document: EP

Kind code of ref document: A1