WO2021004285A1 - 人眼闭合程度的确定方法、眼睛控制方法、装置、设备和存储介质 - Google Patents

人眼闭合程度的确定方法、眼睛控制方法、装置、设备和存储介质 Download PDF

Info

Publication number
WO2021004285A1
WO2021004285A1 PCT/CN2020/098066 CN2020098066W WO2021004285A1 WO 2021004285 A1 WO2021004285 A1 WO 2021004285A1 CN 2020098066 W CN2020098066 W CN 2020098066W WO 2021004285 A1 WO2021004285 A1 WO 2021004285A1
Authority
WO
WIPO (PCT)
Prior art keywords
eye
amplitude value
face
closed
key points
Prior art date
Application number
PCT/CN2020/098066
Other languages
English (en)
French (fr)
Inventor
章菲倩
刘更代
Original Assignee
广州市百果园信息技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 广州市百果园信息技术有限公司 filed Critical 广州市百果园信息技术有限公司
Priority to US17/625,947 priority Critical patent/US11775059B2/en
Publication of WO2021004285A1 publication Critical patent/WO2021004285A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/013—Eye tracking input arrangements
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T13/00—Animation
    • G06T13/20—3D [Three Dimensional] animation
    • G06T13/40—3D [Three Dimensional] animation of characters, e.g. humans, animals or virtual beings
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T13/00—Animation
    • G06T13/80—2D [Two Dimensional] animation, e.g. using sprites
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/161—Detection; Localisation; Normalisation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/168—Feature extraction; Face representation
    • G06V40/171—Local features and components; Facial parts ; Occluding parts, e.g. glasses; Geometrical relationships
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/18—Eye characteristics, e.g. of the iris
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/18—Eye characteristics, e.g. of the iris
    • G06V40/193—Preprocessing; Feature extraction

Definitions

  • the embodiments of the present application relate to the field of image processing technology, such as a method for determining the degree of human eye closure, an eye control method, a device for determining the degree of human eye closure, an eye control device, equipment, and storage medium.
  • Blinking is one of the actions of the eyes.
  • the blinking action can be used to control the facial model to blink, and it can also trigger related instructions to perform corresponding operations by blinking.
  • one way to detect the blinking action is to recognize the blinking action, that is, to identify the two node states of open eyes and closed eyes to determine whether there is a blinking action.
  • the facial expressions in the data are detected and tracked, and the human expressions are transferred to different faces, which can also detect the blinking of the faces.
  • the above two methods can perform blink detection, they do not measure the degree of closure of the human eye during the blinking of the human eye, so that the blink detection cannot be applied to realize related application control scenarios based on the degree of closure of the human eye.
  • the embodiments of the present application provide a method for determining the degree of human eye closure, an eye control method, a device for determining the degree of human eye closure, an eye control device, equipment, and storage medium, so as to solve the problem that the detection of human eye blinking cannot be applied to human eyes.
  • the degree of closure realizes the problems related to application control scenarios.
  • the embodiment of the present application provides a method for determining the degree of human eye closure, including:
  • the closed eye weight of the human eye in the face image is calculated based on the relative amplitude value and the maximum relative amplitude value, and the closed eye weight is used to measure the degree of closure of the human eye.
  • the embodiment of the present application provides an eye control method, including:
  • the closed eye weight is determined by the method for determining the degree of human eye closed described in any embodiment of the present application.
  • the embodiment of the present application provides an eye control method, including:
  • Playing video data the video data has multiple frames of image data, and the image data has human face images;
  • the embodiment of the present application provides a device for determining the degree of human eye closure, including:
  • the face image acquisition module is set to acquire the face image
  • a face data determining module configured to determine the eye opening amplitude value and the reference distance in the face image
  • a relative amplitude value calculation module configured to calculate the relative amplitude value of the human eye opening amplitude value with respect to the reference distance
  • the maximum relative amplitude value acquisition module set to obtain the maximum relative amplitude value
  • the closed eye weight calculation module is configured to calculate the closed eye weight of the human eye in the face image based on the relative amplitude value and the maximum relative amplitude value, and the closed eye weight is used to measure the degree of closure of the human eye.
  • An embodiment of the application provides an eye control device, including:
  • Face image and face model acquisition module set to face image and face model
  • a closed eye weight acquisition module configured to acquire the closed eye weight of the human eye in the face image, and the closed eye weight is used to measure the degree of closure of the human eye;
  • An eye control module configured to control the eyes in the face model based on the closed eye weight
  • the closed eye weight is determined by the device for determining the degree of human eye closed described in any embodiment of the present application.
  • An embodiment of the application provides an eye control device, including:
  • a playing module configured to play video data, the video data has multiple frames of image data, and the image data has a face image;
  • a face model display module configured to display a face model to cover the face image
  • the face data determining module is configured to determine the eye opening amplitude value and the reference distance in each frame of the face image
  • a relative amplitude value calculation module configured to calculate the relative amplitude value of the human eye opening amplitude value with respect to the reference distance
  • the maximum relative amplitude value acquisition module set to obtain the maximum relative amplitude value
  • a closed eye weight calculation module configured to calculate the closed eye weight of the human eye in the face image based on the relative amplitude value and the maximum relative amplitude value;
  • the model driving module is configured to drive the eyes in the face model to blink based on the closed eye weight.
  • An embodiment of the present application provides a device, and the device includes:
  • One or more processors are One or more processors;
  • Storage device set to store one or more programs
  • the one or more processors When the one or more programs are executed by the one or more processors, the one or more processors implement the method for determining the degree of human eye closure and/or eye control according to any embodiment of the present application method.
  • An embodiment of the present application provides a computer-readable storage medium on which a computer program is stored.
  • the computer program is executed by a processor, the method for determining the degree of human eye closure and/or the eye closure described in any of the embodiments of the present application is realized. Control Method.
  • FIG. 1 is a flowchart of a method for determining the degree of human eye closure provided in Embodiment 1 of the present application;
  • 2A is a flow chart of a method for determining the degree of human eye closure provided in the second embodiment of the present application
  • 2B is a schematic diagram of key points of a human face in an embodiment of the present application.
  • FIG. 3 is a flowchart of an eye control method provided in Embodiment 3 of the present application.
  • FIG. 4A is a flowchart of an eye control method provided by Embodiment 4 of the present application.
  • 4B is a schematic diagram of the blinking effect of the face model in the embodiment of the present application.
  • FIG. 5 is a structural block diagram of a device for determining the degree of human eyes closed eyes provided by Embodiment 5 of the present application;
  • FIG. 6 is a structural block diagram of an eye control device provided by Embodiment 6 of the present application.
  • FIG. 7 is a structural block diagram of an eye control device provided by Embodiment 7 of the present application.
  • FIG. 8 is a structural block diagram of a device provided in Embodiment 8 of the present application.
  • FIG. 1 is a flowchart of a method for determining the degree of human eye closure provided in the first embodiment of the application.
  • the embodiment of this application can be applied to the situation of determining the degree of human eye closure.
  • the method can be performed by a device for determining the degree of human eye closure. Execution, the device can be implemented by software and/or hardware, and integrated in the device that executes the method. As shown in Figure 1, the method may include the following steps:
  • the face image may be the user's facial image collected by the camera when the user takes a selfie, short video or live broadcast through a terminal equipped with a camera.
  • the camera collects the image in real time And display the preview image on the display screen of the terminal, you can get the user's face image in the current preview image, or when the user shoots a short video or live broadcast, you can get the user's face in the current video frame in the video data image.
  • the face image may also be a face image pre-stored in the terminal, or a face image in a video frame when the video data is played. The embodiment of the present application does not limit the manner of obtaining the face image.
  • S120 Determine a human eye opening amplitude value and a reference distance in the face image.
  • the face key point detection can be performed on the face image, the face key point is obtained, the eye key point and the reference key point are determined from the face key point, and then the human eye is determined according to the eye key point
  • the opening amplitude value which expresses the opening amplitude of the human eye in the human eye image.
  • the reference key points can be the key points of relatively fixed facial features on the human face. For example, if the nose on the face is fixed, you can use the nose key points as the reference key points to calculate the reference distance, for example, to calculate the nose vertex The distance to the tip of the nose is taken as the reference distance.
  • the reference key point can also be the key point of the corners of the left and right eyes on the human face. The distance between the corners of the left and right eyes is the reference distance. Those skilled in the art will implement this application. For example, the distance between any two relatively fixed points on the face can be selected as the reference distance, which is not limited in the embodiment of the present application.
  • the ratio between the eye opening amplitude value and the reference distance can be calculated, and the ratio can be used as the relative amplitude value of the eye opening amplitude value to the reference distance, that is, the reference distance is used as a reference to measure the face by the relative amplitude value
  • the opening range of the human eye in the image can be calculated, and the ratio can be used as the relative amplitude value of the eye opening amplitude value to the reference distance, that is, the reference distance is used as a reference to measure the face by the relative amplitude value The opening range of the human eye in the image.
  • the maximum relative amplitude value can be the relative amplitude value of the human eye opening amplitude value relative to the reference distance when the human eye opening amplitude value is the maximum value, that is, the human eye opening amplitude value when the human eye opens to the maximum relative to the reference
  • the relative magnitude of the distance It is possible to obtain multiple face images of the same face, calculate relative amplitude values based on the multiple face images to obtain multiple relative amplitude values, and determine the maximum value from them as the maximum relative amplitude value.
  • the maximum relative amplitude value may also be estimated after calculating the width of the human eye, or the maximum relative amplitude value may be set based on experience, etc.
  • the embodiment of the present application does not limit the manner of obtaining the maximum relative amplitude value.
  • the closed eye amplitude value of the human eye in the face image can be calculated first.
  • the closed eye amplitude value can be the eye opening amplitude value of the human eye in a certain state and the human eye when the human eye is fully opened during the blinking of the human eye.
  • the ratio of the opening amplitude value In the embodiment of the present application, the relative amplitude value and the maximum relative amplitude value are both relative to the reference distance, and the closed eye amplitude value may be the ratio of the relative amplitude value and the maximum relative amplitude value.
  • the closed-eye weight can be calculated by the closed-eye amplitude value.
  • the closed-eye weight and the closed-eye amplitude are negatively correlated, and the closed-eye weight expresses the human eye in the face image.
  • the degree of closure the closer the human eye is to a completely closed state, the smaller the value of the closed eye, the greater the weight of the closed eye.
  • the related application can be controlled based on the closed eye weight.
  • the eyes of the virtual face model can be controlled based on the closed eye weight of the face image, so that the face model can follow the person.
  • the blinking of the face realizes the blinking action, or based on the closed eye weight of the human eye in the face image, the instruction is triggered to execute the corresponding operation when the closed eye weight is greater than the preset value, etc.
  • the embodiment of the application does not apply the closed eye weight limit.
  • the embodiment of the present application determines the relative amplitude value according to the opening amplitude value of the human eye and the reference distance, and calculates the closed eye weight of the human eye based on the relative amplitude value and the maximum relative amplitude value to measure the degree of closure of the human eye.
  • the problem of measuring the degree of human eye closure When the weight of human eye closure is used for related application control, related application control can be performed according to the degree of human eye closure, making human eye detection suitable for related application control based on the degree of human eye closure Scenes.
  • FIG. 2A is a flowchart of a method for determining the degree of human eye closure provided in Embodiment 2 of this application. The embodiment of this application is described on the basis of Embodiment 1. As shown in FIG. 2A, the method may include the following steps :
  • S220 Perform face key point detection on the face image to obtain face key points.
  • Face key point detection also known as face key point positioning or face alignment, refers to a given face image, locating key areas of the face, including key areas such as eyebrows, eyes, nose, mouth, and facial contours .
  • Face key points can be extracted through a pre-trained face key point detection model. For example, you can collect a large number of face images and mark the key points on the face images to form training data. After training the model with the training data, The face image input the trained face key point detection model to obtain the face key points of the face image.
  • Figure 2B shows a schematic diagram of the key points of a human face.
  • key points including facial contours (point 0-point 16) and left and right eyebrow key points (point 17-point 26) can be extracted.
  • Nose key points point 27-point 35
  • left and right eye key points point 36-point 47
  • mouth key points point 48-point 59
  • Fig. 2B is an example of the key points of a human face. In practical applications, other key points of the face such as cheekbones and ears can also be added. The embodiment of the present application does not limit the position and number of the key points of the human face.
  • S230 Determine the key points of the human eyes from the key points of the face, and determine the reference key points from the key points of the face.
  • the key points of the eye can be determined from the key points of the face, and the key points of the top of the eye and the key points of the fundus among the key points of the eye are selected as the key points of the human eye, and the key points of the human face
  • the key points of the nose are determined from the points, and the key points of the top of the nose and the key points of the tip of the nose are selected as the reference key points.
  • points 42-47 are key points for the left eye
  • points 36-41 are key points for the right eye
  • points 27-35 are key points for the nose.
  • the key point 27 on the top of the nose and the key point 33 on the tip of the nose can be selected as reference key points.
  • S240 Calculate a human eye opening amplitude value based on the key point of the human eye.
  • the eye opening amplitude value expresses the degree of eye opening, that is, it can be the distance from the top of the upper eyelid to the bottom of the lower eyelid, which corresponds to the key point of the human face, The distance from the key point on the top of the eye to the key point on the fundus can be calculated as the eye opening amplitude value.
  • the eye opening amplitude value of the left eye is the distance between point 44 and point 46, that is
  • the eye opening amplitude value of the right eye Is the distance from point 37 to point 41, that is
  • the reference points are the key point of the nose tip 27 and the key point 33 of the nose tip, and the distance from the key point of the nose tip to the key point of the nose tip can be calculated as the reference distance, that is, the reference distance is from point 27 to The distance to point 33
  • the distance from the top of the nose to the tip of the nose is selected as the reference distance.
  • the key points of the top of the nose and the key points of the tip of the nose can be easily detected regardless of whether the face is on the front or the side, and the positions of the top of the nose and the tip of the nose are relatively fixed.
  • the point is used as a reference point to calculate the reference distance, which can improve the accuracy of the reference distance and further improve the accuracy of the subsequent calculation of the closed eye weight to achieve precise control of blinking.
  • S260 Calculate the relative amplitude value of the human eye opening amplitude value with respect to the reference distance.
  • the relative amplitude value is the ratio between the human eye opening amplitude value and the reference distance, and the calculation method is as follows:
  • r l,i are the relative amplitude values of the left eye
  • r r,i are the relative amplitude values of the right eye. It can be known from the above two formulas that the reference distance
  • the closed eye amplitude value of the left eye is the relative amplitude value r l of the left eye , i and the maximum relative amplitude value of the left eye
  • the ratio of the left eye closed eye amplitude is
  • the closed eye amplitude value of the right eye is the relative amplitude value of the right eye r r, i and the maximum relative amplitude value of the right eye
  • the ratio of the right eye closed eye amplitude is
  • the closed eye amplitude values of the left and right eyes are positively correlated with the relative amplitude value, and negatively correlated with the maximum relative amplitude value.
  • the eye amplitude value is close to 1, it indicates that the human eye is in a fully opened state, and when the eye closed amplitude value is close to 0, it indicates that the human eye is in a completely closed state.
  • the closed eye constant ⁇ can be 0.3, when the left eye closed eye amplitude value , It is considered that the left eye has reached a completely closed eye state, when the right eye is closed When, it is considered that the right eye has reached a completely closed eye state.
  • the closed eye weight of the human eye in the face image i can be calculated by the following formula:
  • w l,i are the closed eye weights of the left eye
  • w r,i are the closed eye weights of the right eye. From the above formula, we can see that taking the left eye as an example, the more the left eye is closed, the relative amplitude value of the left eye r l , the smaller the i , the closed eye amplitude value of the left eye The smaller the weight of the left eye, w l,i is the larger.
  • the embodiment of the application performs face key point detection on the face to obtain the face key points, and calculates the eye opening amplitude value in the face image according to the eye top key points and the fundus key points in the face key points, and according to the nose
  • the top key point and the nose tip key point calculate the reference distance to determine the relative amplitude value according to the eye opening amplitude value and the reference distance, and calculate the closed eye weight of the human eye based on the relative amplitude value and the maximum relative amplitude value, which solves the problem of human eye detection.
  • the problem of measuring the degree of human eye closure When the weight of human eye closure is used for related application control, related application control can be performed according to the degree of human eye closure, making human eye detection suitable for related application control based on the degree of human eye closure Scenes.
  • the distance from the top of the nose to the tip of the nose is selected as the reference distance. No matter the face is on the front or the side, it is easy to detect the key points of the top of the nose and the key points of the tip of the nose, and the positions of the top of the nose and the tip of the nose are relatively fixed, compared to other key points. Calculating the reference distance as a reference point can improve the accuracy of the reference distance, thereby improving the accuracy of the closed eye weight.
  • FIG. 3 is a flowchart of an eye control method provided in the third embodiment of the application.
  • the embodiment of the application can be applied to the situation of controlling the eyes in the face model according to the human eyes in the face image.
  • the method can be controlled by the eyes
  • the device can be implemented by means of software and/or hardware and integrated in the device for executing the method. As shown in Fig. 3, the method may include the following steps:
  • the face image may be the user's facial image collected by the camera when the user takes a selfie, short video or live broadcast through a terminal equipped with a camera.
  • the camera collects the image in real time And display the preview image on the display screen of the terminal, you can get the user's face image in the current preview image, or when the user shoots a short video or live broadcast, you can get the user's face in the current video frame in the video data image.
  • the face image may also be a face image pre-stored in the terminal, or a face image in a video frame when the video data is played. The embodiment of the present application does not limit the manner of obtaining the face image.
  • the face model can be a texture model or other types of models.
  • the face model can be a face model selected by the user.
  • applications such as selfies, short videos, and live broadcasts provide multiple types of face models to For the user to choose, when the user performs a selection operation from the provided facial models, the corresponding facial model can be obtained according to the user’s selection operation.
  • the facial model can be a human face model, an animal face model, Cartoon face models, etc.
  • the face model is used to control the eyes on the face model according to the eyes closed weight in the face image.
  • the face model after adjusting the eyes can be covered in the selfie preview in the form of textures , Short videos or face images in live broadcasts to achieve a texture effect.
  • the way of weighting the closed eyes of the human eyes can refer to Embodiment 1 or Embodiment 2, which will not be described in detail here.
  • the preset opening amplitude value of the eyes in the face model may be obtained first, and the target opening amplitude value of the eyes is calculated based on the closed eye weight and the preset opening amplitude value, and the face model Adjust the opening amplitude value of the eyes in the middle to the target opening amplitude value to complete the eye control of the face model.
  • the initial state of the eyes in the face model is a fully opened state
  • the preset opening amplitude value is the opening amplitude value when the eyes in the face model are fully opened.
  • the model parameters of the model can be stored in advance.
  • the model parameters can include the opening range value when the eyes are fully opened, the distance from the tip of the nose to the top of the nose and other parameters, and the preset opening range can be read from the model parameters value.
  • the product of the closed eye weight and the preset opening amplitude value can be calculated as the target opening amplitude value.
  • the default opening amplitude value in the initial face model is that the eyes can be opened to The maximum distance from the apex of the eye to the fundus point.
  • the target opening amplitude value is half of the preset opening amplitude value.
  • the face model may include a first initial face model and a second initial face model.
  • the first initial face model is the face model when the eyes are completely opened
  • the second initial face model is the face model when the eyes are completely closed. Then, the target opening amplitude value can be obtained by performing interpolation calculation by combining the closed eyes weight with the first initial face model and the second initial face model.
  • the eyes in the face model can be adjusted according to the target opening amplitude value so that the eye opening amplitude value is equal to the target opening amplitude value.
  • the upper eyelid of the eyes in the face model can be adjusted Position, or drive the face model to deform by the target opening amplitude value to obtain the adjusted face model, so as to complete the control of the eyes in the face model.
  • the eyes in the face model can be adjusted through a face image to obtain the adjusted face model.
  • the selfie preview, short video or live broadcast shows multiple continuous video frames, then Multiple frames of continuous face images can be obtained.
  • the eyes in the face model are adjusted in real time through the multiple frames of face images, so that the face model can simulate the blinking action.
  • the closed eye weight of the human eye in the face image is obtained.
  • the closed eye weight is used to measure the degree of closure of the human eye, and then the weight in the face model is calculated based on the closed eye weight Eye control. It solves the problem that human eye detection cannot measure the degree of human eye closure.
  • the eye closure weight is used to control the eyes in the face model, the eyes of the face model can be controlled according to the degree of human eye closure, so that the face model It can simulate the real blinking action of a human face, thereby making the facial expressions of the facial model more realistic without requiring a large number of facial images, and the implementation process is relatively simple.
  • Embodiment 4A is a flowchart of an eye control method provided by Embodiment 4 of this application.
  • This embodiment of the present application is applicable to a situation where the eyes of a facial model are controlled to blink according to a face image in the video data when video data is played.
  • the method can be executed by an eye control device, which can be implemented by software and/or hardware, and integrated in the device that executes the method, as shown in FIG. 4A, the method may include the following steps:
  • Play video data where the video data has multiple frames of image data, and the image data has a face image.
  • the video data may be preview video data, short video data, or live video data formed after the user collects images during self-portrait.
  • Video data includes multiple frames of image data, and the image data includes face images.
  • the face model can be an expression model selected by the user in selfies, short videos or live broadcasts.
  • the face model is used to cover the face image displayed in the video playback interface when playing video data, and the human eyes in the face image Simulate blinking under the driving of, such as overlaying the face image in the form of a texture, and realize the simulated blinking under the driving of the face image.
  • a cartoon face model is used to cover the user's face image in the video.
  • the face key points can be detected on the face image, the face key points are obtained, the eye key points and reference key points are determined from the face key points, and then the human eyes are determined according to the eye key points
  • the opening amplitude value which expresses the opening amplitude of the human eye in the human eye image.
  • the reference key points can be the key points of relatively fixed facial features on the human face. For example, if the nose on the human face is fixed, the nose key points can be used as the reference key points to calculate the reference distance. Optionally, calculate The distance between the apex of the nose and the tip of the nose is used as the reference distance.
  • the reference key point can also be the key point of the corners of the left and right eyes on the human face.
  • the distance between the corners of the left and right eyes is the reference distance.
  • the distance between any two relatively fixed points on the human face can be selected as the reference distance, which is not limited in the embodiment of the present application.
  • the ratio between the eye opening amplitude value and the reference distance can be calculated, and the ratio can be used as the relative amplitude value of the eye opening amplitude value to the reference distance, that is, the reference distance is used as a reference to measure the face by the relative amplitude value
  • the opening range of the human eye in the image can be calculated, and the ratio can be used as the relative amplitude value of the eye opening amplitude value to the reference distance, that is, the reference distance is used as a reference to measure the face by the relative amplitude value The opening range of the human eye in the image.
  • the video data is played in frames, and the relative amplitude value of the eye opening amplitude value relative to the reference distance in the multi-frame face image can be obtained, and then the maximum value is determined from the multiple relative amplitude values. And take the maximum value as the maximum relative amplitude value. For example, for the current frame of image data, it is possible to obtain consecutive N frames of image data adjacent to the current frame, calculate a relative amplitude value for all face images in the N frame of image data, obtain multiple relative amplitude values, and select multiple relative amplitudes The maximum value is the maximum relative amplitude value.
  • S460 Calculate the closed eye weight of the human eye in the face image based on the relative amplitude value and the maximum relative amplitude value.
  • the closed eye amplitude value of the human eye in the face image can be calculated first.
  • the closed eye amplitude value can be the eye opening amplitude value of the human eye in a certain state and the human eye when the human eye is fully opened during the blinking of the human eye.
  • the ratio of the opening amplitude value In the embodiment of the application, the relative amplitude value and the maximum relative amplitude value are both the ratio relative to the reference distance, and the closed eye amplitude value may be the ratio of the relative amplitude value and the maximum relative amplitude value.
  • the eye amplitude value expresses the degree of eye closure in a certain state during the blinking process, and the eye closed amplitude value is less than or equal to 1 value.
  • the closed-eye weight can be calculated by the closed-eye amplitude value.
  • the closed-eye weight and the closed-eye amplitude are negatively correlated, and the closed-eye weight expresses the use of human eyes in the face image.
  • the displayed face model is blinked according to the face image in each frame of video data. For example, multiple frames of face images that are continuous in time can be obtained, and each frame of face image determines A closed eye weight can drive the face model displayed in the playback interface to blink according to the closed eye weight.
  • Figure 4B shows the process and effect of controlling the blink of the face model according to the face image in the video data.
  • the face model in the video data blinks
  • the face model overlaid on the face image follow the face image to simulate blinking.
  • the closed eye weight of the human eyes in the face image can be calculated from the face image in the video data, thereby driving the displayed face according to the closed eye weight
  • the model blinks, which solves the problem that human eye detection cannot measure the degree of human eye closure. It can drive the facial model to simulate blinking through the degree of eye closure during the blinking process of the face, so that the facial model can simulate the real blinking action of the face, thereby making
  • the facial expressions of the facial model are more realistic, and there is no need for a large number of face images, the implementation process is relatively simple, the calculation speed is fast, and a smoother blink effect can be obtained.
  • FIG. 5 is a structural block diagram of a device for determining the degree of human eye closure provided in the fifth embodiment of the present application.
  • the device for determining the degree of human eye closure in an embodiment of the present application may specifically include the following modules: a face image acquisition module 501 configured to Obtain a face image; face data determination module 502, set to determine the eye opening amplitude value and reference distance in the face image; relative amplitude value calculation module 503, set to calculate the eye opening amplitude value The relative amplitude value relative to the reference distance; the maximum relative amplitude value obtaining module 504 is set to obtain the maximum relative amplitude value; the closed eye weight calculation module 505 is set to calculate based on the relative amplitude value and the maximum relative amplitude value The closed eye weight of the human eye in the face image, and the closed eye weight is used to measure the degree of closure of the human eye.
  • a face image acquisition module 501 configured to Obtain a face image
  • face data determination module 502 set to determine the eye opening amplitude value and reference distance
  • the face data determining module 502 includes: a face key point detection submodule, configured to perform face key point detection on the face image to obtain face key points; human eye key points and reference keys
  • the point determination sub-module is set to determine the human eye key points from the face key points and the reference key points from the face key points;
  • the human eye opening amplitude value calculation sub-module is set to be based on The key point of the human eye calculates the value of the opening amplitude of the eye;
  • the reference distance calculation sub-module is set to calculate the reference distance based on the reference key point.
  • the human eye key point and reference key point determination sub-module includes: an eye key point determination unit configured to determine the eye key point from the face key point; and a human eye key point selection unit, It is set to select the key points on the top of the eye and the key points on the fundus among the key points of the eye as the key points of the human eye.
  • the human eye key points include eye top key points and fundus key points
  • the human eye opening amplitude value calculation sub-module includes: a first distance calculation unit configured to calculate the distance between the eye top key points and the fundus key points. State the distance of the key point of the fundus, and use the distance from the key point on the top of the eye to the key point of the fundus as the eye opening amplitude value.
  • the human eye key point and reference key point determination sub-module includes: a nose key point determination unit configured to determine a nose key point from the face key points; a reference key point selection unit configured to The key points on the top of the nose and the key points on the tip of the nose among the key points of the nose are selected as reference key points.
  • the reference key points include key points on the top of the nose and key points on the tip of the nose
  • the reference distance calculation sub-module includes: a second distance calculation unit configured to calculate the distance from the key point on the top of the nose to the key point on the nose tip. The distance from the key point on the tip of the nose to the key point on the tip of the nose is taken as the reference distance.
  • the closed-eye weight calculation module 505 includes: a closed-eye amplitude value calculation sub-module configured to calculate the ratio between the relative amplitude value and the maximum relative amplitude value to obtain the closed-eye amplitude value of the human eye ,
  • the closed-eye amplitude value is positively correlated with the relative amplitude value, and negatively correlated with the maximum relative amplitude value;
  • the closed-eye weight calculation sub-module is set to use the closed-eye amplitude value and the preset closed-eye constant calculation station State the weight of closed eyes.
  • the device for determining the degree of human eye closure provided by the embodiment of the present application can execute the method for determining the degree of human eye closure provided by any embodiment of the present application, and has functional modules corresponding to the execution method.
  • FIG. 6 is a structural block diagram of an eye control device provided in the sixth embodiment of the present application.
  • the eye control device in the embodiment of the present application may specifically include the following modules: a face image and face model acquisition module 601, which is set as a face image and face Model; closed eye weight acquisition module 602, set to acquire the closed eye weight of the human eye in the face image, the closed eye weight is used to measure the degree of closure of the human eye; eye control module 603, set based on the closed eye The weight controls the eyes in the face model.
  • the eye closure weight is determined by the device for determining the degree of human eye closure described in Embodiment 5 of the present application.
  • the eye control module 603 includes: a preset opening amplitude value obtaining submodule, configured to obtain the preset opening amplitude value of the eyes in the initial face model; a target opening amplitude value calculation submodule, Set to calculate the target opening amplitude value of the eye based on the closed eye weight and the preset opening amplitude value; the adjustment sub-module is set to adjust the opening amplitude value of the eyes in the initial face model to be The target opening amplitude value.
  • a preset opening amplitude value obtaining submodule configured to obtain the preset opening amplitude value of the eyes in the initial face model
  • a target opening amplitude value calculation submodule Set to calculate the target opening amplitude value of the eye based on the closed eye weight and the preset opening amplitude value
  • the adjustment sub-module is set to adjust the opening amplitude value of the eyes in the initial face model to be The target opening amplitude value.
  • the target opening amplitude value calculation sub-module includes: a target opening amplitude value determining unit, configured to determine that the target opening amplitude value is 0 when the eye closure weight is greater than a preset value .
  • the eye control device provided in the embodiment of the present application can execute the eye control method provided in the third embodiment of the present application, and has functional modules corresponding to the execution method.
  • FIG. 7 is a structural block diagram of an eye control device provided in the seventh embodiment of the present application.
  • the eye control device in the embodiment of the present application may specifically include the following modules: a playback module 701 configured to play video data, the video data having multiple Frame image data, the image data has a face image; the face model display module 702 is set to display the face model to cover the face image; the face data determination module 703 is set to display the face image in each frame In the face image, determine the eye opening amplitude value and the reference distance; the relative amplitude value calculation module 704 is configured to calculate the relative amplitude value of the human eye opening amplitude value with respect to the reference distance; the maximum relative amplitude value is obtained Module 705, set to obtain the maximum relative amplitude value; closed eye weight calculation module 706, set to calculate the closed eye weight of the human eyes in the face image based on the relative amplitude value and the maximum relative amplitude value; model driving module 707. Set to drive the eyes in the face model to blink based on the closed eye weight.
  • the maximum relative amplitude value acquisition module 705 includes: a relative amplitude value acquisition submodule configured to acquire the relative amplitude value of the eye opening amplitude value with respect to the reference distance in the multi-frame face image;
  • the maximum relative amplitude value determining sub-module is configured to determine the maximum value from a plurality of relative amplitude values, and use the maximum value as the maximum relative amplitude value.
  • the eye control device provided in the embodiment of the present application can execute the eye control method provided in the fourth embodiment of the present application, and has functional modules corresponding to the execution method.
  • the device may specifically include: a processor 80, a memory 81, a display screen 82 with a touch function, an input device 83, an output device 84, and a communication device 85.
  • the number of processors 80 in the device may be one or more, and one processor 80 is taken as an example in FIG. 8.
  • the number of memories 81 in the device may be one or more.
  • One memory 81 is taken as an example in FIG. 8.
  • the processor 80, the memory 81, the display screen 82, the input device 83, the output device 84, and the communication device 85 of the device may be connected by a bus or other means. In FIG. 8, the connection by a bus is taken as an example.
  • the memory 81 can be configured to store software programs, computer-executable programs, and modules, such as the program instructions/modules corresponding to the method for determining the degree of human eye closure described in Embodiment 1 or 2 of this application (for example, the face image acquisition module 501, the face data determination module 502, the relative amplitude value calculation module 503, the maximum relative amplitude value acquisition module 504, and the closed eye weight calculation module 505 in the above-mentioned device for determining the degree of human eye closure), or Program instructions/modules corresponding to the eye control method described in the third embodiment of the present application (for example, the face image and face model acquisition module 601, the closed eye weight acquisition module 602, and the eye control module 603 in the aforementioned eye control device) , Or the program instructions/modules corresponding to the eye control method described in the fourth embodiment of the application (for example, the playback module 701, the face model display module 702, the face data determination module 703, the relative amplitude value in the above eye control
  • the memory 81 may mainly include a storage program area and a storage data area.
  • the storage program area may store an operating device and an application program required by at least one function; the storage data area may store data created according to the use of the device, etc.
  • the memory 81 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other non-volatile solid-state storage devices.
  • the memory 81 may further include a memory remotely provided with respect to the processor 80, and these remote memories may be connected to the device through a network. Examples of the aforementioned networks include but are not limited to the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.
  • the display screen 82 is a display screen 82 with a touch function, which may be a capacitive screen, an electromagnetic screen or an infrared screen.
  • the display screen 82 is set to display data according to the instructions of the processor 80, and is also set to receive touch operations on the display screen 82 and send corresponding signals to the processor 80 or other devices.
  • the display screen 82 is an infrared screen, it also includes an infrared touch frame, which is arranged around the display screen 82, and the display screen 82 can also be set to receive infrared signals and send the infrared signals to Processor 80 or other equipment.
  • the communication device 85 is configured to establish a communication connection with other devices, and it may be a wired communication device and/or a wireless communication device.
  • the input device 83 may be configured to receive input digital or character information, and generate key signal input related to user settings and function control of the device, and may also be a camera configured to obtain images and a sound pickup device to obtain audio data.
  • the output device 84 may include audio equipment such as speakers. The composition of the input device 83 and the output device 84 can be set according to actual conditions.
  • the processor 80 is configured to execute various functional applications and data processing of the device by running software programs, instructions, and modules stored in the memory 81, that is, to realize the above-mentioned method for determining the degree of human eye closure and/or eye control method.
  • the processor 80 is configured to implement the method for determining the degree of human eye closure and/or the method for eye control provided in the embodiment of the present application when one or more programs stored in the memory 81 are executed.
  • the embodiment of the present application also provides a computer-readable storage medium.
  • the instructions in the storage medium are executed by the processor of the device, the device can execute the method for determining the degree of face closure as described in the above method embodiment and/ Or eye control methods.
  • each part of this application can be implemented by hardware, software, firmware or a combination thereof.
  • multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution device.
  • a logic gate circuit configured to implement logic functions for data signals Discrete logic circuits, application specific integrated circuits with suitable combinational logic gate circuits, Programmable Gate Array (PGA), Field Programmable Gate Array (FPGA), etc.

Abstract

本申请实施例公开了一种人眼闭合程度的确定方法、眼睛控制方法、装置、设备和存储介质，人眼闭合程度的确定方法包括：获取人脸图像，在人脸图像中确定人眼张开幅度值和参考距离，计算人眼张开幅度值相对于参考距离的相对幅度值，基于相对幅度值和最大相对幅度值计算人脸图像中人眼的闭眼权重，闭眼权重用于度量人眼的闭合程度。

Description

人眼闭合程度的确定方法、眼睛控制方法、装置、设备和存储介质
本申请要求在2019年07月10日提交中国专利局、申请号为201910621637.5的中国专利申请的优先权，该申请的全部内容通过引用结合在本申请中。
技术领域
本申请实施例涉及图像处理技术领域，例如涉及一种人眼闭合程度的确定方法、眼睛控制方法、人眼闭合程度的确定装置、眼睛控制装置、设备和存储介质。
背景技术
随着智能终端的日渐普及，自拍、短视频、直播等娱乐应用程序得到广泛应用，在上述娱乐应用程序的使用过程中，通过拍摄者的眼睛动作进行相关的应用控制也变得越来越流行。
眨眼是眼睛的动作之一，眨眼动作可以用于控制脸部模型眨眼，还可以通过眨眼触发相关指令以执行相应的操作。然而，相关技术中，对眨眼动作进行检测的一种方式是针对眨眼动作的识别，即识别睁眼和闭眼两个节点状态以确定是否存在眨眼动作，在另一种方式中，通过对视频数据中的人脸表情进行检测与跟踪，并将人类表情迁移到不同人脸上，其同样可以检测人脸的眨眼。
上述两种方式虽然可以进行眨眼检测，但未对人眼眨眼过程中的人眼的闭合程度进行度量，导致眨眼检测无法适用于根据人眼的闭合程度实现相关的应用控制场景。
发明内容
本申请实施例提供一种人眼闭合程度的确定方法、眼睛控制方法、人眼闭合程度的确定装置、眼睛控制装置、设备和存储介质，以解决人眼眨眼动作的检测无法适用于根据人眼的闭合程度实现相关的应用控制场景的问题。
本申请实施例提供了一种人眼闭合程度的确定方法，包括：
获取人脸图像；
在所述人脸图像中确定人眼张开幅度值和参考距离；
计算所述人眼张开幅度值相对于所述参考距离的相对幅度值；
获取最大相对幅度值；
基于所述相对幅度值和所述最大相对幅度值计算所述人脸图像中人眼的闭眼权重，所述闭眼权重用于度量人眼的闭合程度。
本申请实施例提供了一种眼睛控制方法，包括：
获取人脸图像和脸部模型；
获取人脸图像中人眼的闭眼权重，所述闭眼权重用于度量人眼的闭合程度；
基于所述闭眼权重对所述脸部模型中的眼睛进行控制；
其中，所述闭眼权重通过本申请任一实施例所述的人眼闭合程度的确定方法确定。
本申请实施例提供了一种眼睛控制方法，包括：
播放视频数据，所述视频数据中具有多帧图像数据，所述图像数据中具有人脸图像；
显示脸部模型，以覆盖所述人脸图像；
在每帧所述人脸图像中，确定人眼张开幅度值和参考距离；
计算所述人眼张开幅度值相对于所述参考距离的相对幅度值；
获取最大相对幅度值；
基于所述相对幅度值和所述最大相对幅度值计算所述人脸图像中人眼的闭眼权重；
基于所述闭眼权重驱动所述脸部模型中的眼睛进行眨眼。
本申请实施例提供了一种人眼闭合程度的确定装置，包括：
人脸图像获取模块，设置为获取人脸图像；
人脸数据确定模块，设置为在所述人脸图像中确定人眼张开幅度值和参考距离；
相对幅度值计算模块，设置为计算所述人眼张开幅度值相对于所述参考距离的相对幅度值；
最大相对幅度值获取模块，设置为获取最大相对幅度值；
闭眼权重计算模块，设置为基于所述相对幅度值和所述最大相对幅度值计算所述人脸图像中人眼的闭眼权重，所述闭眼权重用于度量人眼的闭合程度。
本申请实施例提供了一种眼睛控制装置，包括：
人脸图像和脸部模型获取模块，设置为人脸图像和脸部模型；
闭眼权重获取模块，设置为获取人脸图像中人眼的闭眼权重，所述闭眼权重用于度量人眼的闭合程度；
眼睛控制模块，设置为基于所述闭眼权重对所述脸部模型中的眼睛进行控制；
其中，所述闭眼权重通过本申请任一实施例所述的人眼闭合程度的确定装置确定。
本申请实施例提供了一种眼睛控制装置，包括：
播放模块，设置为播放视频数据，所述视频数据中具有多帧图像数据，所述图像数据中具有人脸图像；
脸部模型显示模块，设置为显示脸部模型，以覆盖所述人脸图像；
人脸数据确定模块，设置为在每帧所述人脸图像中，确定人眼张开幅度值和参考距离；
相对幅度值计算模块，设置为计算所述人眼张开幅度值相对于所述参考距离的相对幅度值；
最大相对幅度值获取模块，设置为获取最大相对幅度值；
闭眼权重计算模块，设置为基于所述相对幅度值和所述最大相对幅度值计算所述人脸图像中人眼的闭眼权重；
模型驱动模块，设置为基于所述闭眼权重驱动所述脸部模型中的眼睛进行眨眼。
本申请实施例提供了一种设备，所述设备包括：
一个或多个处理器；
存储装置，设置为存储一个或多个程序，
当所述一个或多个程序被所述一个或多个处理器执行，使得所述一个或多个处理器实现本申请任一实施例所述的人眼闭合程度的确定方法和/或眼睛控制方法。
本申请实施例提供了一种计算机可读存储介质，其上存储有计算机程序，该计算机程序被处理器执行时实现本申请任一实施例所述的人眼闭合程度的确定方法和/或眼睛控制方法。
附图说明
图1是本申请实施例一提供的一种人眼闭合程度的确定方法的流程图；
图2A是本申请实施例二提供的一种人眼闭合程度的确定方法的流程图；
图2B是本申请实施例中人脸关键点的示意图；
图3是本申请实施例三提供的一种眼睛控制方法的流程图；
图4A是本申请实施例四提供的一种眼睛控制方法的流程图；
图4B是本申请实施例中脸部模型的眨眼效果的示意图；
图5是本申请实施例五提供的一种人眼闭眼程度的确定装置的结构框图；
图6是本申请实施例六提供的一种眼睛控制装置的结构框图；
图7是本申请实施例七提供的一种眼睛控制装置的结构框图；
图8是本申请实施例八提供的一种设备的结构框图。
具体实施方式
下面结合附图和实施例对本申请进行说明。此处所描述的实施例仅仅用于解释本申请，而非对本申请的限定。为了便于描述，附图中仅示出了与本申请相关的部分而非全部结构。
实施例一
图1为本申请实施例一提供的一种人眼闭合程度的确定方法的流程图，本申请实施例可适用于确定人眼闭合程度的情况，该方法可以由人眼闭合程度的确定装置来执行，该装置可以通过软件和/或硬件的方式来实现，并集成在执行本方法的设备中，如图1所示，该方法可以包括如下步骤：
S110、获取人脸图像。
在本申请实施例中，人脸图像可以是用户通过设置有摄像头的终端进行自拍、拍摄短视频或者直播时，摄像头采集到的用户的脸部图像，例如，在用户自拍时，摄像头实时采集图像并在终端的显示屏显示预览图像，则可以获取当前预览图像中的用户的人脸图像，又或者用户拍摄短视频或者直播时，则可以获取视频数据中的当前视频帧中的用户的人脸图像。人脸图像还可以是预先存储在终端中的人脸图像，或者播放视频数据时视频帧中的人脸图像，本申请实施例对获取人脸图像的方式不加以限制。
S120、在所述人脸图像中确定人眼张开幅度值和参考距离。
在获取人脸图像后，可以对人脸图像进行人脸关键点检测，获取到人脸关键点，从人脸关键点确定人眼关键点和参考关键点，然后根据人眼关键点确定人眼张开幅度值，该人眼张开幅度值表达了人眼图像中人眼的张开幅度。参考 关键点可以是人脸上较为固定的人脸特征的关键点，例如，人脸上鼻子是固定不变的，则可以将鼻子关键点作为参考关键点以计算参考距离，例如，计算鼻顶点到鼻尖点的距离作为参考距离，参考关键点还可以是人脸上左右两个眼睛的眼角的关键点，则左右两个眼睛的眼角的距离为参考距离，本领域技术人员在实施本申请实施例时，可以选择人脸上任意相对固定不变的两个点之间的距离作为参考距离，本申请实施例对此不加以限制。
S130、计算所述人眼张开幅度值相对于所述参考距离的相对幅度值。
可以计算人眼张开幅度值与参考距离之间的比值，并将该比值作为人眼张开幅度值相对于参考距离的相对幅度值，即以参考距离为参照，通过相对幅度值度量人脸图像中人眼的张开幅度。
S140、获取最大相对幅度值。
最大相对幅度值可以是人眼张开幅度值为最大值时，人眼张开幅度值相对于参考距离的相对幅度值，即人眼张开到最大时的人眼张开幅度值相对于参考距离的相对幅度值。可以获取同一人脸的多张人脸图像，根据多张人脸图像计算相对幅度值得到多个相对幅度值，从中确定出最大值作为最大相对幅度值。还可以是计算人眼宽度后估算出最大相对幅度值，还可以是根据经验设置最大相对幅度值等等，本申请实施例对获取最大相对幅度值的方式不加以限制。
S150、基于所述相对幅度值和所述最大相对幅度值计算所述人脸图像中人眼的闭眼权重，所述闭眼权重用于度量人眼的闭合程度。
可选地，可以先计算出人脸图像中人眼的闭眼幅度值，闭眼幅度值可以是人眼眨眼过程中，某一状态下人眼张开幅度值与人眼完全张开时的人眼张开幅度值的比值，在本申请实施例中，相对幅度值和最大相对幅度值均是相对于参考距离的比值，则闭眼幅度值可以是相对幅度值和最大相对幅度值的比值。
在得到闭眼幅度值后，可以通过闭眼幅度值计算闭眼权重，在本申请实施例中，闭眼权重和闭眼幅度值负相关，闭眼权重表达了人脸图像中的人眼的闭合程度，人眼越趋近于完全闭合状态，闭眼幅度值越小，闭眼权重越大。
在确定人眼的闭眼权重后，可以基于闭眼权重实现相关应用的控制，例如，可以基于人脸图像的闭眼权重对虚拟的脸部模型的眼睛进行控制，以实现脸部模型跟随人脸的眨眼实现眨眼动作，又或者基于人脸图像中人眼的闭眼权重，在闭眼权重大于预设值时触发指令执行相应的操作等，本申请实施例对闭眼权重的应用不加以限制。
本申请实施例根据人眼张开幅度值和参考距离确定相对幅度值，并基于相对幅度值和最大相对幅度值计算人眼的闭眼权重以度量人眼的闭合程度，解决 了人眼检测无法度量人眼闭合程度的问题，采用人眼的闭合权重进行相关的应用控制时，能够根据人眼的闭合程度进行相关应用控制，使得人眼检测适用于根据人眼的闭合程度进行相关应用控制的场景。
实施例二
图2A为本申请实施例二提供的一种人眼闭合程度的确定方法的流程图，本申请实施例在前述实施例一的基础上进行说明，如图2A所示，该方法可以包括如下步骤：
S210、获取人脸图像。
S220、对所述人脸图像进行人脸关键点检测，得到人脸关键点。
人脸关键点检测又称人脸关键点定位或者人脸对齐，是指给定人脸图像，定位出人脸面部的关键区域位置，包括眉毛、眼睛、鼻子、嘴巴、脸部轮廓等关键区域。
人脸关键点可以通过预先训练好的人脸关键点检测模型进行提取，例如，可以采集大量的人脸图像并在人脸图像上标注关键点后形成训练数据，通过训练数据训练模型后，将人脸图像输入训练好的人脸关键点检测模型即可以得到该人脸图像的人脸关键点。
图2B所示为人脸关键点的示意图，如图2B所示，对于人脸图像，可以提取出包括脸部轮廓关键点(点0-点16)、左右眉毛关键点(点17-点26)、鼻部关键点(点27-点35)、左右眼部关键点(点36-点47)以及嘴巴关键点(点48-点59)。
图2B为人脸关键点的一个示例，在实际应用中，还可以增加颧骨、耳朵等其他人脸部位的关键点，本申请实施例对人脸关键点的部位和数量不加以限制。
S230、从所述人脸关键点中确定出人眼关键点，以及从所述人脸关键点中确定出参考关键点。
在本申请的可选实施例中，可以从人脸关键点中确定出眼部关键点，选取眼部关键点中的眼顶关键点和眼底关键点作为人眼关键点，以及从人脸关键点中确定出鼻部关键点，选择鼻部关键点中的鼻顶关键点和鼻尖关键点作为参考关键点。
如图2B所示，在所有人脸关键点中可以确定出点42-点47为左眼关键点，点36-点41为右眼关键点，点27-点35为鼻部关键点，对于左眼，选择左眼的眼顶关键点点44和眼底关键点点46作为左眼的人眼关键点，对于右眼，选择眼顶关键点点37和眼底关键点点41作为右眼的人眼关键点，另外，可以选择 鼻顶关键点点27和鼻尖关键点点33为参考关键点。
S240、基于所述人眼关键点计算人眼张开幅度值。
在本申请实施例中，对于人脸图像i，人眼张开幅度值表达了人眼的张开程度，即可以是上眼皮顶点到下眼皮底点的距离，对应于人脸关键点上，可以计算眼顶关键点到眼底关键点的距离作为人眼张开幅度值。
如图2B所示，左眼的人眼张开幅度值为点44到点46之间的距离，即||p 44,i-p 46,i||，右眼的人眼张开幅度值为点37到点41之间的距离，即||p 37,i-p 41,i||。
S250、基于所述参考关键点计算参考距离。
如图2B所示，本申请实施例中，参考点为鼻顶关键点点27和鼻尖关键点点33，则可以计算鼻顶关键点到鼻尖关键点的距离作为参考距离，即参考距离为点27到点33的距离||p 27,i-p 33,i||。
本申请实施例中选择鼻顶到鼻尖的距离作为参考距离，无论人脸处于正面和侧面均容易检测到鼻顶关键点和鼻尖关键点，并且鼻顶和鼻尖位置相对固定，相对于采用其他关键点作为参考点计算参考距离，可以提高参考距离的准确性，进一步提高后续计算闭眼权重的准确度以实现对眨眼进行精准控制。
S260、计算所述人眼张开幅度值相对于所述参考距离的相对幅度值。
在本申请实施例中，相对幅度值为人眼张开幅度值与参考距离之间的比值，计算方式如下：
Figure PCTCN2020098066-appb-000001
上述公式中r l,i为左眼的相对幅度值，r r,i为右眼的相对幅度值，通过以上两个公式可以得知，参考距离||p 27,i-p 33,i||固定不变，相对幅度值r l,i、r r,i的大小取决于人眼的张开幅度，人眼张开幅度越大，相对幅度值越大。
S270、获取最大相对幅度值。
在实际应用中，可以通过同一人脸的多帧人脸图像获取多个相对幅度值，然后从中取最大值作为最大相对幅度值，即
Figure PCTCN2020098066-appb-000002
Figure PCTCN2020098066-appb-000003
为左眼的最大相对幅度值，
Figure PCTCN2020098066-appb-000004
为右眼的最大相对幅度值，F n为同一人脸的n帧人脸图像的集合。
S280、计算所述相对幅度值与所述最大相对幅度值之间的比值，得到人眼的闭眼幅度值，所述闭眼幅度值与所述相对幅度值正相关，与所述最大相对幅度值负相关。
在本申请实施例中，对于人脸图像i，左眼的闭眼幅度值为左眼的相对幅度值r l,i与左眼的最大相对幅度值
Figure PCTCN2020098066-appb-000005
的比值，即左眼的闭眼幅度值为
Figure PCTCN2020098066-appb-000006
右眼的闭眼幅度值为右眼的相对幅度值r r,i与右眼的最大相对幅度值
Figure PCTCN2020098066-appb-000007
的比值，即右眼的闭眼幅度值为
Figure PCTCN2020098066-appb-000008
左眼和右眼的闭眼幅度值均与相对幅度值正相关，与最大相对幅度值负相关，相对幅度值越大，闭眼幅度值也越大，人眼的闭合程度越小，当闭眼幅度值接近于1时，说明人眼处于完全张开状态，当闭眼幅度值接近于0时，说明人眼处于完全闭合状态。
S290、采用所述闭眼幅度值和预设闭眼常量计算所述闭眼权重。
在实际应用中，人脸图像的采集、人脸图像的关键点检测可能存在误差，人眼处于完全闭眼状态时眼顶关键点和眼底关键点可能无法完全重合，可以设置一个闭眼常量α，例如闭眼常量α可以为0.3，当左眼闭眼幅度值
Figure PCTCN2020098066-appb-000009
时，则认为是左眼达到完全闭眼状态，当右眼闭眼幅度值
Figure PCTCN2020098066-appb-000010
时，则认为是右眼达到完全闭眼状态。
在本申请的可选实施例中，可以通过以下公式计算人脸图像i中人眼的闭眼权重：
Figure PCTCN2020098066-appb-000011
w l,i为左眼的闭眼权重，w r,i为右眼的闭眼权重，由以上公式可知，以左眼为例，左眼闭合得越多，左眼的相对幅度值r l,i越小，左眼的闭眼幅度值
Figure PCTCN2020098066-appb-000012
也越小，而左眼的闭眼权重w l,i则越大。
本申请实施例对人脸进行人脸关键点检测得到人脸关键点，并根据人脸关键点中的眼顶关键点和眼底关键点计算人脸图像中人眼张开幅度值，以及根据 鼻顶关键点和鼻尖关键点计算参考距离，以根据人眼张开幅度值和参考距离确定相对幅度值，基于相对幅度值和最大相对幅度值计算人眼的闭眼权重，解决了人眼检测无法度量人眼闭合程度的问题，采用人眼的闭合权重进行相关的应用控制时，能够根据人眼的闭合程度进行相关应用控制，使得人眼检测适用于根据人眼的闭合程度进行相关应用控制的场景。
本申请实施例选择鼻顶到鼻尖的距离作为参考距离，无论人脸处于正面和侧面均容易检测到鼻顶关键点和鼻尖关键点，并且鼻顶和鼻尖位置相对固定，相对于采用其他关键点作为参考点计算参考距离，可以提高参考距离的准确性，进而提高闭眼权重的准确度。
实施例三
图3为本申请实施例三提供的一种眼睛控制方法的流程图，本申请实施例可适用于根据人脸图像中的人眼控制脸部模型中的眼睛的情况，该方法可以由眼睛控制装置来执行，该装置可以通过软件和/或硬件的方式来实现，并集成在执行本方法的设备中，如图3所示，该方法可以包括如下步骤：
S310、获取人脸图像和脸部模型。
在本申请实施例中，人脸图像可以是用户通过设置有摄像头的终端进行自拍、拍摄短视频或者直播时，摄像头采集到的用户的脸部图像，例如，在用户自拍时，摄像头实时采集图像并在终端的显示屏显示预览图像，则可以获取当前预览图像中的用户的人脸图像，又或者用户拍摄短视频或者直播时，则可以获取视频数据中的当前视频帧中的用户的人脸图像。人脸图像还可以是预先存储在终端中的人脸图像，或者播放视频数据时视频帧中的人脸图像，本申请实施例对获取人脸图像的方式不加以限制。
脸部模型可以是贴图模型或者其他类型的模型，可选地，脸部模型可以是用户选择的脸部模型，比如，自拍、短视频和直播等应用程序提供了多种类型的脸部模型以供用户进行选择，当用户从提供的脸部模型中执行选择操作后，可以根据用户的选择操作获取相应的脸部模型，该脸部模型可以是人的脸部模型、动物的脸部模型、卡通脸部模型等，脸部模型用于根据人脸图像中的人眼的闭眼权重对脸部模型上的眼睛进行控制，对眼睛调整后的脸部模型可以以贴图的形式覆盖在自拍预览、短视频或者直播中的人脸图像上，以实现贴图效果。
S320、获取人脸图像中人眼的闭眼权重，所述闭眼权重用于度量人眼的闭合程度。
在本申请实施例中，人眼的闭眼权重的方式可以参考实施例一或者实施例二，在此不再详述。
S330、基于所述闭眼权重对所述脸部模型中的眼睛进行控制。
在本申请的可选实施例中，可以先获取脸部模型中眼睛的预设张开幅度值，基于闭眼权重和预设张开幅度值计算眼睛的目标张开幅度值，将脸部模型中的眼睛的张开幅度值调整为目标张开幅度值以完成对脸部模型的眼睛控制。
在实际应用中，脸部模型中的眼睛的初始状态为处于完全张开的状态，预设张开幅度值为脸部模型中眼睛完全张开时的张开幅度值。对于一个脸部模型，可以预先存储该模型的模型参数，模型参数可以包括眼睛完全张开时的张开幅度值、鼻尖到鼻顶的距离等参数，则可以从模型参数中读取预设张开幅度值。
在获取预设张开幅度值后，可以计算闭眼权重和预设张开幅度值的乘积作为目标张开幅度值，例如，初始脸部模型中预设张开幅度值为眼睛能够张开到最大时眼顶点到眼底点的距离，当闭眼权重为0.5时，目标张开幅度值为预设张开幅度值的一半。
在本申请的可选实施例中，根据闭眼权重的计算公式可知，人眼闭合幅度值
Figure PCTCN2020098066-appb-000013
或
Figure PCTCN2020098066-appb-000014
越小，说明r r,i或r l,i越小，即眼顶关键点到眼底关键点的距离越小，眼睛闭合得越多，闭眼权重越大，在闭眼权重大于预设值时，确定目标张开幅度值为0，即眼睛完全闭合。
脸部模型可以包括第一初始脸部模型和第二初始脸部模型，第一初始脸部模型为眼睛完全张开时的脸部模型，第二初始脸部模型为眼睛完全闭合时的脸部模型，则可以通过闭眼权重结合第一初始脸部模型和第二初始脸部模型进行插值计算得到目标张开幅度值。
在得到目标张开幅度值后，可以根据目标张开幅度值调整脸部模型中的眼睛，使得眼睛张开幅度值等于目标张开幅度值，例如，可以调整脸部模型中眼睛的上眼皮的位置，或者通过目标张开幅度值驱动脸部模型变形得到调整后的脸部模型，以完成对脸部模型中的眼睛进行控制。
在实际应用中，通过一帧人脸图像可以对脸部模型中的眼睛进行调整得到调整后的脸部模型，当自拍预览、拍摄短视频或者直播时显示的是多帧连续的视频帧，则可以获得多帧连续的人脸图像，视频中的人脸眨眼时，通过多帧人脸图像实时调整脸部模型中的眼睛，从而实现脸部模型模拟眨眼动作。
本申请实施例在获取人脸图像和脸部模型后，获取人脸图像中人眼的闭眼权重，闭眼权重用于度量人眼的闭合程度，然后基于闭眼权重对脸部模型中的眼睛进行控制。解决了人眼检测无法度量人眼闭合程度的问题，在采用人眼的 闭合权重对脸部模型中的眼睛进行控制时，能够根据人眼的闭合程度控制脸部模型的眼睛，使得脸部模型能够模拟人脸的真实眨眼动作，进而使得脸部模型的表情更逼真，也无需大量人脸图像，实现过程也较为简单。
实施例四
图4A为本申请实施例四提供的一种眼睛控制方法的流程图，本申请实施例可适用于在播放视频数据时，根据视频数据中的人脸图像控制脸部模型的眼睛进行眨眼的情况，该方法可以由眼睛控制装置来执行，该装置可以通过软件和/或硬件的方式来实现，并集成在执行本方法的设备中，如图4A所示，该方法可以包括如下步骤：
S410、播放视频数据，所述视频数据中具有多帧图像数据，所述图像数据中具有人脸图像。
在本申请实施例中，视频数据可以是用户自拍时采集图像后形成的预览视频数据、短视频数据或者直播视频数据。视频数据包括多帧图像数据，图像数据中包含人脸图像。
S420、显示脸部模型，以覆盖所述人脸图像。
脸部模型可以是用户在自拍、拍摄短视频或者直播中选择的表情模型，脸部模型用于在播放视频数据时覆盖视频播放界面中显示的人脸图像，并在人脸图像中的人眼的驱动下模拟眨眼，例如以贴图的形式覆盖在人脸图像上，并在人脸图像的驱动下实现模拟眨眼。
如图4B所示，在播放视频数据时，采用了卡通脸部模型覆盖了视频中用户的人脸图像。
S430、在每帧所述人脸图像中，确定人眼张开幅度值和参考距离。
对于每帧人脸图像，可以对人脸图像进行人脸关键点检测，获取到人脸关键点，从人脸关键点确定人眼关键点和参考关键点，然后根据人眼关键点确定人眼张开幅度值，该人眼张开幅度值表达了人眼图像中人眼的张开幅度。参考关键点可以是人脸上较为固定的人脸特征的关键点，例如，人脸上鼻子是固定不变的，则可以将鼻子关键点作为参考关键点以计算参考距离，可选地，计算鼻顶点到鼻尖点的距离作为参考距离，参考关键点还可以是人脸上左右两个眼睛的眼角的关键点，则左右两个眼睛的眼角的距离为参考距离，本领域技术人员在实施本申请实施例时，可以选择人脸上任意相对固定不变的两个点之间的距离作为参考距离，本申请实施例对此不加以限制。
S440、计算所述人眼张开幅度值相对于所述参考距离的相对幅度值。
可以计算人眼张开幅度值与参考距离之间的比值，并将该比值作为人眼张开幅度值相对于参考距离的相对幅度值，即以参考距离为参照，通过相对幅度值度量人脸图像中人眼的张开幅度。
S450、获取最大相对幅度值。
在本申请实施例中，视频数据按帧播放，则可以获取多帧人脸图像中人眼张开幅度值相对于参考距离的相对幅度值，然后从多个相对幅度值中确定出最大值，并将所述最大值作为最大相对幅度值。例如，对于当前帧图像数据，可以获取当前帧临近的连续N帧图像数据，对N帧图像数据中的人脸图像均计算出一个相对幅度值，得到多个相对幅度值，选择多个相对幅度值的最大值为最大相对幅度值。
S460、基于所述相对幅度值和所述最大相对幅度值计算所述人脸图像中人眼的闭眼权重。
可选地，可以先计算出人脸图像中人眼的闭眼幅度值，闭眼幅度值可以是人眼眨眼过程中，某一状态下人眼张开幅度值与人眼完全张开时的人眼张开幅度值的比值，在本申请实施例中，相对幅度值和最大相对幅度值均是相对于参考距离的比值，则闭眼幅度值可以是相对幅度值和最大相对幅度值的比值，闭眼幅度值表达了眨眼过程中某一状态下人眼的闭合程度，闭眼幅度值为小于或等于1的值。
在得到闭眼幅度值后，可以通过闭眼幅度值计算闭眼权重，在本申请实施例中，闭眼权重和闭眼幅度值负相关，闭眼权重表达了人脸图像中的人眼用于控制脸部模型眨眼时，脸部模型中眼睛在眨眼的某一时刻，眼睛的闭合程度。
S470、基于所述闭眼权重驱动所述脸部模型中的眼睛进行眨眼。
随着视频数据的播放，根据每帧视频数据中的人脸图像对已显示的脸部模型进行眨眼控制，例如，可以获得多帧在时间上连续的人脸图像，每帧人脸图像确定出一个闭眼权重，可以根据闭眼权重驱动显示在播放界面中的脸部模型进行眨眼。
如图4B所示为根据视频数据中的人脸图像控制脸部模型眨眼的过程和效果，在图4B中，当视频数据中的人脸图像眨眼时，覆盖在人脸图像上的脸部模型跟随着人脸图像模拟眨眼。
本申请实施例中，在播放视频数据时，显示脸部模型后，可以通过视频数据中的人脸图像计算人脸图像中的人眼的闭眼权重，从而根据闭眼权重驱动显示的脸部模型眨眼，解决了人眼检测无法度量人眼闭合程度的问题，能够通过人脸眨眼过程中眼睛的闭合程度驱动脸部模型模拟眨眼，使得脸部模型能够模 拟人脸的真实眨眼动作，进而使得脸部模型的表情更逼真，也无需大量人脸图像，实现过程也较为简单，计算速度快，能够获得较为流畅的眨眼效果。
实施例五
图5是本申请实施例五提供的一种人眼闭合程度的确定装置的结构框图，本申请实施例的人眼闭合程度的确定装置具体可以包括如下模块：人脸图像获取模块501，设置为获取人脸图像；人脸数据确定模块502，设置为在所述人脸图像中确定人眼张开幅度值和参考距离；相对幅度值计算模块503，设置为计算所述人眼张开幅度值相对于所述参考距离的相对幅度值；最大相对幅度值获取模块504，设置为获取最大相对幅度值；闭眼权重计算模块505，设置为基于所述相对幅度值和所述最大相对幅度值计算所述人脸图像中人眼的闭眼权重，所述闭眼权重用于度量人眼的闭合程度。
可选地，所述人脸数据确定模块502包括：人脸关键点检测子模块，设置为对所述人脸图像进行人脸关键点检测，得到人脸关键点；人眼关键点和参考关键点确定子模块，设置为从所述人脸关键点中确定出人眼关键点，以及从所述人脸关键点中确定出参考关键点；人眼张开幅度值计算子模块，设置为基于所述人眼关键点计算人眼张开幅度值；参考距离计算子模块，设置为基于所述参考关键点计算参考距离。
可选地，所述人眼关键点和参考关键点确定子模块包括：眼部关键点确定单元，设置为从所述人脸关键点中确定出眼部关键点；人眼关键点选择单元，设置为选取所述眼部关键点中的眼顶关键点和眼底关键点作为人眼关键点。
可选地，所述人眼关键点包括眼顶关键点和眼底关键点，所述人眼张开幅度值计算子模块包括：第一距离计算单元，设置为计算所述眼顶关键点到所述眼底关键点的距离，并将所述眼顶关键点到所述眼底关键点的距离作为人眼张开幅度值。
可选地，所述人眼关键点和参考关键点确定子模块包括：鼻部关键点确定单元，设置为从所述人脸关键点中确定出鼻部关键点；参考关键点选择单元，设置为选择所述鼻部关键点中的鼻顶关键点和鼻尖关键点作为参考关键点。
可选地，所述参考关键点包括鼻顶关键点和鼻尖关键点，所述参考距离计算子模块包括：第二距离计算单元，设置为计算所述鼻顶关键点到所述鼻尖关键点的距离，并将所述鼻顶关键点到所述鼻尖关键点的距离作为参考距离。
可选地，所述闭眼权重计算模块505包括：闭眼幅度值计算子模块，设置为计算所述相对幅度值与所述最大相对幅度值之间的比值，得到人眼的闭眼幅度值，所述闭眼幅度值与所述相对幅度值正相关，与所述最大相对幅度值负相 关；闭眼权重计算子模块，设置为采用所述闭眼幅度值和预设闭眼常量计算所述闭眼权重。
本申请实施例所提供的人眼闭合程度的确定装置可执行本申请任意实施例所提供的人眼闭合程度的确定方法，具备执行方法相应的功能模块。
实施例六
图6是本申请实施例六提供的一种眼睛控制装置的结构框图，本申请实施例的眼睛控制装置具体可以包括如下模块：人脸图像和脸部模型获取模块601，设置为人脸图像和脸部模型；闭眼权重获取模块602，设置为获取人脸图像中人眼的闭眼权重，所述闭眼权重用于度量人眼的闭合程度；眼睛控制模块603，设置为基于所述闭眼权重对所述脸部模型中的眼睛进行控制。所述闭眼权重通过本申请实施例五所述的人眼闭合程度的确定装置确定。
可选地，所述眼睛控制模块603包括：预设张开幅度值获取子模块，设置为获取所述初始脸部模型中眼睛的预设张开幅度值；目标张开幅度值计算子模块，设置为基于所述闭眼权重和所述预设张开幅度值计算眼睛的目标张开幅度值；调整子模块，设置为将所述初始脸部模型中的眼睛的张开幅度值调整为所述目标张开幅度值。
可选地，所述目标张开幅度值计算子模块包括：目标张开幅度值确定单元，设置为在所述闭眼权重大于预设值的情况下，确定所述目标张开幅度值为0。
本申请实施例所提供的眼睛控制装置可执行本申请实施例三所提供的眼睛控制方法，具备执行方法相应的功能模块。
实施例七
图7是本申请实施例七提供的一种眼睛控制装置的结构框图，本申请实施例的眼睛控制装置具体可以包括如下模块：播放模块701，设置为播放视频数据，所述视频数据中具有多帧图像数据，所述图像数据中具有人脸图像；脸部模型显示模块702，设置为显示脸部模型，以覆盖所述人脸图像；人脸数据确定模块703，设置为在每帧所述人脸图像中，确定人眼张开幅度值和参考距离；相对幅度值计算模块704，设置为计算所述人眼张开幅度值相对于所述参考距离的相对幅度值；最大相对幅度值获取模块705，设置为获取最大相对幅度值；闭眼权重计算模块706，设置为基于所述相对幅度值和所述最大相对幅度值计算所述人脸图像中人眼的闭眼权重；模型驱动模块707，设置为基于所述闭眼权重驱动所述脸部模型中的眼睛进行眨眼。
可选地，所述最大相对幅度值获取模块705包括：相对幅度值获取子模块，设置为获取多帧人脸图像中所述人眼张开幅度值相对于所述参考距离的相对幅 度值；最大相对幅度值确定子模块，设置为从多个相对幅度值中确定出最大值，并将所述最大值作为所述最大相对幅度值。
本申请实施例所提供的眼睛控制装置可执行本申请实施例四所提供的眼睛控制方法，具备执行方法相应的功能模块。
实施例八
参照图8，示出了本申请一个示例中的一种设备的结构示意图。如图8所示，该设备具体可以包括：处理器80、存储器81、具有触摸功能的显示屏82、输入装置83、输出装置84以及通信装置85。该设备中处理器80的数量可以是一个或者多个，图8中以一个处理器80为例。该设备中存储器81的数量可以是一个或者多个，图8中以一个存储器81为例。该设备的处理器80、存储器81、显示屏82、输入装置83、输出装置84以及通信装置85可以通过总线或者其他方式连接，图8中以通过总线连接为例。
存储器81作为一种计算机可读存储介质，可设置为存储软件程序、计算机可执行程序以及模块，如本申请实施例一或二所述的人眼闭合程度的确定方法对应的程序指令/模块(例如，上述人眼闭合程度的确定装置中的人脸图像获取模块501、人脸数据确定模块502、相对幅度值计算模块503、最大相对幅度值获取模块504、闭眼权重计算模块505)，或如本申请实施例三所述的眼睛控制方法对应的程序指令/模块(例如，上述眼睛控制装置中的人脸图像和脸部模型获取模块601、闭眼权重获取模块602和眼睛控制模块603)，或如本申请实施例四所述的眼睛控制方法对应的程序指令/模块(例如，上述眼睛控制装置中的播放模块701、脸部模型显示模块702、人脸数据确定模块703、相对幅度值计算模块704、最大相对幅度值获取模块705、闭眼权重计算模块706和模型驱动模块707)。存储器81可主要包括存储程序区和存储数据区，存储程序区可存储操作装置、至少一个功能所需的应用程序；存储数据区可存储根据设备的使用所创建的数据等。此外，存储器81可以包括高速随机存取存储器，还可以包括非易失性存储器，例如至少一个磁盘存储器件、闪存器件、或其他非易失性固态存储器件。在一些实例中，存储器81可进一步包括相对于处理器80远程设置的存储器，这些远程存储器可以通过网络连接至设备。上述网络的实例包括但不限于互联网、企业内部网、局域网、移动通信网及其组合。
显示屏82为具有触摸功能的显示屏82，其可以是电容屏、电磁屏或者红外屏。一般而言，显示屏82设置为根据处理器80的指示显示数据，还设置为接收作用于显示屏82的触摸操作，并将相应的信号发送至处理器80或其他装置。可选的，当显示屏82为红外屏时，其还包括红外触摸框，该红外触摸框设置在显示屏82的四周，显示屏82还可以设置为接收红外信号，并将该红外信号发 送至处理器80或者其他设备。
通信装置85，设置为与其他设备建立通信连接，其可以是有线通信装置和/或无线通信装置。
输入装置83可设置为接收输入的数字或者字符信息，以及产生与设备的用户设置以及功能控制有关的键信号输入，还可以是设置为获取图像的摄像头以及获取音频数据的拾音设备。输出装置84可以包括扬声器等音频设备。输入装置83和输出装置84的组成可以根据实际情况设定。
处理器80设置为通过运行存储在存储器81中的软件程序、指令以及模块，从而执行设备的多种功能应用以及数据处理，即实现上述人眼闭合程度的确定方法和/或眼睛控制方法。
本实施例中，处理器80设置为在执行存储器81中存储的一个或多个程序的情况下，实现本申请实施例提供的人眼闭合程度的确定方法和/或眼睛控制方法。
本申请实施例还提供一种计算机可读存储介质，所述存储介质中的指令由设备的处理器执行时，使得设备能够执行如上述方法实施例所述的人脸闭合程度的确定方法和/或眼睛控制方法。
对于装置、设备、存储介质实施例而言，由于其与方法实施例基本相似，所以描述的比较简单，相关之处参见方法实施例的部分说明即可。
通过以上关于实施方式的描述，所属领域的技术人员可以清楚地了解到，本申请可借助软件及必需的通用硬件来实现，也可以通过硬件实现，但很多情况下前者是更佳的实施方式。基于这样的理解，本申请的技术方案可以以软件产品的形式体现出来，该计算机软件产品可以存储在计算机可读存储介质中，如计算机的软盘、只读存储器(Read-Only Memory,ROM)、随机存取存储器(Random Access Memory,RAM)、闪存(FLASH)、硬盘或光盘等，包括若干指令用以使得一台计算机设备(可以是机器人，个人计算机，服务器，或者网络设备等)执行本申请任意实施例所述的人眼闭合程度的确定方法和/或眼睛控制方法。
值得注意的是，上述人眼闭合程度的确定装置和/或眼睛的控制装置中，所包括的多个单元和模块只是按照功能逻辑进行划分的，但并不局限于上述的划分，只要能够实现相应的功能即可；另外，多个功能单元的名称也只是为了便于相互区分，并不用于限制本申请的保护范围。
应当理解，本申请的每个部分可以用硬件、软件、固件或它们的组合来实现。在上述实施方式中，多个步骤或方法可以用存储在存储器中且由合适的指 令执行装置执行的软件或固件来实现。例如，如果用硬件来实现，和在另一实施方式中一样，可用本领域公知的下列技术中的任一项或他们的组合来实现：具有设置为对数据信号实现逻辑功能的逻辑门电路的离散逻辑电路，具有合适的组合逻辑门电路的专用集成电路，可编程门阵列(Programmable Gate Array，PGA)，现场可编程门阵列(Field Programmable Gate Array，FPGA)等。

Claims (17)

  1. 一种人眼闭合程度的确定方法，包括：
    获取人脸图像；
    在所述人脸图像中确定人眼张开幅度值和参考距离；
    计算所述人眼张开幅度值相对于所述参考距离的相对幅度值；
    获取最大相对幅度值；
    基于所述相对幅度值和所述最大相对幅度值计算所述人脸图像中人眼的闭眼权重，所述闭眼权重用于度量人眼的闭合程度。
  2. 如权利要求1所述的方法，其中，所述在所述人脸图像中确定人眼张开幅度值和参考距离，包括：
    对所述人脸图像进行人脸关键点检测，得到人脸关键点；
    从所述人脸关键点中确定出人眼关键点，以及从所述人脸关键点中确定出参考关键点；
    基于所述人眼关键点计算人眼张开幅度值；
    基于所述参考关键点计算参考距离。
  3. 如权利要求2所述的方法，其中，所述从所述人脸关键点中确定出人眼关键点，包括：
    从所述人脸关键点中确定出眼部关键点；
    选取所述眼部关键点中的眼顶关键点和眼底关键点作为人眼关键点。
  4. 如权利要求2所述的方法，其中，所述人眼关键点包括眼顶关键点和眼底关键点，所述基于所述人眼关键点计算人眼张开幅度值，包括：
    计算所述眼顶关键点到所述眼底关键点的距离，并将所述眼顶关键点到所述眼底关键点的距离作为人眼张开幅度值。
  5. 如权利要求2所述的方法，其中，所述从所述人脸关键点中确定出参考关键点，包括：
    从所述人脸关键点中确定出鼻部关键点；
    选择所述鼻部关键点中的鼻顶关键点和鼻尖关键点作为参考关键点。
  6. 如权利要求2所述的方法，其中，所述参考关键点包括鼻顶关键点和鼻尖关键点，所述基于所述参考关键点计算参考距离，包括：
    计算所述鼻顶关键点到所述鼻尖关键点的距离，并将所述鼻顶关键点到所述鼻尖关键点的距离作为参考距离。
  7. 如权利要求1-6任一项所述的方法，其中，所述基于所述相对幅度值和所述最大相对幅度值计算所述人脸图像中人眼的闭眼权重，包括：
    计算所述相对幅度值与所述最大相对幅度值之间的比值，得到人眼的闭眼幅度值，所述闭眼幅度值与所述相对幅度值正相关，与所述最大相对幅度值负相关；
    采用所述闭眼幅度值和预设闭眼常量计算所述闭眼权重。
  8. 一种眼睛控制方法，包括：
    获取人脸图像和脸部模型；
    获取所述人脸图像中人眼的闭眼权重，所述闭眼权重用于度量人眼的闭合程度；
    基于所述闭眼权重对所述脸部模型中的眼睛进行控制；
    其中，所述闭眼权重通过权利要求1-7任一项所述的人眼闭合程度的确定方法确定。
  9. 如权利要求8所述的方法，其中，所述基于所述闭眼权重对所述脸部模型中的眼睛进行控制，包括：
    获取所述脸部模型中眼睛的预设张开幅度值；
    基于所述闭眼权重和所述预设张开幅度值计算眼睛的目标张开幅度值；
    将所述脸部模型中的眼睛的张开幅度值调整为所述目标张开幅度值。
  10. 如权利要求9所述的方法，其中，所述基于所述闭眼权重和所述预设张开幅度值计算眼睛的目标张开幅度值，包括：
    在所述闭眼权重大于预设值的情况下，确定所述目标张开幅度值为0。
  11. 一种眼睛控制方法，其中，包括：
    播放视频数据，所述视频数据中具有多帧图像数据，所述图像数据中具有人脸图像；
    显示脸部模型，以覆盖所述人脸图像；
    在每帧所述人脸图像中，确定人眼张开幅度值和参考距离；
    计算所述人眼张开幅度值相对于所述参考距离的相对幅度值；
    获取最大相对幅度值；
    基于所述相对幅度值和所述最大相对幅度值计算所述人脸图像中人眼的闭眼权重；
    基于所述闭眼权重驱动所述脸部模型中的眼睛进行眨眼。
  12. 如权利要求11所述的方法，其中，所述获取最大相对幅度值，包括：
    获取多帧所述人脸图像中所述人眼张开幅度值相对于所述参考距离的相对幅度值；
    从多个相对幅度值中确定出最大值，并将所述最大值作为最大相对幅度值。
  13. 一种人眼闭合程度的确定装置，其中，包括：
    人脸图像获取模块，设置为获取人脸图像；
    人脸数据确定模块，设置为在所述人脸图像中确定人眼张开幅度值和参考距离；
    相对幅度值计算模块，设置为计算所述人眼张开幅度值相对于所述参考距离的相对幅度值；
    最大相对幅度值获取模块，设置为获取最大相对幅度值；
    闭眼权重计算模块，设置为基于所述相对幅度值和所述最大相对幅度值计算所述人脸图像中人眼的闭眼权重，所述闭眼权重用于度量人眼的闭合程度。
  14. 一种眼睛控制装置，包括：
    人脸图像和脸部模型获取模块，设置为人脸图像和脸部模型；
    闭眼权重获取模块，设置为获取所述人脸图像中人眼的闭眼权重，所述闭眼权重用于度量所述人眼的闭合程度；
    眼睛控制模块，设置为基于所述闭眼权重对所述脸部模型中的眼睛进行控制；
    其中，所述闭眼权重通过权利要求13所述的人眼闭合程度的确定装置确定。
  15. 一种眼睛控制装置，包括：
    播放模块，设置为播放视频数据，所述视频数据中具有多帧图像数据，所述图像数据中具有人脸图像；
    脸部模型显示模块，设置为显示脸部模型，以覆盖所述人脸图像；
    人脸数据确定模块，设置为在每帧所述人脸图像中，确定人眼张开幅度值和参考距离；
    相对幅度值计算模块，设置为计算所述人眼张开幅度值相对于所述参考距离的相对幅度值；
    最大相对幅度值获取模块，设置为获取最大相对幅度值；
    闭眼权重计算模块，设置为基于所述相对幅度值和所述最大相对幅度值计算所述人脸图像中人眼的闭眼权重；
    模型驱动模块，设置为基于所述闭眼权重驱动所述脸部模型中的眼睛进行眨眼。
  16. 一种设备，包括：
    至少一个处理器；
    存储装置，设置为存储至少一个程序，
    当所述至少一个程序被所述至少一个处理器执行，使得所述至少一个处理器实现以下至少之一：权利要求1-7任一所述的人眼闭合程度的确定方法；权利要求8-10任一所述的眼睛控制方法；权利要求11-12任一所述的眼睛控制方法。
  17. 一种计算机可读存储介质，存储有计算机程序，所述计算机程序被处理器执行时实现以下至少之一：权利要求1-7任一所述的人眼闭合程度的确定方法；权利要求8-10任一所述的眼睛控制方法；权利要求11-12任一所述的眼睛控制方法。
PCT/CN2020/098066 2019-07-10 2020-06-24 人眼闭合程度的确定方法、眼睛控制方法、装置、设备和存储介质 WO2021004285A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US17/625,947 US11775059B2 (en) 2019-07-10 2020-06-24 Method for determining human eye close degrees, method for controlling eyes, device, and storage medium

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201910621637.5 2019-07-10
CN201910621637.5A CN110363135B (zh) 2019-07-10 2019-07-10 人眼闭合程度的确定方法、眼睛控制方法和装置

Publications (1)

Publication Number Publication Date
WO2021004285A1 true WO2021004285A1 (zh) 2021-01-14

Family

ID=68218637

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/098066 WO2021004285A1 (zh) 2019-07-10 2020-06-24 人眼闭合程度的确定方法、眼睛控制方法、装置、设备和存储介质

Country Status (3)

Country Link
US (1) US11775059B2 (zh)
CN (1) CN110363135B (zh)
WO (1) WO2021004285A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113487711A (zh) * 2021-07-23 2021-10-08 北京达佳互联信息技术有限公司 虚拟形象的眨眼控制方法、装置、电子设备及存储介质

Families Citing this family (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110363135B (zh) * 2019-07-10 2022-09-27 广州市百果园信息技术有限公司 人眼闭合程度的确定方法、眼睛控制方法和装置
CN113138659A (zh) * 2020-01-16 2021-07-20 七鑫易维(深圳)科技有限公司 一种工作模式的控制方法、装置、设备及存储介质
CN111583372B (zh) * 2020-05-09 2021-06-25 腾讯科技（深圳）有限公司 虚拟角色面部表情生成方法和装置、存储介质及电子设备
CN116030509A (zh) * 2021-10-25 2023-04-28 脸萌有限公司 图像处理方法、装置、电子设备和计算机可读存储介质
CN116029948A (zh) * 2021-10-25 2023-04-28 脸萌有限公司 图像处理方法、装置、电子设备和计算机可读存储介质
CN113946221A (zh) * 2021-11-03 2022-01-18 广州繁星互娱信息科技有限公司 眼部驱动控制方法和装置、存储介质及电子设备
CN116704080B (zh) * 2023-08-04 2024-01-30 腾讯科技（深圳）有限公司 眨眼动画生成方法、装置、设备及存储介质

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20060082579A1 (en) * 2004-10-18 2006-04-20 Reallusion Inc. Caricature generating system and method
CN103279969A (zh) * 2013-05-08 2013-09-04 中国科学技术大学 一种建立富有情感表达能力的三维虚拟眼睛运动模型的方法
CN105139438A (zh) * 2014-09-19 2015-12-09 电子科技大学 视频人脸卡通动画生成方法
CN110363135A (zh) * 2019-07-10 2019-10-22 广州市百果园信息技术有限公司 人眼闭合程度的确定方法、眼睛控制方法和装置

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7416302B2 (en) * 2001-09-14 2008-08-26 Stmicroelectronics S.A. Ring location
JP5055166B2 (ja) * 2008-02-29 2012-10-24 キヤノン株式会社 眼の開閉度判定装置、方法及びプログラム、撮像装置
CN106446822B (zh) 2016-09-20 2018-07-10 西安科技大学 基于圆拟合的眨眼检测方法
CN109815937A (zh) * 2019-02-25 2019-05-28 湖北亿咖通科技有限公司 疲劳状态智能识别方法、装置及电子设备

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20060082579A1 (en) * 2004-10-18 2006-04-20 Reallusion Inc. Caricature generating system and method
CN103279969A (zh) * 2013-05-08 2013-09-04 中国科学技术大学 一种建立富有情感表达能力的三维虚拟眼睛运动模型的方法
CN105139438A (zh) * 2014-09-19 2015-12-09 电子科技大学 视频人脸卡通动画生成方法
CN110363135A (zh) * 2019-07-10 2019-10-22 广州市百果园信息技术有限公司 人眼闭合程度的确定方法、眼睛控制方法和装置

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113487711A (zh) * 2021-07-23 2021-10-08 北京达佳互联信息技术有限公司 虚拟形象的眨眼控制方法、装置、电子设备及存储介质
CN113487711B (zh) * 2021-07-23 2024-02-09 北京达佳互联信息技术有限公司 虚拟形象的眨眼控制方法、装置、电子设备及存储介质

Also Published As

Publication number Publication date
CN110363135B (zh) 2022-09-27
US20220261070A1 (en) 2022-08-18
US11775059B2 (en) 2023-10-03
CN110363135A (zh) 2019-10-22

Similar Documents

Publication Publication Date Title
WO2021004285A1 (zh) 人眼闭合程度的确定方法、眼睛控制方法、装置、设备和存储介质
US11461876B2 (en) Video data processing method and apparatus, and storage medium
JP7095722B2 (ja) 情報処理装置およびプログラム
US9973711B2 (en) Content-based zooming and panning for video curation
US10070046B2 (en) Information processing device, recording medium, and information processing method
WO2019137131A1 (zh) 图像处理方法、装置、存储介质及电子设备
CN104519263B (zh) 一种图像获取的方法及电子设备
CN105450911B (zh) 图像处理装置、图像处理方法
US10382704B2 (en) Image processing device that generates a composite image
CN103428461A (zh) 一种授课影像录制的系统和方法
CN103916586B (zh) 图像解析装置以及图像解析方法
WO2020108656A1 (zh) 一种图片选择方法及相关设备
JP7424375B2 (ja) 画像処理装置、および画像処理方法、並びにプログラム
US9807301B1 (en) Variable pre- and post-shot continuous frame buffering with automated image selection and enhancement
CN103327235A (zh) 图像处理装置以及图像处理方法
WO2022121963A1 (zh) 一种图像遮挡检测方法、装置、拍摄设备及介质
TWI525454B (zh) 視訊播放系統與方法
CN105988580A (zh) 移动终端的屏幕控制方法和装置
CN109859115A (zh) 一种图像处理方法、终端及计算机可读存储介质
JP2014225772A (ja) 動画再生装置、動画再生方法及びプログラム
CN105608469B (zh) 图像分辨率的确定方法及装置
RU2814199C2 (ru) Способ определения степени закрытия глаза человека, способ управления глазом, аппарат, устройство и носитель данных
CN110766631A (zh) 人脸图像的修饰方法、装置、电子设备和计算机可读介质
WO2024027063A1 (zh) 直播方法、装置、存储介质、电子设备及产品
US20220321772A1 (en) Camera Control Method and Apparatus, and Terminal Device

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20837029

Country of ref document: EP

Kind code of ref document: A1

122 Ep: pct application non-entry in european phase

Ref document number: 20837029

Country of ref document: EP

Kind code of ref document: A1