WO2020252969A1

WO2020252969A1 - Eye key point labeling method and apparatus, and training method and apparatus for eye key point detection model

Info

Publication number: WO2020252969A1
Application number: PCT/CN2019/108077
Authority: WO
Inventors: 李源; 杨燕丹; 王晋玮
Original assignee: 初速度（苏州）科技有限公司
Priority date: 2019-06-21
Filing date: 2019-09-26
Publication date: 2020-12-24
Also published as: CN110956071A; CN110956071B

Abstract

Disclosed in the embodiments of the present invention are an eye key point labeling method and apparatus, and a training method and apparatus for an eye key point detection model. The method comprises: obtaining a face image and eyelid labeling curves corresponding to the face image; for each eyelid labeling curve, integrating the eyelid labeling curve on the basis of a mathematical integral principle to determine a first curve length of an upper eyelid labeling curve and a second curve length of a lower eyelid labeling curve in the eyelid labeling curve; determining a plurality of eyelid points to be used from the upper eyelid labeling curve and the lower eyelid labeling curve; determining, on the basis of the first curve length of the upper eyelid labeling curve, the second curve length of the lower eyelid labeling curve, said eyelid points and a preset number of equally-divided points, equally-divided upper eyelid points and equally-divided lower eyelid points from the upper eyelid labeling curve and the lower eyelid labeling curve respectively, and then determining the eye key points corresponding to each face image on the basis of said upper and lower eyelid points in combination with labeled eye corner points. Therefore, eye key points having obvious semantic features are labeled, and the labeling efficiency is improved.

Description

Method and device for labeling key points of eyes and its detection model training

Technical field

The present invention relates to the technical field of intelligent detection, in particular to a method and device for labeling key points of the eye and its detection model.

Background technique

The detection of the key points of the human eyes in the image plays an important role in many aspects. For example, in the fatigue detection process, the fatigue detection system can detect the key points contained in the image based on the pre-trained eye key point detection model. The key points of the upper and lower eyelids of the human eye, and then, based on the key points of the upper and lower eyelids, the distance between the upper and lower eyelids of the human eye is calculated as the opening and closing distance of the human eye, and then based on the opening and closing distance of the human eye in the prediction Set the opening and closing distance within the time length to determine whether the person corresponding to the image is in a fatigue state, and realize the detection of the fatigue state of the person. For another example: in the process of beautifying, the beautifying camera can detect the eye key points of the upper and lower eyelids contained in the image based on the pre-trained eye key point detection model. The positions of the key points of the upper and lower eyelids are used to scale and shrink the human eyes to beautify the human eyes. Wherein, the above-mentioned pre-trained eye key point detection model is obtained by training based on sample images marked with human eye key points.

It can be seen that in order to obtain an eye key point detection model, it is very important to label the eye key points of the human eye. In related technologies, the above-mentioned key eye points are generally manually marked by the annotator. For the key eye points manually marked by the annotator, the marking standards for the key points of the eye by different annotators are not uniform. The semantic features of the key points of the eye are not obvious, and the efficiency of manually marking the key points of the eye is low.

Summary of the invention

The present invention provides a method and device for marking eye key points and a detection model training method and device, so as to realize marking eye key points with obvious semantic features and improve marking efficiency to a certain extent. The specific technical solution is as follows.

In the first aspect, an embodiment of the present invention provides a method for labeling key points of the eye. The method includes: obtaining a face image and an eyelid curve corresponding to each face image, wherein the face image is marked with Contains the marked corner points of the eyes and the marked eyelid points of the upper and lower eyelids. Each marked eyelid curve includes: based on the corresponding marked eyelid points and marked eye corner points, the marked upper eyelid curve representing the upper eyelid and the marked lower eyelid representing the lower eyelid are generated curve;

For each marked eyelid curve, based on the principle of mathematical integration, the marked eyelid curve is integrated to determine the first curve length marked with the upper eyelid curve and the second curve length marked with the lower eyelid curve in the marked eyelid curve; In the marked upper eyelid curve and the marked lower eyelid curve, multiple eyelid points to be used are determined;

For each marked eyelid curve, based on the first curve length of the upper eyelid curve marked on the marked eyelid curve, the second curve length marked on the lower eyelid curve, the multiple eyelid points to be used and the preset number of equal division points, In marking the eyelid curve, marking the upper eyelid curve and marking the lower eyelid curve, respectively determining the preset number of equal division points minus 1 equal division upper eyelid point and the preset number of equal division points minus 1 equal division lower eyelid point;

The marked eye corner points, equal division of upper eyelid points, and equal division of lower eyelid points in each face image are determined as key eye points corresponding to each face image.

Optionally, the step of obtaining the eyelid curve corresponding to each face image includes:

For each face image, the upper eyelid curve that characterizes the upper eyelid is fitted based on the marked eye corner points in the face image, the marked eyelid points of the upper eyelid, and a preset curve fitting algorithm; and based on the face The marked eye corner points, the marked eyelid points of the lower eyelid and the preset curve fitting algorithm in the image fit the lower eyelid curve that characterizes the lower eyelid to obtain the eyelid curve corresponding to the face image.

Optionally, after the step of determining the marked eye corner points, equally divided upper eyelid points, and equally divided lower eyelid points in each face image as the key eye points corresponding to each face image, the method further Including: for each face image, based on the key points of the eyes in the face image, cut out the image of the area where the eyes are located from the face image to obtain an eye image marked with the key points of the eyes;

The eye image and its corresponding calibration information are determined as the training data of the eye key point detection model for detecting the eye key points of the eyes in the image, wherein the calibration information includes the eye in the corresponding eye image. The location information of key points.

Optionally, the step of determining the eye image and its corresponding calibration information as the training data of the eye key point detection model for detecting the eye key points of the eyes in the image includes:

Obtain the real eye opening and closing length corresponding to each face image;

Obtain the measured eye opening and closing length corresponding to each face image, where the measured eye opening and closing length is: based on the length determined by the upper and lower eyelids of the eyes in the target 3D face model corresponding to the face image, the target 3D person The face model is: a face model determined based on the face feature points in the corresponding face image and a preset three-dimensional face model, and the target three-dimensional face model includes: based on the marked eye corner points in the face image, Equally divide the upper eyelid point and the lower eyelid point to construct the upper and lower eyelids of the eye;

For each face image, calculate the ratio of the actual eye opening and closing length corresponding to the face image to the measured eye opening and closing length as the measured deviation corresponding to the face image;

The eye image and its corresponding calibration information are determined as the training data of the eye key point detection model used to detect the upper and lower eyelids of the eye in the image, where the calibration information includes: the corresponding eye image The position information of the marked eye key points, and the actual measured deviation corresponding to the face image corresponding to the eye image.

Optionally, before the step of obtaining a face image and an eyelid curve corresponding to each face image, the method further includes: a process of marking the upper and lower eyelids of the eyes of the person’s face in each face image, wherein , For each face image, perform the following steps to mark the upper and lower eyelids of the person’s face in each face image:

Obtain and display a first face image, wherein the first face image includes eyes of a person's face, and the first face image is one of the face images;

Receiving an annotation instruction triggered by an annotator for the upper and lower eyelids of the eyes in the first face image, wherein the annotation instruction carries position information of the marked annotation point;

If it is detected that the tagger has triggered the tagging instruction for the specified eyelid in the first face image at least twice, based on the location information of the tag point carried by the tagging instruction triggered by the tagger at least twice, and the preset A curve fitting algorithm to determine a specified eyelid curve that characterizes the specified eyelid, wherein the specified eyelid is: the upper eyelid and the lower eyelid of the eye in the first face image;

The specified eyelid curve is displayed in the first face image, so that the annotator detects whether the annotated point is an eyelid point or an eye corner point on the specified eyelid.

In the second aspect, an embodiment of the present invention provides a method for training an eye key point detection model, the method including:

Obtain training data, where the training data includes the upper and lower eyelid equidistant eyelid points marked with the eyes and the eye images marked with the corner points of the eyes, and calibration information corresponding to each eye image, and the calibration information includes: the marking in the corresponding eye image The position information of the divided eyelid points and the marked eye corner points, the divided eyelid points include: based on the marked eyelid curve corresponding to the face image and the principle of mathematical integration, the divided upper eyelid of the eye in the face image is determined The eyelid point and the lower eyelid are equally divided into the lower eyelid points; the marking eyelid curve includes: marking the upper eyelid, which is generated based on the marked corner points of the eyes marked in the corresponding face image and the marked eyelid points of the upper and lower eyelids, and represents the upper eyelid The curve and the marked lower eyelid curve that characterizes the lower eyelid, and each eye image is an image of the area where the eye is located from the corresponding face image;

Input the eye image and the position information of the divided eyelid points and the marked eye corner points included in the calibration information corresponding to each eye image into the initial eye key point detection model to be trained to detect the up and down of the eyes in the image An eye key point detection model that divides the eyelid point and the corner point of the eyelid.

Optionally, in the said eye image and the position information of the divided eyelid points and the marked eye corner points included in the calibration information corresponding to each eye image, input the initial eye key point detection model to obtain training Before the step of detecting the key point detection model of the eyelid points and the corner points in the upper and lower eyelids of the eyes in the image, the method further includes:

For each eye image, performing the correction processing on the eye image to obtain the correction image, wherein the correction processing is: processing that the ordinates in the position information of the marked corner points in the eye image are all the same;

Based on the position information of the equal-divided eyelid points and the marked eye corner points of each converted image, update the position information of the divided eyelid points and marked eye corner points included in the calibration information corresponding to each converted image;

The eye image and the position information of the divided eyelid points and the marked eye corner points included in the calibration information corresponding to each eye image are input into an initial eye key point detection model to be trained to detect the eyes in the image The steps of detecting the key points of the eyelid points and the corner points of the upper and lower eyelids include:

Input the position information of the divided eyelid points and the marked corner points of each converted image and the calibration information corresponding to each converted image into the initial eye key point detection model to be trained to detect the up and down of the eyes in the image An eye key point detection model that divides the eyelid point and the corner point of the eyelid.

Optionally, the eye image includes a left eye image and a right eye image corresponding to the left eye image;

In the above-mentioned each converted image, and the calibration information corresponding to each converted image, the position information of the divided eyelid points and the marked eye canthus points are input into the initial eye key point detection model to be trained to be used in the detection image Before the step of detecting the key points of the eyelid point and the corner point of the eye in the upper and lower eyelids of the eye, the method further includes: mirroring the left-eye image or the right-eye image corresponding to the left-eye image , Get the mirror image;

The mirror image and the unmirrored image are spliced to obtain a spliced image, wherein if the left-eye image is mirrored, the unmirrored image is the right-eye image corresponding to the left-eye image; The right-eye image corresponding to the left-eye image is mirrored, and the image that is not mirrored is the left-eye image;

Said each converted image and the calibration information corresponding to each converted image includes the position information of the divided eyelid points and the marked eye corner points, and input the initial eye key point detection model to be trained to detect the eyes in the image The steps of detecting the key points of the eyelid point and the corner point of the eyelid in the upper and lower eyelids include:

Input the position information of the divided eyelid points and the marked eye corner points included in each stitched image and the calibration information corresponding to each stitched image into the initial eye key point detection model to train the upper and lower eyelids for detecting the eyes in the image The key point detection model of the eyelid points and the corner points of the eye in, where the calibration information includes: the equally divided eyelid points in the mirror image and the position information of the marked eye corner points after mirroring and the position information in the image without mirroring Equally divide the eyelid points and mark the position information of the eye corner points.

Optionally, the calibration information corresponding to each eye image further includes: a measured deviation corresponding to the eye image, where the measured deviation is: the ratio of the actual eye opening and closing length corresponding to the eye image to the measured eye opening and closing length; The measured eye opening and closing length is the length determined based on the upper and lower eyelids of the eyes in the target three-dimensional face model corresponding to the face image. The target three-dimensional face model is based on the face feature points and predictions in the corresponding face image. A face model determined by a three-dimensional face model, where the target three-dimensional face model includes: the upper and lower eyelids of the eye constructed based on the marked eye corner points, equal division of upper eyelid points, and equal division of lower eyelid points in the face image ；

The eye image and the position information of the divided eyelid points and the marked eye corner points included in the calibration information corresponding to each eye image are input into an initial eye key point detection model to be trained to detect the eyes in the image The steps of detecting the key points of the eyelid point and the corner point of the eyelid in the upper and lower eyelids include:

The eye image and the calibration information corresponding to each eye image include the position information of the divided eyelid points and the marked eye corner points and the measured deviation corresponding to the eye image, and input the initial eye key point detection model to train the eye The key point detection model of the eye, wherein the key point detection model of the eye is used to detect the equally divided eyelid points and the corner points of the upper and lower eyelids of the eyes in the image, and detect the actual measured deviation corresponding to the image.

Optionally, in the said eye image and the position information of the divided eyelid points and the marked eye corner points included in the calibration information corresponding to each eye image, input the initial eye key point detection model to obtain training After the step of detecting the key point detection model of the eyelid points and the corner points in the upper and lower eyelids of the eyes in the image, the method further includes:

Obtaining an image to be inspected, where the image to be inspected includes the eyes of the person to be inspected;

The to-be-detected image is input into the eye key point detection model, and the equally divided eyelid points and eye corner points of the upper and lower eyelids of the to-be-detected person’s eyes in the to-be-detected image are determined.

Optionally, after the step of inputting the image to be detected into the key ocular point detection model, and determining the equally divided eyelid points and corner points of the upper and lower eyelids of the person to be detected in the image to be detected, The method further includes: using the Sobel algorithm to perform edge extraction on the image to be detected to obtain a grayscale edge map corresponding to the image to be detected;

Based on the equally divided eyelid points and corner points of the upper and lower eyelids of the eyes in the image to be detected, and a preset curve fitting algorithm, the eyelid curve of the upper and lower eyelids of the person to be detected is determined as the eyelid curve to be detected, and In the grayscale edge map, the eyelid curve to be detected is drawn, wherein the eyelid curve to be detected includes: an upper eyelid curve to be detected representing the upper eyelid of the person to be detected and a lower eyelid curve representing the person to be detected The lower eyelid curve of the eyelid to be tested;

Based on the equally divided eyelid points in the eyelid curve to be detected, multiple sets of reference points are determined in the grayscale edge map, wherein each group of reference points includes points corresponding to the equally divided eyelid points in the eyelid curve to be detected ；

For each set of reference points, based on the set of reference points, the corner points of the eyelid curve to be detected, and the preset curve fitting algorithm, the reference curve corresponding to the set of reference points is determined, and the reference curve is determined in the gray scale. A reference curve corresponding to each set of reference points is drawn in the edge map, where the reference curve corresponding to each set of reference points includes: a reference upper eyelid curve that characterizes the upper eyelid of the person to be inspected and a reference curve that characterizes the lower eyelid of the person to be inspected Refer to the lower eyelid curve;

In the grayscale edge map, for each first upper eyelid curve, the sum of the pixel values of the pixel points corresponding to the first upper eyelid curve is determined, wherein the first upper eyelid curve includes: each set of reference points The corresponding reference upper eyelid curve and the upper eyelid curve to be detected;

From the sum of the pixel values of the pixel points corresponding to each first upper eyelid curve, the sum with the largest value is determined, and the first upper eyelid curve corresponding to the largest sum is determined as a characterizing the person to be tested The target upper eyelid curve of the upper eyelid;

In the grayscale edge map, for each first lower eyelid curve, determine the sum of the pixel values of the pixel points corresponding to the first lower eyelid curve, where the first lower eyelid curve includes: each set of reference points The corresponding reference lower eyelid curve and the lower eyelid curve to be detected;

From the sum of the pixel values of the pixel points corresponding to each first lower eyelid curve, the sum with the largest value is determined, and the first lower eyelid curve corresponding to the largest sum is determined as a characterizing the person to be tested The target lower eyelid curve of the lower eyelid;

Based on the principle of mathematical integration, the target eyelid curve is integrated to determine the third curve length of the target upper eyelid curve and the fourth curve length of the target lower eyelid curve in the target eyelid curve; In the lower eyelid curve of the target, multiple reference eyelid points are determined;

Based on the third curve length of the target upper eyelid curve in the target eyelid curve, the fourth curve length of the target lower eyelid curve, the multiple reference eyelid points and the preset number of equal points, the target upper eyelid curve is selected from the target eyelid curve. In the eyelid curve and the target lower eyelid curve, the preset number of equal division points minus one equal upper eyelid point and the preset number of equal division points minus one equal lower eyelid point are respectively determined.

In a third aspect, an embodiment of the present invention provides a device for marking eye key points. The device includes: a first obtaining module configured to obtain a face image and an eyelid curve corresponding to each face image. The face image is marked with the marked corner points of the eyes and the marked eyelid points of the upper and lower eyelids, and each marked eyelid curve includes: based on the corresponding marked eyelid point and marked eye corner points, the marked upper eyelid curve representing the upper eyelid is generated And the marked lower eyelid curve representing the lower eyelid;

The first determining module is configured to, for each marked eyelid curve, integrate the marked eyelid curve based on the principle of mathematical integration, and determine the first curve length of the marked upper eyelid curve and the marked lower eyelid curve in the marked eyelid curve The second curve length; respectively determine a plurality of eyelid points to be used from the marked upper eyelid curve and the marked lower eyelid curve;

The second determining module is configured to, for each marked eyelid curve, based on the first curve length marking the upper eyelid curve in the marked eyelid curve, the second curve length marking the lower eyelid curve, the multiple eyelid points to be used, and Preset the number of equal points, from the marked upper eyelid curve and marked lower eyelid curve from the marked eyelid curve, respectively determine the preset number of equal points minus 1 equal upper eyelid point and the preset number of equal points minus 1 Divide the lower eyelid points equally;

The third determining module is configured to determine the marked eye corner point, the equally divided upper eyelid point, and the equally divided lower eyelid point in each face image as the key eye points corresponding to each face image.

In a fourth aspect, an embodiment of the present invention provides a training device for an eye key point detection model, the device including:

The second obtaining module is configured to obtain training data, where the training data includes the upper and lower eyelid equal eyelid points marked with the eyes and the eye images marked with the corner points of the eyes, and the calibration information corresponding to each eye image, the calibration information Including: the position information of the divided eyelid points and the marked eye corner points marked in the corresponding eye image. The divided eyelid points include: based on the marked eyelid curve corresponding to the face image and the principle of mathematical integration, the determined face image The equal division of the upper eyelid of the eye and the equal division of the lower eyelid point of the lower eyelid; the marked eyelid curve includes: generated based on the marked corner points of the eye marked in the corresponding face image and the marked eyelid points of the upper and lower eyelids , The marked upper eyelid curve representing the upper eyelid and the marked lower eyelid curve representing the lower eyelid, each eye image is an image of the area where the eye is cut out from the corresponding face image;

The input module is configured to input the eye image and the position information of the equally divided eyelid points and the marked eye corner points included in the calibration information corresponding to each eye image into the initial eye key point detection model to be trained for An eye key point detection model that detects the equally divided eyelid points and the corner points of the upper and lower eyelids of the eyes in the image.

It can be seen from the above content that the method and device for annotating key points of the eye and its detection model training provided by the embodiments of the present invention can obtain a face image and an eyelid curve corresponding to each face image, where the face image There are the marked corner points of the eyes and the marked eyelid points of the upper and lower eyelids. Each marked eyelid curve includes: based on the corresponding marked eyelid points and marked eye corner points, the marked upper eyelid curve and the lower eyelid representing the upper eyelid are generated The marked lower eyelid curve; for each marked eyelid curve, based on the principle of mathematical integration, the marked eyelid curve is integrated to determine the first curve length of the marked upper eyelid curve and the second marked lower eyelid curve in the marked eyelid curve Curve length; respectively, from the marked upper eyelid curve and marked lower eyelid curve, determine multiple eyelid points to be used; for each marked eyelid curve, based on the marked eyelid curve, mark the first curve length of the upper eyelid curve, mark the lower The second curve length of the eyelid curve, multiple eyelid points to be used, and the preset number of equal points. From the marked upper eyelid curve and the marked lower eyelid curve, the preset number of equal points minus one is determined respectively. The upper eyelid point and the preset number of equal points minus 1 equal lower eyelid point; the marked eye corner point, equal upper eyelid point and equal lower eyelid point in each face image are determined as corresponding to each face image Key points of the eye.

By applying the embodiment of the present invention, the eyelid marking curve corresponding to the face image can be integrated, and the first curve length marking the upper eyelid curve and the second curve length marking the lower eyelid curve in the marking eyelid curve can be determined, and the markings are respectively The eyelid curve and the marked lower eyelid curve are densely selected to determine multiple eyelid points to be used from the marked upper eyelid curve and the marked lower eyelid curve, and then mark the upper eyelid curve based on the first curve length of the marked upper eyelid curve The number of eyelid points to be used and the preset number of equal division points are determined from the marked upper eyelid curve, and the preset number of equal division points minus one equal upper eyelid point is determined; and the second curve length and label are based on the marked lower eyelid curve. The multiple eyelid points to be used on the lower eyelid curve and the preset number of equal division points are determined from the marked lower eyelid curve to determine the preset number of equal division points minus 1 equal lower eyelid point to achieve semi-automatic inclusion from the face image The upper and lower eyelids of the eyes are marked with equally divided eyelid points with obvious semantic characteristics, which realizes marking the key points of the eye with obvious semantic characteristics and improves the marking efficiency to a certain extent. Of course, implementing any product or method of the present invention does not necessarily need to achieve all the advantages described above at the same time.

The innovative points of the embodiments of the present invention include:

1. It can integrate the marked eyelid curve corresponding to the face image, determine the first curve length of the upper eyelid curve and the second curve length of the lower eyelid curve in the marked eyelid curve, and mark the upper eyelid curve and the annotation respectively The lower eyelid curve is intensively selected to determine multiple eyelid points to be used from the upper eyelid curve and the lower eyelid curve. Then, based on the first curve length of the upper eyelid curve, the multiple eyelid points on the upper eyelid curve are marked. Using the eyelid point and the preset number of equal points, from the marked upper eyelid curve, determine the preset number of equal points minus 1 equal upper eyelid point; and mark the lower eyelid curve based on the second curve length of the marked lower eyelid curve The multiple eyelid points to be used and the preset number of equal division points, from the marked lower eyelid curve, determine the preset number of equal division points minus 1 equal division lower eyelid point, in order to achieve semi-automatically from the upper and lower sides of the eyes contained in the face image The eyelids are marked with the equally divided eyelid points with obvious semantic characteristics, which realizes marking the key points of the eye with obvious semantic characteristics and improves the marking efficiency to a certain extent.

2. Using the marked eye corner points, equal division of the upper eyelid point and equal division of the lower eyelid point in each face image, the image of the area where the eye is located is intercepted from the face image to obtain the eye image marked with the key points of the eye, and The eye image and its corresponding calibration information including the position information of the eye key points in the corresponding eye image are determined as the training data of the eye key point detection model for detecting the eye key points of the eyes in the image, and Eye key points with obvious semantic features are used as the training data of the eye key point detection model. To a certain extent, the eye key point detection model with high stability and detection accuracy can be trained.

3. Using the actual eye opening and closing length corresponding to each face image and the measured eye opening and closing length, the measured deviation corresponding to each face image is determined. The measured deviation can characterize the measured distance between the upper and lower eyelids of the eye and the real The difference between the distance between the upper and lower eyelids of the eyes, the measured deviation is used as part of the training data of the eye key point detection model used to detect the upper and lower eyelids of the eye, which can make the trained eye The key point detection model can be based on the measured deviation between the distance between the upper and lower eyelids measured by the image output and the actual distance between the upper and lower eyelids, and further, can correct the measured distance between the upper and lower eyelids according to the measured deviation , To a certain extent, improve the accuracy of the measured distance between the upper and lower eyelids.

4. When the annotator is marking the upper and lower eyelids of the eyes in the face image, the specified eyelid curve of the specified eyelid can be generated in real time according to the annotation points marked by the annotator, and displayed, so that the annotator can mark the marked points Check to determine whether the marked point is the eyelid point or eye corner point on the designated eyelid. To a certain extent, it can efficiently ensure the accuracy of the eyelid point or eye corner point marked by the labeler, and improve the labeling efficiency of the labeler.

5. Utilize the eye image that includes the marked equal-divided eyelid points and marked eye corner points, and use the corresponding training data including calibration information including the marked equal-divided eyelid points and marked eye corner points to train the initial eye key The point detection model is trained to obtain an eye key point detection model used to detect the equally divided eyelid points and the corner points of the upper and lower eyelids of the eyes in the image. The eye key point detection model can determine the eye key points with obvious semantic features from the image, and to a certain extent guarantee the stability and detection accuracy of the eye key point detection model.

6. Before training the eye key point detection model, correct the eye image, and then train the eye key point detection model based on the corrected eye image and its corresponding calibration information, which can reduce the training of the eye to a certain extent The training burden of the key point detection model, and to a certain extent improve the detection accuracy of the key point detection model of the eye in the image.

7. Perform mirror image processing on the left eye image or the right eye image to obtain a mirror image, and then splice the mirror image and the unmirrored image to obtain a spliced image; subsequently, the spliced image and its corresponding calibration information can be used to train the initial eye The key point detection model is trained to obtain the eye key point detection model used to detect the equally divided eyelid points and corner points of the upper and lower eyelids of the eyes in the image, which can shorten the training time to a certain extent. Later, when using the eye key point detection model to detect the image, the left eye image and the right eye image determined from the image can be processed first, and the correction, mirroring, and stitching processing can be performed to make the eye The key point detection model detects the key points of the eyes in the two human eyes in the processed image at the same time, that is, through a detection process, the upper and lower eyelids of the two human eyes in the processed image can be detected The key points of the eye simplifies the detection process of the key points of the eye using the eye key point detection model.

8. After determining the eyelid points and corner points of the upper and lower eyelids in the image using the key point detection model obtained by training, use the sobel algorithm to extract the edges of the image to obtain the grayscale edge map corresponding to the image, and then, Using the equally divided eyelid points and eye corner points of the upper and lower eyelids, multiple sets of reference points corresponding to the upper eyelid and multiple sets of reference points corresponding to the lower eyelid are determined respectively, and then the upper and lower eyelids are respectively targeted based on their corresponding multiple sets of reference points and presets The curve fitting algorithm determines multiple reference curves that characterize the upper eyelid or the lower eyelid, and use the reference curve characterizing the upper eyelid and the sum of the pixel values of the pixels on the upper eyelid curve to be detected and the grayscale edge map to determine The curve that best fits the upper eyelid in the grayscale edge map is used as the target upper eyelid curve to characterize the upper eyelid; in the same way, use the reference curve that characterizes the lower eyelid and the upper eyelid curve to be detected and the upper eyelid curve in the grayscale edge map The sum of the pixel values of the pixel points determines the curve that best fits the lower eyelid in the grayscale edge map as the target lower eyelid curve that characterizes the upper eyelid. To a certain extent, improve the detection accuracy of the upper and lower eyelids in the image.

Description of the drawings

In order to explain the embodiments of the present invention or the technical solutions in the prior art more clearly, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained from these drawings without creative work.

FIG. 1 is a schematic flowchart of a method for marking eye key points according to an embodiment of the present invention;

2A, 2B, 2C, and 2D are schematic diagrams of marking eyelid curves corresponding to face images;

FIG. 3 is a schematic flowchart of a method for training an eye key point detection model provided by an embodiment of the present invention;

4 is a schematic diagram of a structure of an eye key point marking device provided by an embodiment of the present invention;

FIG. 5 is a schematic structural diagram of a training device for an eye key point detection model provided by an embodiment of the present invention.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the protection scope of the present invention.

It should be noted that the terms "including" and "having" in the embodiments of the present invention and the drawings and any variations thereof are intended to cover non-exclusive inclusions. For example, the process, method, system, product or device that contains a series of steps or units is not limited to the listed steps or units, but optionally also includes unlisted steps or units, or optionally also includes Other steps or units inherent to these processes, methods, products or equipment.

The embodiment of the present invention discloses an eye key point labeling and a training method and device for its detection model, so as to realize labeling eye key points with obvious semantic features and improve labeling efficiency to a certain extent. The embodiments of the present invention will be described in detail below.

FIG. 1 is a schematic flowchart of a method for marking key eye points according to an embodiment of the present invention. This method is applied to an electronic device, which can be a device with strong computing and processing capabilities, such as a server. In order to make the layout clear, in the subsequent description, the electronic device that implements the method of marking the key points of the eye can be the first An electronic device. The method specifically includes the following steps:

S101: Obtain a face image and an eyelid curve corresponding to each face image.

Among them, the face image is marked with the marked corner points of the eyes and the marked eyelid points of the upper and lower eyelids, and each marked eyelid curve includes: the marked upper eyelid which is generated based on the corresponding marked eyelid point and marked eye corner points and represents the upper eyelid Curves and labeled lower eyelid curves that characterize the lower eyelid.

In the embodiment of the present invention, the human face image may include two eyes of the person's face, as shown in Figures 2B and 2D, or only one eye of the face, as shown in Figures 2A and 2C, this is both possible . Wherein, when the face image includes two eyes of a person's face, the face image is marked with the marked corner points of the two eyes and the marked eyelid points of the upper and lower eyelids. At this time, the marked eyelid curve corresponding to the face image includes : The marked eyelid curve corresponding to the left eye, and the marked eyelid curve corresponding to the right eye. The labeled eyelid curve corresponding to the left eye includes: a labeled upper eyelid curve that represents the upper eyelid of the left eye and a labeled lower eyelid curve that represents the lower eyelid of the left eye, which is generated based on the labeled eyelid point and the labeled eye corner point corresponding to the left eye. The labeled eyelid curve corresponding to the right eye includes: a labeled upper eyelid curve that represents the upper eyelid of the right eye and a labeled lower eyelid curve that represents the lower eyelid of the right eye, which is generated based on the labeled eyelid points and the labeled eye corner points corresponding to the right eye.

In an implementation of the present invention, the marked eyelid curve corresponding to each face image may be based on the marked eyelid point of the upper eyelid of the marked eye when the marking staff is marking the upper and lower eyelids of the eyes in the face image And generated by a preset curve fitting algorithm, and generated based on the marked eyelid points of the lower lid of the marked eye and a preset curve fitting algorithm. Furthermore, when the first electronic device obtains the face image, it also obtains the marked eyelid curve corresponding to the face image.

In another implementation of the present invention, the step of obtaining the eyelid curve corresponding to each face image may include: for each face image, based on the marked eye corner point, the marked eyelid point of the upper eyelid in the face image, and The preset curve fitting algorithm fits the upper eyelid curve that characterizes the upper eyelid; and based on the marked eye corner points, the lower eyelid marked eyelid points in the face image, and the preset curve fitting algorithm, The lower eyelid curve representing the lower eyelid is drawn to obtain the eyelid curve corresponding to the face image.

In this implementation, after the first electronic device obtains the face image, it can be fitted based on the marked corner points of the eyes contained in the face image and the marked eyelid points of the upper eyelid, and a preset curve fitting algorithm. The upper eyelid curve that characterizes the upper eyelid of the eye is used as the marked upper eyelid curve; and based on the marked corner points of the included eyes and the marked eyelid points of the lower eyelid marked in the face image, and the preset curve fitting algorithm, The lower eyelid curve that characterizes the lower eyelid of the eye is obtained as the labeled lower eyelid curve. Wherein, the preset curve fitting algorithm may be a cubic spline interpolation algorithm. It is understandable that each eye includes two marked eye corner points, which are the intersection of the marked upper eyelid curve representing the upper eyelid of the eye and the marked lower eyelid curve representing the lower eyelid of the eyelid.

S102: For each marked eyelid curve, based on the principle of mathematical integration, integrate the marked eyelid curve to determine the first curve length marked with the upper eyelid curve and the second curve length marked with the lower eyelid curve in the marked eyelid curve; respectively; From marking the upper eyelid curve and marking the lower eyelid curve, multiple eyelid points to be used are determined.

In this step, for each marked eyelid curve, the first electronic device may integrate the marked eyelid curve based on the principle of data integration, and respectively integrate the marked upper eyelid curve in the marked eyelid curve, and determine the marked upper eyelid curve. The curve length of the eyelid curve is taken as the first curve length; the marked lower eyelid curve in the marked eyelid curve is integrated to determine the curve length of the marked lower eyelid curve as the second curve length.

Furthermore, for each marked eyelid curve, the first electronic device intensively selects points on the marked upper eyelid curve in the marked eyelid curve, and determines multiple eyelid points to be used, for example: taking a preset number of points, that is, determining A preset number of eyelid points to be used.

In the same way, the first electronic device for each marked eyelid curve intensively selects points on the marked lower eyelid curve in the marked eyelid curve, and determines multiple eyelid points to be used, for example: take a preset number of points, namely Determine the preset number of eyelid points to be used.

S103: For each marked eyelid curve, based on the first curve length of the upper eyelid curve marked on the marked eyelid curve, the second curve length marked on the lower eyelid curve, multiple eyelid points to be used, and a preset number of equal division points, In marking the eyelid curve, marking the upper eyelid curve and marking the lower eyelid curve, respectively determine the preset number of equal points minus 1 equal upper eyelid point and the preset number of equal points minus 1 equal lower eyelid point.

In this step, after the first electronic device determines the multiple eyelid points to be used on the upper eyelid curve and the multiple eyelid points to be used on the lower eyelid curve in each eyelid curve, the first curve length and the preset Set the number of equally divided points to determine the distance between each two adjacent equally divided upper eyelid points that need to be marked, where the distance between each adjacent two equally divided upper eyelid points is equal to the length of the first curve and The ratio of the number of equal points is preset. Subsequently, the first electronic device may calculate the distance between the corner point of the eye and the eyelid point to be used from a certain corner of the eye. When the distance between a certain eyelid point to be used at the corner of the eye is determined, it is every adjacent If the distance between the two halved upper eyelid points is an integer multiple of, the eyelid point to be used can be determined as the halved upper eyelid point. The integer multiple can be 1 time to the preset number of equal division points minus 1 time. Or, the electronic device can calculate the distance between the eyelid point and the eyelid point to be used from a certain corner of the eye, and when it is determined that the distance between the eyelid point to be used and the eyelid point is every two adjacent If the distance between the upper eyelid points is divided into two equal parts, the eyelid point to be used is determined as the first divided upper eyelid point, and then the first divided upper eyelid point is taken as the starting position, and the first divided upper eyelid point is sequentially traversed. The eyelid points to be used after the upper eyelid points are divided, and the distance between a certain eyelid point to be used and the first upper eyelid point is determined to be between every two adjacent upper eyelid points Then, the eyelid point to be used is determined to be the second equally divided upper eyelid point, and so on to determine the preset number of equally divided upper eyelid points minus 1 equally divided upper eyelid point.

In the same way, the length of the second curve and the preset number of equal points are used to determine the distance between each two adjacent equally divided lower eyelid points that need to be marked, where each adjacent two equally divided lower eyelids The distance between the points is equal to the ratio of the length of the second curve to the preset number of equal points. Subsequently, the first electronic device may calculate the distance between the corner point of the eye and the eyelid point to be used from a certain corner of the eye. When the distance between a certain eyelid point to be used at the corner of the eye is determined, it is every adjacent Is an integer multiple of the distance between the two equally divided lower eyelid points, then the eyelid point to be used can be determined as the equally divided lower eyelid point. The integer multiple can be 1 time to the preset number of equal division points minus 1 time. Or, the electronic device can calculate the distance between the eyelid point and the eyelid point to be used from a certain corner of the eye, and when it is determined that the distance between the eyelid point to be used and the eyelid point is every two adjacent If the distance between the lower eyelid points is divided into two, it is determined that the eyelid point to be used is the first equally divided lower eyelid point, and then, taking the first equally divided lower eyelid point as the starting position, traverse the first The to-be-used eyelid points after the lower eyelid points are divided, and the distance between a certain to-be-used eyelid point and the first lower eyelid point is determined to be between every two adjacent lower eyelid points Then, the eyelid point to be used is determined as the second equally divided lower eyelid point, and so on to determine the preset number of equally divided lower eyelid points minus one equally divided lower eyelid point.

Wherein, when the face image includes the left eye and the right eye, the above steps are performed for the marked eyelid curve corresponding to the left eye, and the above steps are performed for the marked eyelid curve corresponding to the right eye.

S104: Determine the marked eye corner point, equally divided upper eyelid point, and equally divided lower eyelid point in each face image as the key eye point corresponding to each face image.

After the first electronic device determines the halved eyelid points of the upper and lower eyelids of the eyes contained in the face image, the halved eyelid points of the upper and lower eyelids of the eyes contained in the face image and mark the eye corner points are determined as the face image Corresponding key points of the eye. Wherein, the halved eyelid points of the upper and lower eyelids of the eye include the halved upper eyelid points of the upper eyelid and the halved lower eyelid points of the lower eyelid.

Furthermore, the first electronic device may mark the divided eyelid points in the upper and lower eyelids of the eyes of the face image based on the position information of the divided eyelid points of the upper and lower eyelids of the determined human face image, and save the The eyelid points are divided into the upper and lower eyelids of the eyes and the face images are marked with the corner points of the eyes. In one case, the labeled eyelid curve corresponding to each face image can also be saved. It is also possible to save the position information of the equally divided eyelid points of the upper and lower eyelids of the eyes and the marked eye corner points in the face image in the form of text.

By applying the embodiment of the present invention, the eyelid marking curve corresponding to the face image can be integrated, and the first curve length marking the upper eyelid curve and the second curve length marking the lower eyelid curve in the marking eyelid curve can be determined, and the markings are respectively The eyelid curve and the marked lower eyelid curve are densely selected to determine multiple eyelid points to be used from the marked upper eyelid curve and the marked lower eyelid curve, and then mark the upper eyelid curve based on the first curve length of the marked upper eyelid curve The number of eyelid points to be used and the preset number of equal division points are determined from the marked upper eyelid curve, and the preset number of equal division points minus one equal upper eyelid point is determined; and the second curve length and label are based on the marked lower eyelid curve. The multiple eyelid points to be used on the lower eyelid curve and the preset number of equal division points are determined from the marked lower eyelid curve to determine the preset number of equal division points minus 1 equal lower eyelid point to achieve semi-automatic inclusion from the face image The upper and lower eyelids of the eyes are marked with equally divided eyelid points with obvious semantic characteristics, which realizes marking the key points of the eye with obvious semantic characteristics and improves the marking efficiency to a certain extent.

In another embodiment of the present invention, in the step of determining the marked eye corner point, equally divided upper eyelid point, and equally divided lower eyelid point in each face image as the key eye point corresponding to each face image After that, the method may further include:

For each face image, based on the key points of the eyes in the face image, cut out the image of the area where the eyes are located from the face image to obtain the eye image marked with the key points of the eye; the eye image and its corresponding calibration The information is determined as the training data of the eye key point detection model used to detect the eye key points of the eyes in the image, where the calibration information includes position information of the eye key points in the corresponding eye image.

In this embodiment, after the first electronic device determines the eye key points in each face image, it can determine the eye area based on the position of the eye key points in the face image, and then obtain the Intercept the image of the area where the eye is located, and obtain the eye image marked with the key points of the eye. Wherein, the area where the eye is located may be the smallest rectangular area containing the eyes, or it may be an area where the smallest rectangular area containing the eyes extends to the surrounding by preset pixel points, which is all right. Wherein, extending the preset pixel points to the surrounding is: respectively expanding the preset pixel points in the upper, lower, left, and right directions of the smallest rectangular area including the eyes. Each face image has a corresponding relationship with the eye image cut out.

After determining the eye image, for each eye image, determine the position information of the eye key points marked in the eye image in the eye image as the calibration information corresponding to the eye image, and mark the eye key points The eye image of and the calibration information corresponding to the eye image determine the training data of the eye key point detection model used to detect the eye key points of the eyes in the image.

It is understandable that through the eye image marked with the eye key points and the calibration information corresponding to the eye image, it is possible to train to detect the eyelid points and the eye corner points in the upper and lower lids of the eyes in the image. Point detection model. Among them, in order to make the layout clear, the process of training the eye key point detection model will be introduced later.

In another embodiment of the present invention, the step of determining the eye image and its corresponding calibration information as the training data of the eye key point detection model for detecting the eye key points of the eyes in the image may include : Obtain the real eye opening and closing length corresponding to each face image;

Obtain the measured eye opening and closing length corresponding to each face image, where the measured eye opening and closing length is: based on the length determined by the upper and lower eyelids of the target 3D face model corresponding to the face image, the target 3D face model is: A face model determined based on the facial feature points in the corresponding face image and a preset three-dimensional face model. The target three-dimensional face model includes: based on the marked eye corner points in the face image, equally divided upper eyelid points and Divide the lower eyelid points equally to construct the upper and lower eyelids of the eye;

The eye image and its corresponding calibration information are determined as the training data of the eye key point detection model used to detect the upper and lower eyelids of the eye in the image, where the calibration information includes: the eye marked in the corresponding eye image The location information of the key points, and the measured deviation corresponding to the face image corresponding to the eye image.

In one case, when the face image is a face image collected by a multi-image acquisition device system, it can be based on multiple image acquisition devices in the multi-image acquisition device system to target the face of the same person at the same time. The collected images containing the person's face are constructed to construct a real three-dimensional face model of the person's face, and the real three-dimensional face model of the person's face includes the upper and lower eyelids of the person's eyes. Based on the center eyelid points of the upper and lower eyelids of the person’s eyes contained in the real three-dimensional face model, the distance between the upper and lower eyelids of the person’s eyes can be determined as the real eye opening and closing corresponding to each image containing the person’s face Length, that is, the actual eye opening and closing length corresponding to each face image containing a person's face. Wherein, the above-mentioned real three-dimensional face model can be constructed based on any current technology that can reconstruct a three-dimensional face model of a person based on a plurality of images containing the face of a person.

Wherein, the above-mentioned process based on the center eyelid points of the upper and lower eyelids of the person's eyes included in the real three-dimensional face model to determine the distance between the upper and lower eyelids of the person's eyes may be:

Take an eye of a person as an example: obtain the first two-dimensional position information of the first corner point of the eye in the face image, the second two-dimensional position information of the second corner point, and each equally divided eyelid point in the upper and lower eyelids Determine the first three-dimensional position information of the first corner of the eye and the second three-dimensional position information of the second corner of the eye from the real three-dimensional face model; based on the first three-dimensional position information, the second three-dimensional The position information and the preset curve equation are used to construct the first corner of the eye constraint, where the first corner of the eye constraint is expressed by formula (1); based on the preset first value, the preset second value and the first corner of the eye constraint, the first corner of the eye constraint is constructed The second eye corner constraint, the first value and the second value are used to constrain the value range of the independent variable in the first eye corner constraint. Among them, the first value can be 0, the second value can be 1, and the second eye canthus constraint is expressed by formula (2); based on the curve equation, the third position information of each eyelid point, and the pose information of each image acquisition device and The internal reference information is used to construct the re-projection error constraints corresponding to the equally divided eyelid points, where the above-mentioned re-projection error constraints can be based on the third and second position information of each equally divided eyelid point, and the true three-dimensional corresponding to each equally divided eyelid point The distance between the equally divided eyelid space points in the face model and the projection position in the face image is constructed; based on the order of the equally divided eyelid points in the face image, the order constraint can be constructed by the formula ( 3); Based on the first corner of the eye constraint, the second corner of the eye constraint, the reprojection error constraint, and the order constraint, construct the eyelid space curve equation used to characterize the upper and lower eyelids of the eye, that is, the four constraints corresponding to the above are combined The equation is solved to obtain the eyelid space curve equation used to characterize the upper eyelid and the lower eyelid of the eye. Based on the eyelid space curve equations of the upper and lower eyelids, determine the bisecting point of the upper eyelid and the bisecting point of the lower eyelid, and calculate the distance between the bisecting point of the upper eyelid and the bisecting point of the lower eyelid , As the distance between the upper and lower eyelids of the eye.

Among them, formula (1):

Among them, (x ₀ , y ₀ , z ₀ ) represents the first three-dimensional position information, (x ₁ , y ₁ , z ₁ ) represents the second three-dimensional position information, a ₁ , a ₂ , a ₃ , b ₁ , b ₂ , B ₃ , c ₁ , c ₂ , and c ₃ are the required coefficients, and t is the independent variable.

Formula (2):

Formula (3): 0≤t ₁ ≤t ₂ …≤t _i …≤t _M ≤1(3); among them, when the eye space curve equation characterizing the upper eyelid is determined, t _i represents the equivalence of the ith upper eyelid The third two-dimensional position information of the eyelid points. Correspondingly, M represents the number of equal eyelid points of the upper eyelid; when the eye space curve equation representing the lower eyelid is determined, t _i represents the equal division of the i-th lower eyelid The third two-dimensional position information of the eyelid points, correspondingly, M represents the number of equal eyelid points of the lower eyelid.

After obtaining the real eye opening and closing length corresponding to each face image, the first electronic device can obtain the actual measured eye opening and closing length corresponding to each face image. Among them, the actual measured eye opening and closing length can be: the length determined based on the upper and lower eyelids of the eyes in the target three-dimensional face model corresponding to the face image, the target three-dimensional face model can be: using 3DMM (3D Morphable Models, three-dimensional deformable Model) technology, a face model determined based on the facial feature points in the corresponding face image and a preset three-dimensional face model. The target three-dimensional face model includes: based on the marked eye corner points in the face image, etc. The upper eyelid point and the lower eyelid point are divided equally to construct the upper and lower eyelids of the eye.

Furthermore, for each face image, the first electronic device calculates the ratio of the actual eye opening and closing length corresponding to the face image to the actual measured eye opening and closing length as the actual measurement deviation corresponding to the face image; wherein the actual measurement deviation can be determined The degree of difference between the actual eye opening and closing length of the corresponding eye and the measured eye opening and closing length, where the greater the actual measurement deviation, the greater the degree of difference between the actual eye opening and closing length of the corresponding eye and the measured eye opening and closing length Big. Among them, when the face image includes the left eye and the right eye, the real eye opening and closing length and the measured eye opening and closing length corresponding to the left eye in the face image can be obtained respectively, and the real eye opening and closing length and the measured eye corresponding to the right eye Opening and closing length; based on the actual eye opening and closing length corresponding to the left eye and the measured eye opening and closing length, determine the actual measurement deviation corresponding to the left eye; based on the real eye opening and closing length corresponding to the right eye and the measured eye opening and closing length, determine the right eye correspondence The measured deviation of is taken as the measured deviation corresponding to the face image.

Furthermore, the first electronic device determines the eye image and its corresponding calibration information as the training data of the eye key point detection model for detecting the upper and lower eyelids of the eye in the image, and the calibration information includes the corresponding eye The position information of the key points of the eye marked by the image and the actual measured deviation corresponding to the face image corresponding to the eye image.

Subsequent, the eye key point detection model trained based on the eye image and its corresponding calibration information can not only detect the equally divided eyelid points in the upper and lower lids of the eyes in the image, but also detect the corresponding eye in the image The actual measurement deviation, and further, the actual measurement eye opening and closing length corresponding to the image can be corrected based on the actual measurement deviation, so as to obtain a more accurate eye opening and closing length. Furthermore, when performing other tasks with the more accurate eye opening and closing length, the accuracy of the results of other tasks can be improved.

In another embodiment of the present invention, before the step of obtaining a face image and an eyelid curve corresponding to each face image, the method may further include:

The process of labeling the upper and lower eyelids of the eyes of the person's face in each face image, wherein for each face image, the following steps are performed to label the upper and lower eyelids of the eyes of the person's face in each face image:

Obtain and display a first face image, where the first face image contains the eyes of the person's face, and the first face image is one of the face images;

Receiving an annotation instruction triggered by an annotator for the upper and lower eyelids of the eyes in the first face image, where the annotation instruction carries position information of the marked annotation point;

If it is detected that the tagger has triggered the tagging instruction for the specified eyelid in the first face image at least twice, based on the location information of the tag point carried by the tagging instruction triggered by the tagger at least twice, and the preset curve fitting algorithm, determine The specified eyelid curve representing the specified eyelid, where the specified eyelid is: the upper eyelid and the lower eyelid of the eye in the first face image;

The designated eyelid curve is displayed in the first face image, so that the labeler detects whether the labelled point is an eyelid point or an eye corner point on the designated eyelid.

Before obtaining the face image and the labeled eyelid curve corresponding to each face image, a process of labeling the upper and lower eyelids of the eyes of the person's face in each face image may also be included. The labeling process is a process in which the labeling person marks the corner points and the eyelid points of the upper and lower eyelids of the eyes of the person's face in the face image.

In the embodiment of the present invention, the first electronic device can obtain the first face image after detecting the face image annotation start instruction triggered by the annotator, and the first face image is any one of the aforementioned face images, The first face image contains the eyes of the person's face. The first electronic device displays the first face image, the annotator can mark the upper and lower eyelids of the eyes in the first face image, and the first electronic device receives the trigger from the annotator for the upper and lower eyelids of the eyes in the first face image. A marking instruction, which carries the position information of the marked marking point.

The first electronic device displays a preset labeling icon at a corresponding position in the first face image based on the position information of the labeling point carried in the labeling instruction; the first electronic device can count the labeling personnel’s real-time statistics on the first face The number of marking instructions triggered by the specified eyelid in the image; if it is detected that the marking person has triggered the marking instruction for the specified eyelid in the first face image at least twice, that is, when the specified eyelid includes at least two marking points, the first electronic device is based on The position information of the label points carried by the labeling instructions triggered by the labeler at least twice, and the preset curve fitting algorithm, determine the specified eyelid curve representing the specified eyelid; and display the specified eyelid curve in the first face image, and mark The person can observe the specified eyelid curve and detect whether the specified eyelid curve coincides with the specified eyelid in the first face image, that is, detect whether the marked point is an eyelid point or an eye corner point on the specified eyelid.

Subsequently, when the annotator determines that the position of the marked point needs to be modified, the annotator can trigger the marking point position modification instruction, and the first electronic device obtains the marking point position modification instruction, wherein the marking point position modification instruction carries the Modify the current position information of the marked point of the modified position and the target position information to be modified, the first electronic device moves the marked point of the position to be modified from the position corresponding to the current position information to the position corresponding to the target position information to be modified , That is, display the preset label icon at the location corresponding to the target location information, and delete the preset label icon displayed at the location corresponding to the current location information. In addition, the first electronic device determines a new specified eyelid curve that characterizes the specified eyelid based on the new position information of the modified marked point in the specified eyelid and the position information of other marked points, and a preset curve fitting algorithm, And it is displayed in the first face image, so that the annotator continues to detect whether the marked point is an eyelid point or an eye corner point on the designated eyelid.

Until the first electronic device detects that the annotator triggers a save instruction for the first face image, the first electronic device saves the first face image and the annotation points contained at the moment when the save instruction is triggered, and each annotation Point location information. In one case, the marked points included in the trigger moment of the save instruction in the first face image may include two corner points of the eyes, the upper eyelid point of the upper eyelid and the lower eyelid point of the lower eyelid. Wherein, the number of upper eyelid points and lower eyelid points can be the same or different, for example: the number of upper eyelid points can be 3, the number of lower eyelid points can be 4, and so on. Wherein, the preset labeling icon can be a solid circle or a hollow circle, and can also be a solid image or a hollow image of other shapes, which is all right.

It is understandable that the labeling process can be executed on the first electronic device, or can be executed on another electronic device different from the first electronic device, all of which are possible. If the labeling process is performed on another electronic device different from the first electronic device, it may be that after the labeling personnel finish labeling the face image, upload the face image with the eye corner points and eyelid points marked on the upper and lower eyelids of the eyes. Cloud, so that when the first electronic device performs the marking of the key points of the eye, it can obtain the face image of the upper and lower eyelids marked with the eye corner points and the eyelid points from the cloud.

The embodiments of the present invention can efficiently ensure the accuracy of the eyelid points or the corner points of the eyes marked by the labeler to a certain extent, and improve the labeling efficiency of the labeler.

As shown in FIG. 2A, FIG. 2B, FIG. 2C, and FIG. 2D, it is a schematic diagram of an eyelid curve corresponding to a face image. Wherein, the face image shown in FIG. 2A includes one eye, and the eye can be completely detected, and the marked eyelid curve corresponding to the face image may include the marked upper eyelid curve and the marked lower eyelid curve of the eye. Wherein, the occluded position in FIG. 2A, FIG. 2B, FIG. 2C, and FIG. 2D is the position where the human face is located.

The face image shown in FIG. 2B includes two eyes, and both eyes can be completely detected. The labeled eyelid curve corresponding to the face image may include the labeled upper eyelid curve and the labeled lower eyelid curve of the left eye, and the right The eye is marked with the upper eyelid curve and the lower eyelid curve. The right image in Figure 2B is a partial enlarged view of the position of the eye in the left image.

The face image shown in Figure 2C includes an eye, and the inner corner of the eye is occluded. When annotating the eyelid points and corner points of the upper and lower lids of the partially occluded eyes in this type of face image, one In this case, the eyelid points and eye corner points at the occluded position of the occluded eye can be directly marked through experience. In another case, when the face image is an image collected by a multi-image acquisition device system, a three-dimensional face model of the person corresponding to the face image can be reconstructed based on other face images corresponding to the face image , And then, from the three-dimensional face model, determine the eye space point corresponding to the position where the eye is blocked in the face image, and then reproject the eye space point to the face image to determine the The eyelid point and/or eye corner point at the occluded position. The above-mentioned partially blocked eyes may refer to eyes whose blocked area does not exceed a preset area. The right image in Figure 2C is a partial enlarged view of the position of the eye in the left image.

The face image shown in FIG. 2D includes two eyes, and one eye can be detected at night, and one eye is partially blocked. At this time, the annotator can directly mark the eyelid points and eye corner points at the occluded position of the occluded eye based on experience. Or, when the face image is an image collected based on a multi-image acquisition device system, a three-dimensional face model of the person corresponding to the face image can be reconstructed based on other face images corresponding to the face image, and further, From the three-dimensional face model, determine the eye space point corresponding to the position where the eye is blocked in the face image, and then reproject the eye space point to the face image to determine the blocked position Eyelid point and/or corner point of the eye. The right image in Figure 2D is a partial enlarged view of the position of the left eye in the left image.

Wherein, the aforementioned face image and other face images corresponding to the face image are all images collected by the multi-image collection device system, and are images collected at the same time.

Among them, in order to avoid misrecognition of the first electronic device, for example, the marked points marked for the lower eyelid are recognized as marked points marked for the upper eyelid, or the marked points marked for the upper eyelid are recognized as marked marked for the lower eyelid Point, the first electronic device can be set to restrict the labeling staff to separate the labeling points of the upper and lower eyelids of the eyes. For example, the labeling staff can be instructed to label the labeling points of the upper eyelids of the eyes. At this time, the labeling staff cannot label the eye The marking point of the lower eyelid; after detecting that the labeler triggers the instruction indicating that the marking point of the upper eyelid has been marked, the labeling staff is then instructed to mark the marking point of the lower eyelid of the eye. At this time, the marking staff cannot mark the upper eyelid of the eye Marked points.

Correspondingly, in one case, when the annotator marks the upper and lower eyelids of the eyes in the face image, the intersection of the upper and lower eyelids of the eyes is the inner and outer corners of the corresponding eyes. The first electronic device ensures that the eyelid curves of the upper and lower eyelids can intersect with the corresponding inner and outer corners of the eye respectively. The first electronic device detects that the marking point in the upper eyelid has been marked. When the position of a marking point marked in the lower eyelid by the marking staff is less than a preset distance from the marked eye corner point, Snap the label point from the current position to the position of the corner of the eye. Wherein, the marked corner point of the eye may refer to the marked point with the smallest and largest horizontal axis coordinate of the marked point in the marked upper eyelid.

3 is a schematic flowchart of a method for training an eye key point detection model provided by an embodiment of the present invention. This method is applied to an electronic device, which can be a device with strong computing and processing capabilities, such as a server, etc., in which, for clear layout, in the subsequent description, the electronic device that implements the training method of the eye key point detection model can be used Become a second electronic device. The method specifically includes the following steps:

S301: Obtain training data. Among them, the training data includes the upper and lower eyelid equal eyelid points marked with the eyes and the eye images marked with the corner points of the eyes, and the calibration information corresponding to each eye image. The calibration information includes: the equal eyelid points and labels marked in the corresponding eye image The position information of the corner points of the eyes. The equally divided eyelid points include: based on the marked eyelid curve corresponding to the face image and the principle of mathematical integration, the equal division of the upper eyelid of the eye in the face image is determined. The upper eyelid point and the lower eyelid are divided equally Eyelid points; marking eyelid curves include: based on the marked corner points of the eyes and the marked eyelid points of the upper and lower eyelids in the corresponding face image, the marked upper eyelid curve representing the upper eyelid and the marked lower eyelid curve representing the lower eyelid are generated, Each eye image is an image of the area where the eye is cut out from the corresponding face image.

The second electronic device may first obtain training data, where the training data may include multiple eye images and calibration information corresponding to each eye image. The eye image is marked with the upper and lower eyelids of the eye and the eyelid points and the corner points. Among them, the specific obtaining process of dividing the eyelid points and marking the eye corner points can refer to the obtaining process of obtaining the equal eyelid points and marking the eye corner points in the marking process of the above-mentioned key points of the eye, and will not be repeated here.

S302: Input the eye image and the position information of the equally divided eyelid points and the marked eye corner points included in the calibration information corresponding to each eye image into the initial eye key point detection model to be trained to detect the upper and lower positions of the eyes in the image An eye key point detection model that divides the eyelid point and the corner point of the eyelid.

In this step, the initial eye key point detection model may be: a neural network model based on deep learning. The second electronic device inputs the eye image and the position information of the eye key points included in the calibration information corresponding to each eye image into the initial eye key point detection model, where the eye key points include the upper and lower eyelids of the corresponding eye The halves of the eyelid point and the corner point of the eye.

The second electronic device uses the initial eye key point detection model to extract the image features in the eye image for each eye image, and based on the extracted image features, detects the eye key points and the key points of the eye in the eye image. Its position information; and the detected position information of the key points of the eye is matched with the position information of the key points of the eye in the corresponding calibration information. If the matching is successful, it is determined that the initial key point detection model of the eye has converged to obtain The trained eye key point detection model; if the matching is unsuccessful, determine that the initial eye key point detection model has not converged, adjust the parameters of the initial eye key point detection model, and return to execute the eye image and each eye image The corresponding calibration information includes the position information of the equally divided eyelid points and the marked eye corner points, and the steps of inputting the initial eye key point detection model until the matching is successful, confirm that the initial eye key point detection model converges, and obtain the trained eye Key point detection model.

Among them, there is a correspondence between the eye image and the detected position information of the key points of the eye, and there is a correspondence between the eye image and the calibration information, and correspondingly, the detected position information of the key points of the eye has a corresponding relationship with the calibration information.

The above process of matching the position information of the detected key points of the eye with the position information of the key points of the eye in the corresponding calibration information may be: using a preset loss function to calculate each detected eye The loss value between the position information of the key point and the position information of the eye key point in the corresponding calibration information, to determine whether the loss value is less than the preset loss threshold; if it is determined that the loss value is less than the preset loss threshold, and, if The current judgment loss value is less than the preset loss threshold for more than a predetermined number of times, or the ratio of the current judgment loss value less than the preset loss threshold to the total number of times exceeds the preset ratio threshold, the match is determined to be successful, and the initial eye can be determined The key point detection model converges to obtain a trained eye key point detection model; if it is determined that the loss value is not less than the preset loss threshold, it is determined that the matching is unsuccessful.

The above process is only an example of determining the convergence of the initial eye key point detection model. The embodiment of the present invention can use any determination method that can characterize the model convergence to determine whether the initial eye key point detection model has converged, and then The key point detection model of the eye is trained.

Among them, the aforementioned preset loss function may be a loss function such as smooth L1 loss (smooth 1 norm loss), wing loss (wing loss), and KL loss (that is, KL divergence loss).

The above process of adjusting the parameters of the initial eye key point detection model can be based on the position information of the eye key points detected by the eye key point detection model in the training process and the eye parts in the corresponding calibration information. The "gap" between the location information of key points is getting smaller and smaller, and adjustments should be made. Among them, SGD (Stochastic Gradient Descent) or SGDR (stochastic gradient descent with restarts) can be used. Method) and other optimization strategies. Among them, the batch size during the training process can be 256, and the initial learning rate can be 0.04.

It is understandable that there is a correspondence between the key eye points detected from the eye image and the key eye points corresponding to the position information included in the calibration information corresponding to the eye image. The calculation of the key points of each eye detected When the loss value between the position information and the position information of the eye key points in the corresponding calibration information, it is necessary to calculate the position information of each pair of detected key eye points in the corresponding relationship, and the eye in the calibration information The loss value between the location information of the key points.

Among them, the more the number of eye images included in the obtained training data, the higher the stability of the trained eye key point detection model, and the accuracy of the detection result obtained based on the eye key point detection model Also higher.

Applying the embodiment of the present invention, using the eye image including the marked equal-divided eyelid points and the marked eye corner points, with its corresponding training data including the marked equal-divided eyelid points and the position information of the marked eye corner points, the initial training The eye key point detection model is trained to obtain the eye key point detection model used to detect the equally divided eyelid points and eye corner points in the upper and lower eyelids of the eyes in the image. The eye key point detection model can determine the eye key points with obvious semantic features from the image, and to a certain extent guarantee the stability and detection accuracy of the eye key point detection model.

In another embodiment of the present invention, before the S302, the method may further include:

For each eye image, performing the correction processing on the eye image to obtain the correction image, where the correction processing is: processing that the ordinates in the position information of the marked corner points in the eye image are all the same;

The S302 may include: inputting the position information of the halved eyelid points and the marked eye corner points included in each converted image and the calibration information corresponding to each converted image into the initial eye key point detection model to be used for training An eye key point detection model for detecting the equally divided eyelid points and eye corner points in the upper and lower eyelids of the eyes in the image.

In one case, the person’s face in the face image may be tilted. In this embodiment, in order to improve the accuracy of the detection result of the trained eye key point detection model, the eye image and its calibration information are used Before training the initial eye key point detection model, the eye image can be corrected first to obtain the corrected image, that is, the ordinates in the position information of the marked eye corner points in the eye image are all the same. And re-determine the position information of the key points of the eye in the corrected image, and update the calibration information corresponding to each corrected image based on the key points of the eye in each corrected image, that is, the position information of the eyelid points and the marked eye corner points. The key points of the eye are the position information of the eyelid points and the corner points of the eyes. And input each converted image and the calibration information corresponding to each converted image into the initial eye key point detection model. This can reduce the training burden of training the eye key point detection model to a certain extent, and improve the detection accuracy of the eye key point detection model of the eye in the image to a certain extent.

In another embodiment of the present invention, the eye image includes a left eye image and a right eye image corresponding to the left eye image;

In the above-mentioned each converted image, and the calibration information corresponding to each converted image, the position information of the divided eyelid points and the marked eye canthus points are input into the initial eye key point detection model to be trained to be used in the detection image Before the step of detecting the key points of the eyelid point and the corner point of the eye in the upper and lower eyelids of the eye, the method may further include: mirroring the left eye image or the right eye image corresponding to the left eye image to obtain a mirror image image;

The mirror image and the unmirrored image are spliced to obtain a spliced image. If the left-eye image is mirrored, the unmirrored image is the right-eye image corresponding to the left-eye image; if the right-eye image corresponding to the left-eye image is The eye image is mirrored, and the image that is not mirrored is the left eye image;

Said each converted image and the calibration information corresponding to each converted image includes the position information of the divided eyelid points and the marked eye corner points, and input the initial eye key point detection model to be trained to detect the eyes in the image The steps of detecting the key points of the eyelid point and the corner point of the eyelid in the upper and lower eyelids may include:

Input the position information of the divided eyelid points and the marked eye corner points included in each stitched image and the calibration information corresponding to each stitched image into the initial eye key point detection model to train the upper and lower eyelids for detecting the eyes in the image The key point detection model of the eyelid points and the corner points of the eye in, where the calibration information includes: the equally divided eyelid points in the mirror image and the position information of the marked eye corner points after mirroring, and the equal division in the image without mirroring The position information of the eyelid points and the marked eye corner points.

Wherein, the eye image includes: an image containing the left eye of the target person, which may be called a left eye image; and an image containing the right eye of the target person, which may be called a right eye image corresponding to the left eye image.

In order to reduce to a certain extent the use of the trained eye key point detection model to detect the complexity of the eye key point of the eye in the image, and shorten the eye key point detection model obtained using the training to detect the eye key point in the image The detection time required for key points of the eye. In this embodiment, the left-eye image or the left-eye image may be mirrored to obtain a mirrored image, and then the mirrored image and the unmirrored image are spliced to obtain the spliced image corresponding to the eye image.

Each stitched image and its corresponding calibration information are input into the initial eye key point detection model to train the initial eye key point detection model. Mirroring the left-eye image or the right-eye image corresponding to the left-eye image can make the left-eye image mirror the mirrored right-eye image, or make the right-eye image mirror the mirrored left-eye image, to a certain extent. Shorten training time. Later, when using the eye key point detection model to detect the image, the left eye image and the right eye image determined from the image can be processed first, that is, the correction, mirroring, and stitching processing can make the eye The key point detection model detects the key points of the eyes in the two human eyes in the processed image at the same time, that is, through a detection process, the upper and lower eyelids of the two human eyes in the processed image can be detected The key points of the eye simplifies the detection process of the key points of the eye using the eye key point detection model.

Among them, the above-mentioned process of splicing the mirror image and the unmirrored image to obtain the spliced image may be: splicing the mirror image and the unmirrored image in the spatial dimension or the channel dimension, where the spatial dimension is The splicing can be: splicing the mirror image and the unmirrored image left and right or up and down. Left and right splicing can be: the right side of the mirror image is spliced with the left side of the image that is not mirrored, and the left side of the mirror image is spliced with the right side of the image that is not mirrored. Top and bottom splicing may be: the upper side of the mirror image is spliced with the lower side of the image that is not mirrored, and the lower side of the mirror image is spliced with the upper side of the image that is not mirrored. The splicing of the aforementioned channel dimensions may be: splicing the mirror image and the unmirrored image back and forth, that is, superimposing and splicing the mirror image and the unmirrored image.

Correspondingly, at this time, the ordinate value of the corner point of the eye in the original image corresponding to the mirror image and the ordinate value of the corner point of the image without mirroring can be adjusted to the same value during the previous normalization process. Among them, the original image corresponding to the mirror image is: the mirror image is obtained by performing mirror image processing.

In another embodiment of the present invention, the calibration information corresponding to each eye image may further include: a measured deviation corresponding to the eye image, and the measured deviation is: the actual eye opening and closing length corresponding to the eye image and the measured eye opening and closing length The actual measured eye opening and closing length is: based on the length determined by the upper and lower eyelids of the eyes in the target three-dimensional face model corresponding to the face image, the target three-dimensional face model is: based on the facial feature points in the corresponding face image As well as the face model determined by the preset 3D face model, the target 3D face model includes: the upper and lower eyelids of the eye constructed based on the marked eye corner points, equal division of upper eyelid points and equal division of lower eyelid points in the face image ；

The S302 may include: inputting the eye image and the calibration information corresponding to each eye image including the position information of the divided eyelid points and the marked eye corner points and the actual measurement deviation corresponding to the eye image, and inputting the initial eye key point detection model , The key point detection model of the eye is obtained by training, where the key point detection model of the eye is used to detect the equal-divided eyelid points and corner points of the upper and lower lids of the eyes in the image, and detect the actual measured deviation corresponding to the image.

In this implementation, the position information of the eye image, the position information of the eye corner points and the measured deviation corresponding to the eye image can be used as the calibration information corresponding to the eye image, and then the eye image and the calibration corresponding to the eye image can be used Information, train the initial eye key point detection model, that is, input the eye image and the calibration information corresponding to the eye image into the initial eye key point detection model to train to obtain the eye key point detection model. In this implementation, The trained eye key point detection model: it can be used to detect the equally divided eyelid points and corner points of the upper and lower lids of the eyes in the image, and can also detect the actual measured deviations corresponding to the image.

Among them, the initial eye key point detection model includes: a feature extraction layer, a first feature classification layer, and a second feature classification layer. The feature extraction layer may refer to the layer used to extract image features of the image, such as the convolutional layer and the pooling layer; the first feature classification layer may refer to: the key points of the eyes in the image and their position information are detected based on the image features The fully connected layer. The second feature classification layer may refer to a fully connected layer used to detect the actual measured deviation corresponding to the image.

The above process of inputting the eye image and the calibration information corresponding to the eye image into the initial eye key point detection model to train the eye key point detection model may be: first input each eye image to the feature extraction layer, and extract the The image features in the eye image; then the image features are input to the first feature classification layer to determine the current position information of the eye key points in the eye image; and then the current position information and the corresponding calibration information of the eye key points The position information is matched. If the matching is successful, it is determined that the feature point detection branch model in the initial eye key point detection model is converged, and the middle eye key point detection model is obtained; where the feature point detection branch model includes the feature extraction layer and The model of the first feature classification layer; if the matching fails, adjust the parameters of the feature extraction layer and the first feature classification layer, and return to execute the step of inputting each eye image into the feature extraction layer to extract the image features in the eye image, Until it is determined that the matching is successful, the key point detection model of the middle eye is obtained. The middle eye key point detection model: it can detect the position information of the eye key points in the image. The middle eye key point detection model includes a trained feature extraction layer, a first feature classification layer, and an untrained second feature classification layer.

Furthermore, the eye image is input to the feature extraction layer of the middle eye key point detection model to obtain the image features of the eye image; the image feature is input to the first feature classification layer of the middle eye key point detection model to determine the eye key in the eye image Point position information; based on the position information of the key points of the eye in the eye image, determine the current measured deviation corresponding to the eye image; input the current measured deviation into the second feature classification layer in the middle eye key point detection model, and the current measured deviation Match with the actual measured deviation in the corresponding calibration information. If the matching is successful, it is determined that the middle eye key point detection model converges, and the trained eye key point detection model is obtained. The trained eye key point detection model includes training The completed feature extraction layer and the first feature classification layer and the second feature classification layer after training; if the matching fails, adjust the parameters of the second feature classification layer of the middle eye key point detection model, and return to execute the input of the eye image The feature extraction layer of the key point detection model of the middle eye, the step of obtaining the image features of the eye image, until the matching is successful, it is determined that the key point detection model of the middle eye is converged, and the key point detection model obtained by training is obtained. The eye key point detection model obtained by this training can not only detect the eye key points in the image, but also detect the actual measured deviation corresponding to the image.

In another case, it can also be: first, based on the above-mentioned eye image and the calibration information corresponding to the eye image that contains the position information of the key points of the eye, the training obtains the eye that can detect the position information of the key point of the eye in the image. The central key point detection model is used as the middle eye key point detection model, where the middle eye key point detection model includes a trained feature extraction layer, a first feature classification layer, and an untrained second feature classification layer. The feature extraction layer may refer to: the convolutional layer and the pooling layer of the middle eye key point detection model are used to extract features of the image; the first feature classification layer may refer to: the middle eye key point detection model The fully connected layer used to detect the key points of the eyes and their position information in the image. The above-mentioned second feature classification layer may refer to: a fully connected layer of the middle eye key point detection model used to detect the actual measured deviation corresponding to the image.

The second electronic device obtains multiple other eye images, and obtains calibration information corresponding to each other eye image. The calibration information corresponding to the other eye image includes the actual measured deviation corresponding to the other eye image; The calibration information corresponding to other eye images is input to the middle eye key point detection model; for each other eye image, the image features of the other eye images are extracted based on the trained feature extraction layer, and the image features are input to the trained The first feature classification layer obtains the position information of the key points of the other eye images.

The second electronic device determines the current measured deviation corresponding to the other eye image based on the position information of the eye key points of the other eye image, and inputs the current measured deviation into the untrained second feature classification of the middle eye key point detection model Layer, the current measured deviation is matched with the measured deviation in the corresponding calibration information. If the matching is successful, it is determined that the middle eye key point detection model has converged, and the trained eye key point detection model is obtained. The training completed The eye key point detection model includes the above-mentioned trained feature extraction layer, the first feature classification layer, and the trained second feature classification layer; if the matching fails, adjust the second feature classification layer of the middle eye key point detection model Return to the step of inputting other eye images and the calibration information corresponding to each other eye image to the middle eye key point detection model until the matching is successful, confirm that the middle eye key point detection model converges, and obtain the training result Eye key point detection model. The eye key point detection model obtained by this training can not only detect the eye key points in the image, but also detect the actual measured deviation corresponding to the image.

Among them, the specific calculation process of the current actual measurement deviation can be referred to the specific calculation process of the actual measurement deviation in the above-mentioned eye key point marking process, which will not be repeated here.

In another embodiment of the present invention, after the S302, the method may further include:

Obtain an image to be inspected, where the image to be inspected includes the eyes of the person to be inspected;

Input the image to be detected into the key point detection model of the eye, and determine the equal eyelid points and corner points of the upper and lower eyelids of the person to be detected in the image to be detected.

In this embodiment, after the second electronic device obtains the trained eye key point detection model, the trained eye key point detection model can be used to detect the corner points in the eyes of the person to be detected in the image to be detected And the equally divided corner points in the upper and lower eyelids, and the position information of the corner points in the image to be detected, and the position information of the equally divided corner points in the upper and lower eyelids in the image to be detected.

In an implementation manner, before the image to be detected is input into the eye key point detection model, the facial feature points of the face of the person to be detected can also be detected, where the facial feature points are used to characterize the face of the person to be detected. Where the parts are located, various parts of the face may include nose, lips, eyebrows, eyes, jaw, cheeks, ears, and forehead. The facial feature points of each part of the face can respectively include: feature points in the face that characterize the position of the nose, such as nose wings, nose bridge, and tip of the nose; can also include feature points that characterize the position of the lips, such as lips The feature points of the corners of the mouth and the outer edges of the lips; it can also include the feature points that characterize the position of the eyebrows, such as the edge of the eyebrows; it can also include the feature points that characterize the location of the eyes, such as the corner of the eye feature points , Orbital feature points, pupil feature points, etc.; can also include feature points that characterize the position of the mandible, such as feature points on the contour of the mandible, that is, feature points on the contour of the chin, etc.; can also include characterizing the location of the ear The feature points of the position, such as the feature points on the contours of the ears, etc.; it can also include the feature points that characterize the position of the forehead, such as the feature points on the contour of the forehead, such as the intersection of the hair and the forehead. Feature points.

Through the facial feature points, the area of the eyes is determined from the image to be detected, and the area of the eyes is cut out from the image to be detected, and the eye image corresponding to the image to be detected is obtained as the image of the eye to be detected. The eye image is input into the eye key point detection model obtained by the training, so as to improve the accuracy of the detected corner points in the eyes of the person to be detected and the equally divided eye corner points in the upper and lower eyelids to a certain extent.

In an implementation manner, when the eye image to be detected includes the left eye image to be detected and the right eye image to be detected, both the left eye image to be detected and the right eye image to be detected may be subjected to normalization processing, that is, the left eye image to be detected The ordinate values in the position information of the two eye corner points in the image are the same, so that the ordinate values in the position information of the two eye corner points in the right-eye image to be detected are the same; input the corrected eye image to the training eye The key point detection model is used to improve the accuracy of the detected corner points in the eyes of the person’s eyes and the equally divided eye corner points in the upper and lower eyelids to a certain extent, and to a certain extent lower the key point detection of the eyes obtained by this training The difficulty of model detection.

In one implementation, the left-eye image to be detected or the right-eye image to be detected after the normalization process is obtained, and the mirroring process is performed to obtain a mirrored eye image, and the mirrored eye image and the unmirrored eye image are stitched together to obtain the stitched eye Image, input the stitched eye image into the trained eye key point detection model, so that the trained eye key point detection model can detect the key points and their position information in the mirrored eye image at one time , And the eye key points and their position information in the unmirrored eye image; subsequently, the position information of the eye key points in the mirrored eye image is mirrored to obtain the eye key points corresponding to the mirrored eye image Position information in the image before mirroring. Furthermore, the position information of the key eye points in the left-eye image to be detected after the normalization processing and the position information of the key eye points in the right-eye image to be detected are obtained.

In another embodiment of the present invention, after the step of inputting the image to be detected into the eye key point detection model, and determining the equally divided eyelid points and corner points of the upper and lower eyelids of the person to be detected in the image to be detected, The method may also include:

Use the Sobel algorithm to extract the edge of the image to be detected, and obtain the grayscale edge map corresponding to the image to be detected;

Based on the equally divided eyelid points and corner points of the upper and lower eyelids of the eyes in the image to be detected, and the preset curve fitting algorithm, determine the eyelid curve of the upper and lower eyelids of the person to be detected as the eyelid curve to be detected, and set it in the grayscale edge map Draw the eyelid curve to be detected, where the eyelid curve to be detected includes: the upper eyelid curve to be detected that characterizes the upper eyelid of the person to be detected and the lower eyelid curve to be detected that represents the lower eyelid of the person to be detected;

Based on the equally divided eyelid points in the eyelid curve to be detected, multiple sets of reference points are determined in the grayscale edge map, where each group of reference points includes points corresponding to the equally divided eyelid points in the eyelid curve to be detected;

For each set of reference points, based on the set of reference points, the corner points in the eyelid curve to be detected, and the preset curve fitting algorithm, determine the reference curve corresponding to the set of reference points, and draw each set in the grayscale edge map The reference curve corresponding to the reference point, wherein the reference curve corresponding to each set of reference points includes: a reference upper eyelid curve representing the upper eyelid of the person to be inspected and a reference lower eyelid curve representing the lower eyelid of the person to be inspected;

In the grayscale edge map, for each first upper eyelid curve, determine the sum of the pixel values of the pixel points corresponding to the first upper eyelid curve, where the first upper eyelid curve includes: the reference upper eyelid corresponding to each set of reference points Eyelid curve and upper eyelid curve to be tested;

From the sum of the pixel values of each pixel corresponding to the first upper eyelid curve, determine the sum with the largest value, and determine the first upper eyelid curve corresponding to the largest sum as the target for characterizing the upper eyelid of the person to be inspected Upper eyelid curve;

In the grayscale edge map, for each first lower eyelid curve, determine the sum of the pixel values of the pixel points corresponding to the first lower eyelid curve, where the first lower eyelid curve includes: the reference lower eyelid corresponding to each set of reference points An eyelid curve and the lower eyelid curve to be detected;

From the sum of the pixel values of the pixels corresponding to each first lower eyelid curve, determine the sum with the largest value, and determine the first lower eyelid curve corresponding to the largest sum as the target for characterizing the lower eyelid of the person to be inspected Lower eyelid curve

Based on the principle of mathematical integration, the target eyelid curve is integrated to determine the third curve length of the target upper eyelid curve and the fourth curve length of the target lower eyelid curve in the target eyelid curve; from the target upper eyelid curve and the target lower eyelid curve respectively , To determine multiple reference eyelid points;

Based on the third curve length of the target upper eyelid curve in the target eyelid curve, the fourth curve length of the target lower eyelid curve, multiple reference eyelid points and the preset number of equal points, the target upper eyelid curve and the target lower eyelid curve from the target eyelid curve In the curve, the preset number of equal division points minus 1 equal upper eyelid point and the preset number of equal division points minus 1 equal lower eyelid point are respectively determined.

In this embodiment, in order to more accurately determine the upper and lower eyelids of the eyes of the person to be inspected in the image to be inspected, the position information of the corner points of the eyes in the image to be inspected is output by the key point detection model of the obtained training And after the position information of the equally divided eyelid points of the upper and lower eyelids, you can continue to use the Sobel sobel algorithm to perform edge extraction on the image to be detected, and obtain the gray-scale edge map corresponding to the image to be detected. The edges are all extracted. The pixel value of the pixels corresponding to the upper and lower eyelids of the eyes in the grayscale edge map can be 255, and the pixel values of the pixels corresponding to other parts at the edge can be 0, so as to indicate the edge of the image to be detected Locations, such as the upper and lower eyelids of the eye.

Based on the equally divided eyelid points and corner points of the upper and lower eyelids of the eyes in the image to be detected, and a preset curve fitting algorithm, the eyelid curve of the upper and lower eyelids of the person to be detected can be determined as the eyelid curve to be detected. It can be understood that there is a one-to-one correspondence between each pixel in the grayscale edge map and each pixel in the image to be detected. Based on the corresponding relationship, the determined eyelid curve to be detected is placed on the edge of the grayscale. It is drawn in the figure, and the eyelid curve to be detected drawn in the grayscale edge map can be used to determine the corner point of the corresponding eye and its position information, and the equally divided eyelid point and its position information.

Based on the equally divided eyelid points in the eyelid curve to be detected, multiple sets of reference points are determined around the eyelid curve to be detected in the grayscale edge map, wherein each set of reference points includes points corresponding to the equally divided eyelid points of the eyelid curve to be detected. As shown in FIG. 4, at least one set of reference points may be determined respectively at the upper and lower positions of the eyelid curve to be detected in the grayscale edge map. It is understandable that the eyelid curve to be detected includes: the upper eyelid curve to be detected that represents the upper eyelid of the eye and the lower eyelid curve to be detected that represents the lower eyelid. The foregoing determination of at least one set of reference points at the upper and lower positions of the eyelid curve to be detected in the grayscale edge map may be: respectively determining at least one set of reference points for the upper and lower positions of the upper eyelid curve to be detected, and for the lower At least one set of reference points are respectively determined at the upper and lower positions of the eyelid curve. Among them, the white curves in Fig. 4 respectively represent the position of the upper eyelid and the lower eyelid of the eye in the grayscale edge diagram, and the gray curve in Fig. 4 represents the upper eyelid curve to be detected and the lower eyelid curve to be detected in the eyelid curve to be detected. Among them, the white solid points on the gray curve indicate that the eyelid curve to be detected is equally divided into the eyelid point and the eye corner point. As shown in Figure 4, the middle eyelid points of the upper eyelid curve to be detected correspond to two sets of reference points, which are white and gray hollow points, and the middle eyelid points of the lower eyelid curve to be detected correspond to two sets of reference points, respectively White hollow dots and gray hollow dots.

For each set of reference points, based on the set of reference points, the corner points in the eyelid curve to be detected, and the preset curve fitting algorithm, determine the reference curve corresponding to the set of reference points, and draw each set in the grayscale edge map The reference curve corresponding to the reference point. In the grayscale edge map, for each first upper eyelid curve, determine the sum of the pixel values of the pixel points corresponding to the first upper eyelid curve, that is, determine the pixel points corresponding to each set of reference points corresponding to the reference upper eyelid curve The sum of pixel values, and the sum of pixel values of pixels corresponding to the upper eyelid curve to be detected.

Among them, the pixel value of the pixel at the upper and lower eyelid positions of the eye in the grayscale edge map is 255. Among the sum of the pixel values of each pixel corresponding to the first upper eyelid curve, the more the sum of the pixel values is Larger, it indicates that the first upper eyelid curve fits the upper eyelid of the eye in the grayscale edge diagram more closely, and accordingly, the first upper eyelid curve with the largest sum of pixel values is determined as the upper eyelid of the person to be inspected The target upper eyelid curve.

Similarly, in the grayscale edge map, for each first lower eyelid curve, determine the sum of the pixel values of the pixels corresponding to the first lower eyelid curve, that is, determine the reference lower eyelid curve corresponding to each set of reference points The sum of the pixel values of the pixels and the sum of the pixel values of the pixels corresponding to the lower eyelid curve to be detected. Among them, the pixel value of the pixel at the upper and lower eyelid positions of the eye in the grayscale edge map is 255. Among the sum of the pixel values of each pixel corresponding to the first lower eyelid curve, the more the sum of the pixel values is Larger, it indicates that the first lower eyelid curve fits the lower eyelid of the eye in the grayscale edge map more closely, and accordingly, the first lower eyelid curve with the largest sum of pixel values is determined as the lower eyelid of the person to be inspected The target lower eyelid curve.

Furthermore, based on the principle of mathematical integration, the target eyelid curve is integrated to determine the third curve length of the target upper eyelid curve and the fourth curve length of the target lower eyelid curve in the target eyelid curve; respectively mark the target upper eyelid curve and the lower eyelid curve The curve is intensively selected, for example: take out two points of the preset book; that is, determine multiple reference eyelid points from the target upper eyelid curve, and determine multiple reference eyelid points from the target lower eyelid curve; further, based on the target The third curve length of the upper eyelid curve, multiple reference eyelid points in the target upper eyelid curve, and the preset number of equal division points are determined from the target upper eyelid curve minus 1 equal division upper eyelid point. Based on the fourth curve length of the target lower eyelid curve, multiple reference eyelid points in the target lower eyelid curve, and the preset number of equal points, the preset number of equal points minus 1 equal lower eyelid point is determined from the target lower eyelid curve .

Wherein, based on the third curve length of the target upper eyelid curve, multiple reference eyelid points in the target upper eyelid curve, and the preset number of equal division points, the preset number of equal division points minus 1 equal division is determined from the target upper eyelid curve For the process of upper eyelid point, please refer to the first curve length of the upper eyelid curve marked on the marked eyelid curve, multiple eyelid points to be used, and the preset number of equal points, and the preset number of equal points is determined from the marked upper eyelid curve. The process of dividing the upper eyelid point by 1 equal. In addition, based on the fourth curve length of the target lower eyelid curve, multiple reference eyelid points in the target lower eyelid curve, and the preset number of equal division points, the preset number of equal division points minus 1 equal division is determined from the target lower eyelid curve. For the process of lower eyelid points, refer to the second curve length of the lower eyelid curve marked on the marked eyelid curve, multiple eyelid points to be used, and the preset number of equal points, and the preset number of equal points can be determined from the marked lower eyelid curve. The process of subtracting one point of the lower eyelid equally will not be repeated here.

Corresponding to the foregoing method embodiment, FIG. 5 is a schematic structural diagram of an eye key point marking device provided in an embodiment of the present invention. The device includes:

The first obtaining module 510 is configured to obtain a face image and a marked eyelid curve corresponding to each face image, wherein the face image is marked with marked corner points of the eyes and marked eyelid points of the upper and lower eyelids. A marked eyelid curve includes: a marked upper eyelid curve representing the upper eyelid and a marked lower eyelid curve representing the lower eyelid generated based on the corresponding marked eyelid points and marked eye corner points;

The first determining module 520 is configured to, for each marked eyelid curve, integrate the marked eyelid curve based on the principle of mathematical integration, and determine the first curve length of the marked upper eyelid curve and the marked lower eyelid curve in the marked eyelid curve The second curve length; respectively from the marked upper eyelid curve and the marked lower eyelid curve, determine multiple eyelid points to be used;

The second determination module 530 is configured to, for each marked eyelid curve, based on the first curve length marked with the upper eyelid curve in the marked eyelid curve, the second curve length marked with the lower eyelid curve, and the plurality of eyelid points to be used And the preset number of equal points, from the marked upper eyelid curve and the marked lower eyelid curve, the preset number of equal points minus 1 and the preset number of equal points minus 1 are respectively determined Two equally divided lower eyelid points

The third determination module 540 is configured to determine the marked eye corner point, equal division upper eyelid point, and equal division lower eyelid point in each face image as the key eye point corresponding to each face image.

In another embodiment of the present invention, the first obtaining module 510 is specifically configured as

In another embodiment of the present invention, the device further includes: an interception module (the image is not shown) configured to mark the corner points of the eyes, divide the upper eyelid points equally, and divide the lower eyelid points equally in each face image. The eyelid points are determined as the key eye points corresponding to each face image. For each face image, based on the eye key points in the face image, the eye area image is cut out from the face image to obtain the annotation Eye image with key points of the eye;

The fourth determining module (the image is not shown) is configured to determine the eye image and its corresponding calibration information as the training data of the eye key point detection model for detecting the eye key points of the eyes in the image, Wherein, the calibration information includes position information of key eye points in the corresponding eye image.

In another embodiment of the present invention, the third determining module 540 is specifically configured to: obtain the actual eye opening and closing length corresponding to each face image; obtain the actual measured eye opening and closing length corresponding to each face image, wherein, The measured eye opening and closing length is: based on the length determined by the upper and lower eyelids of the eyes in the target three-dimensional face model corresponding to the face image, and the target three-dimensional face model is: based on the face features in the corresponding face image Points and a face model determined by a preset three-dimensional face model. The target three-dimensional face model includes: eyes constructed based on the marked eye corner points, equal division of upper eyelid points and equal division of lower eyelid points in the face image The upper and lower eyelids; for each face image, calculate the ratio of the actual eye opening and closing length corresponding to the face image to the measured eye opening and closing length as the actual measured deviation corresponding to the face image; compare the eye image and its corresponding The calibration information is determined as the training data of the eye key point detection model used to detect the upper and lower eyelids of the eye in the image, wherein the calibration information includes: the position information of the eye key points marked in the corresponding eye image , And the measured deviation corresponding to the face image corresponding to the eye image.

In another embodiment of the present invention, the device further includes: a labeling module (the image is not shown), configured to, before obtaining the face image and labeling the eyelid curve corresponding to each face The process of marking the upper and lower eyelids of the eyes of a person’s face in an image, wherein the marking module includes: an obtaining and displaying unit (the image is not shown) configured to obtain and display a first face image, wherein the first The face image contains the eyes of the person’s face, and the first face image is one of the face images; the receiving unit (the image is not shown) is configured to receive an annotator’s reference to the first face image The marking instruction triggered by the upper and lower eyelids of the eye, wherein the marking instruction carries the position information of the marked marking point; the determining unit (the image is not shown) is configured to trigger the targeting at least twice if the marking person is detected The labeling instruction for the designated eyelid in the first face image is determined to characterize the designation of the designated eyelid based on the position information of the labeling points carried in at least two labeling instructions triggered by the labeling personnel, and a preset curve fitting algorithm Eyelid curve, wherein the specified eyelid is: the upper eyelid and the lower eyelid of the eye in the first face image; a display unit (image not shown) is configured to display the first face image The specified eyelid curve is described, so that the annotator detects whether the marked point is an eyelid point or an eye corner point on the specified eyelid.

Corresponding to the foregoing method embodiment, as shown in FIG. 6, an embodiment of the present invention provides an eye key point detection model training device, which includes:

The second obtaining module 610 is configured to obtain training data, where the training data includes the upper and lower eyelid equal eyelid points marked with the eyes and the eye images marked with the corner points of the eyes and the calibration information corresponding to each eye image. The calibration The information includes: the position information of the divided eyelid points and the marked eye corner points marked in the corresponding eye image, and the divided eyelid points include: the face image determined based on the marked eyelid curve corresponding to the face image and the principle of mathematical integration The equal division of the upper eyelid of the middle eye, the upper eyelid point and the equal division of the lower eyelid point of the lower eyelid; the marked eyelid curve includes: generating based on the marked corner points of the eye marked in the corresponding face image and the marked eyelid points of the upper and lower eyelids The marked upper eyelid curve representing the upper eyelid and the marked lower eyelid curve representing the lower eyelid, each eye image is an image of the area where the eye is intercepted from the corresponding face image;

The input module 620 is configured to input the eye image and the position information of the divided eyelid points and the marked eye corner points included in the calibration information corresponding to each eye image into the initial eye key point detection model to be used for training An eye key point detection model for detecting the equally divided eyelid points and eye corner points in the upper and lower eyelids of the eyes in the image.

In another embodiment of the present invention, the device further includes: a straightening module (not shown in the figure), configured to divide the eye image and the calibration information corresponding to each eye image into equal parts. Before the position information of the eyelid points and the marked eye corner points, input the initial eye key point detection model to train the eye key point detection model used to detect the equally divided eyelid points and the corner points of the upper and lower eyelids in the image, For each eye image, performing the correction processing on the eye image to obtain the correction image, wherein the correction processing is: processing that the ordinates in the position information of the marked corner points in the eye image are all the same;

The update module (not shown in the figure) is configured to update the calibration information corresponding to each converted image including the divided eyelid points and the marked eye canthus points based on the position information of the equally divided eyelid points and the marked eye corner points of each converted image location information;

The input module 620 is specifically configured to: input the position information of the divided eyelid points and the marked eye corner points included in each converted image and the calibration information corresponding to each converted image into the initial eye key point detection model, Through training, an eye key point detection model used to detect the equally divided eyelid points and eye corner points in the upper and lower eyelids of the eyes in the image is obtained.

In another embodiment of the present invention, the eye image includes a left-eye image and a right-eye image corresponding to the left-eye image; the device further includes: a mirroring module (not shown in the figure), configured to For each converted image, and the calibration information corresponding to each converted image, including the position information of the divided eyelid points and the marked eye corner points, input the initial eye key point detection model to train the upper and lower eyelids used to detect the eyes in the image Before the eye key point detection model that equally divides the eyelid point and the corner point of the eye, the left eye image or the right eye image corresponding to the left eye image is mirrored to obtain a mirror image; stitching module (not shown in the figure) Out), configured to splice the mirror image and the unmirrored image to obtain a spliced image, wherein, if the left-eye image is mirrored, the unmirrored image is corresponding to the left-eye image Right-eye image; if mirroring is performed on the right-eye image corresponding to the left-eye image, the image that is not mirrored is the left-eye image; the input module 620 is specifically configured to: each stitched image and each The calibration information corresponding to a stitched image includes the position information of the equally divided eyelid points and the marked eye corner points, and the initial eye key point detection model is input to train to obtain the equally divided eyelid points and the upper and lower eyelids of the eyes in the image. An eye key point detection model for corner points, wherein the calibration information includes: the position information of the divided eyelid points and the marked corner points in the mirror image, and the divided eyelid points and marked eye corners in the image without mirroring Point location information.

In another embodiment of the present invention, the calibration information corresponding to each eye image further includes: a measured deviation corresponding to the eye image, and the measured deviation is: the actual eye opening and closing length corresponding to the eye image and the measured eye opening and closing length The actual measured eye opening and closing length is the length determined based on the upper and lower eyelids of the eyes in the target three-dimensional face model corresponding to the face image, and the target three-dimensional face model is: based on the person in the corresponding face image Face feature points and a face model determined by a preset three-dimensional face model. The target three-dimensional face model includes: constructing based on the marked eye corner points, equal division of the upper eyelid point and equal division of the lower eyelid point in the face image The upper and lower eyelids of your eyes;

The input module 620 is specifically configured to input the eye image, the position information of the divided eyelid points and the marked eye corner points included in the calibration information corresponding to each eye image, and the actual measured deviation corresponding to the eye image, into the initial An eye key point detection model to train to obtain an eye key point detection model, wherein the eye key point detection model is used to detect the equally divided eyelid points and eye corner points in the upper and lower lids of the eyes in the image, and detect the corresponding image The measured deviation.

In another embodiment of the present invention, the device further includes: a third obtaining module (not shown in the figure) configured to include the eye image and the calibration information corresponding to each eye image in the Divide the eyelid points and label the position information of the eye corner points, and input the initial eye key point detection model to train to obtain the eye key point detection model for detecting the divided eyelid points and the corner points of the upper and lower eyelids in the image Afterwards, an image to be detected is obtained, wherein the image to be detected includes the eyes of the person to be detected; a fifth determining module (not shown in the figure) is configured to input the image to be detected into the key point detection of the eye The model determines the equal-divided eyelid points and corner points of the upper and lower eyelids of the eyes of the person to be inspected in the image to be inspected.

In another embodiment of the present invention, the device further includes: an extraction module (not shown in the figure) configured to input the to-be-detected image into the eye key point detection model to determine the After the halved eyelid points and corner points of the upper and lower eyelids of the person to be detected in the image to be detected, the Sobel algorithm is used to perform edge extraction on the image to be detected to obtain the grayscale edge map corresponding to the image to be detected The first determination drawing module (not shown in the figure) is configured to determine the to-be-detected image based on the equally divided eyelid points and corner points of the upper and lower eyelids of the eyes in the image to be detected, and a preset curve fitting algorithm The eyelid curve of the upper and lower eyelids of the inspector is used as the eyelid curve to be inspected, and the eyelid curve to be inspected is drawn in the grayscale edge map, wherein the eyelid curve to be inspected includes: The upper eyelid curve to be detected of the upper eyelid and the lower eyelid curve to be detected that characterizes the lower eyelid of the person to be detected; the sixth determination module (not shown in the figure) is configured to be based on the equivalence in the eyelid curve to be detected The eyelid points are divided, and multiple groups of reference points are determined in the grayscale edge map, wherein each group of reference points includes points corresponding to the equally divided eyelid points of the eyelid curve to be detected; the second determination drawing module (not shown in the figure) Shown), configured to determine, for each set of reference points, a reference curve corresponding to the set of reference points based on the set of reference points, the corner points in the eyelid curve to be detected, and the preset curve fitting algorithm , And draw a reference curve corresponding to each set of reference points in the grayscale edge map, where the reference curve corresponding to each set of reference points includes: a reference upper eyelid curve that characterizes the upper eyelid of the person to be tested and The reference lower eyelid curve of the lower eyelid of the person to be inspected; the seventh determining module (not shown in the figure) is configured to determine the first upper eyelid curve for each first upper eyelid curve in the grayscale edge map The sum of the pixel values of the pixel points corresponding to the eyelid curve, wherein the first upper eyelid curve includes: the reference upper eyelid curve corresponding to each set of reference points and the upper eyelid curve to be detected; an eighth determining module (not shown in the figure) Shown), is configured to determine the sum with the largest value from the sum of the pixel values of the pixel points corresponding to each first upper eyelid curve, and determine the first upper eyelid curve corresponding to the sum with the largest value as A target upper eyelid curve that characterizes the upper eyelid of the person to be inspected; a ninth determination module (not shown in the figure), configured to determine the first lower eyelid curve in the grayscale edge map The sum of the pixel values of the pixel points corresponding to the first lower eyelid curve, where the first lower eyelid curve includes: a reference lower eyelid curve corresponding to each set of reference points and the lower eyelid curve to be detected; a tenth determination module ( (Not shown in the figure), is configured to determine the sum with the largest value from the sum of the pixel values of the pixel points corresponding to each first lower eyelid curve, and determine the first lower eyelid corresponding to the sum with the largest value Curve, as the target lower eyelid curve that characterizes the lower eyelid of the person to be inspected; the eleventh determination module (not shown in the figure) is configured to be based on mathematical integration The principle is to integrate the target eyelid curve to determine the third curve length of the target upper eyelid curve and the fourth curve length of the target lower eyelid curve in the target eyelid curve; from the target upper eyelid curve and the target lower curve respectively In the eyelid curve, multiple reference eyelid points are determined; the twelfth determination module (not shown in the figure) is configured to be based on the third curve length of the target upper eyelid curve and the target lower eyelid curve in the target eyelid curve The fourth curve length, the multiple reference eyelid points, and the preset number of equal division points, from the target upper eyelid curve and the target lower eyelid curve in the target eyelid curve, respectively determine the preset number of equal division points minus 1 equal division The upper eyelid point and the preset number of equal division points minus 1 equal division lower eyelid point.

The foregoing device embodiment corresponds to the method embodiment, and has the same technical effect as the method embodiment. For specific description, refer to the method embodiment. The device embodiment is obtained based on the method embodiment, and the specific description can be found in the method embodiment part, which will not be repeated here.

A person of ordinary skill in the art can understand that the drawings are only schematic diagrams of an embodiment, and the modules or processes in the drawings are not necessarily necessary for implementing the present invention.

A person of ordinary skill in the art can understand that the modules in the device in the embodiment may be distributed in the device in the embodiment according to the description of the embodiment, or may be located in one or more devices different from this embodiment with corresponding changes. The modules of the above-mentioned embodiments can be combined into one module or further divided into multiple sub-modules.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: The technical solutions recorded in the foregoing embodiments are modified, or some of the technical features are equivalently replaced; these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

A method for marking key points of eyes, which is characterized in that it includes:

Obtain the face image and the marked eyelid curve corresponding to each face image, wherein the face image is marked with the marked corner points of the eyes and the marked eyelid points of the upper and lower eyelids, and each marked eyelid curve includes: based on the corresponding Annotated upper eyelid curve generated by marking eyelid points and marking eye corner points, which characterizes the upper eyelid, and marked lower eyelid curve which characterizes the lower eyelid;

For each marked eyelid curve, based on the principle of mathematical integration, the marked eyelid curve is integrated to determine the first curve length marked with the upper eyelid curve and the second curve length marked with the lower eyelid curve in the marked eyelid curve; In the marked upper eyelid curve and the marked lower eyelid curve, multiple eyelid points to be used are determined;

For each marked eyelid curve, based on the first curve length of the upper eyelid curve marked on the marked eyelid curve, the second curve length marked on the lower eyelid curve, the multiple eyelid points to be used and the preset number of equal division points, In marking the eyelid curve, marking the upper eyelid curve and marking the lower eyelid curve, respectively determining the preset number of equal division points minus 1 equal division upper eyelid point and the preset number of equal division points minus 1 equal division lower eyelid point;

The marked eye corner points, equal division of upper eyelid points, and equal division of lower eyelid points in each face image are determined as key eye points corresponding to each face image.
The method of claim 1, wherein the marking of the corner of the eye, the equal division of the upper eyelid point and the equal division of the lower eyelid point in each face image are determined as the eye key corresponding to each face image After the step of point, the method further includes:

For each face image, based on the eye key points in the face image, cut out the eye area image from the face image to obtain the eye image marked with the eye key points;

The eye image and its corresponding calibration information are determined as the training data of the eye key point detection model for detecting the eye key points of the eyes in the image, wherein the calibration information includes the eye in the corresponding eye image. The location information of key points.
The method of claim 2, wherein the eye image and its corresponding calibration information are determined as training data of an eye key point detection model for detecting eye key points of the eyes in the image The steps include:

Obtain the real eye opening and closing length corresponding to each face image;

Obtain the measured eye opening and closing length corresponding to each face image, where the measured eye opening and closing length is: based on the length determined by the upper and lower eyelids of the eyes in the target 3D face model corresponding to the face image, the target 3D person The face model is: a face model determined based on the face feature points in the corresponding face image and a preset three-dimensional face model, and the target three-dimensional face model includes: based on the marked eye corner points in the face image, Equally divide the upper eyelid point and the lower eyelid point to construct the upper and lower eyelids of the eye;

For each face image, calculate the ratio of the actual eye opening and closing length corresponding to the face image to the measured eye opening and closing length as the measured deviation corresponding to the face image;

The eye image and its corresponding calibration information are determined as the training data of the eye key point detection model used to detect the upper and lower eyelids of the eye in the image, where the calibration information includes: the corresponding eye image The position information of the marked eye key points, and the actual measured deviation corresponding to the face image corresponding to the eye image.
The method according to any one of claims 1 to 3, characterized in that, before the step of obtaining a face image and an eyelid curve corresponding to each face image, the method further comprises:

The process of labeling the upper and lower eyelids of the eyes of the person's face in each face image, wherein for each face image, the following steps are performed to label the upper and lower eyelids of the eyes of the person's face in each face image:

Obtain and display a first face image, wherein the first face image includes eyes of a person's face, and the first face image is one of the face images;

Receiving an annotation instruction triggered by an annotator for the upper and lower eyelids of the eyes in the first face image, wherein the annotation instruction carries position information of the marked annotation point;

If it is detected that the tagger has triggered the tagging instruction for the specified eyelid in the first face image at least twice, based on the location information of the tag point carried by the tagging instruction triggered by the tagger at least twice, and the preset A curve fitting algorithm to determine a specified eyelid curve that characterizes the specified eyelid, wherein the specified eyelid is: the upper eyelid and the lower eyelid of the eye in the first face image;

The specified eyelid curve is displayed in the first face image, so that the annotator detects whether the annotated point is an eyelid point or an eye corner point on the specified eyelid.
A method for training an eye key point detection model, characterized in that the method includes:

Obtain training data, where the training data includes the upper and lower eyelid equidistant eyelid points marked with the eyes and the eye images marked with the corner points of the eyes, and calibration information corresponding to each eye image, and the calibration information includes: the marking in the corresponding eye image The position information of the divided eyelid points and the marked eye corner points, the divided eyelid points include: based on the marked eyelid curve corresponding to the face image and the principle of mathematical integration, the divided upper eyelid of the eye in the face image is determined The eyelid point and the lower eyelid are equally divided into the lower eyelid points; the marking eyelid curve includes: marking the upper eyelid, which is generated based on the marked corner points of the eyes marked in the corresponding face image and the marked eyelid points of the upper and lower eyelids, and represents the upper eyelid The curve and the marked lower eyelid curve that characterizes the lower eyelid, and each eye image is an image of the area where the eye is located from the corresponding face image;

Input the eye image and the position information of the divided eyelid points and the marked eye corner points included in the calibration information corresponding to each eye image into the initial eye key point detection model to be trained to detect the up and down of the eyes in the image An eye key point detection model that divides the eyelid point and the corner point of the eyelid.
The method of claim 5, wherein the calibration information corresponding to each eye image further comprises: an actual measurement deviation corresponding to the eye image, and the actual measurement deviation is: the actual eye opening and closing length corresponding to the eye image and the actual measurement deviation The ratio of the opening and closing length of the eyes; the actual measured opening and closing length of the eyes is the length determined based on the upper and lower eyelids of the eyes in the target 3D face model corresponding to the face image, and the target 3D face model is based on the corresponding face Face feature points in the image and a face model determined by a preset three-dimensional face model. The target three-dimensional face model includes: annotated eye corner points, equal division of upper eyelid points and equal division of lower eyelid points based on the face image. Eyelid points, the upper and lower eyelids of the constructed eye;

The eye image and the position information of the divided eyelid points and the marked eye corner points included in the calibration information corresponding to each eye image are input into an initial eye key point detection model to be trained to detect the eyes in the image The steps of detecting the key points of the eyelid point and the corner point of the eyelid in the upper and lower eyelids include:

The eye image and the calibration information corresponding to each eye image include the position information of the divided eyelid points and the marked eye corner points and the measured deviation corresponding to the eye image, and input the initial eye key point detection model to train the eye The key point detection model of the eye, wherein the key point detection model of the eye is used to detect the equally divided eyelid points and the corner points of the upper and lower eyelids of the eyes in the image, and detect the actual measured deviation corresponding to the image.
The method according to claim 5, characterized in that, in the said eye image, and the calibration information corresponding to each eye image, including the equally divided eyelid points and the position information of the marked eye corner points, input the initial eye After the key point detection model is trained to obtain the eye key point detection model used to detect the equally divided eyelid points and the corner points of the upper and lower eyelids of the eyes in the image, the method further includes:

Obtaining an image to be inspected, where the image to be inspected includes the eyes of the person to be inspected;

The to-be-detected image is input into the eye key point detection model, and the equally divided eyelid points and eye corner points of the upper and lower eyelids of the to-be-detected person’s eyes in the to-be-detected image are determined.
The method according to claim 7, wherein in said inputting said image to be detected into said eye key point detection model, the equal division of the upper and lower eyelids of the eyes of the person to be detected in said image to be detected is determined After the steps of eyelid point and eye corner point, the method further includes:

Using the Sobel algorithm to perform edge extraction on the image to be detected to obtain a grayscale edge map corresponding to the image to be detected;

Based on the equally divided eyelid points and corner points of the upper and lower eyelids of the eyes in the image to be detected, and a preset curve fitting algorithm, the eyelid curve of the upper and lower eyelids of the person to be detected is determined as the eyelid curve to be detected, and In the grayscale edge map, the eyelid curve to be detected is drawn, wherein the eyelid curve to be detected includes: an upper eyelid curve to be detected representing the upper eyelid of the person to be detected and a lower eyelid curve representing the person to be detected The lower eyelid curve of the eyelid to be tested;

Based on the equally divided eyelid points in the eyelid curve to be detected, multiple sets of reference points are determined in the grayscale edge map, wherein each group of reference points includes points corresponding to the equally divided eyelid points in the eyelid curve to be detected ；

For each set of reference points, based on the set of reference points, the corner points of the eyelid curve to be detected, and the preset curve fitting algorithm, the reference curve corresponding to the set of reference points is determined, and the reference curve is determined in the gray scale. A reference curve corresponding to each set of reference points is drawn in the edge map, where the reference curve corresponding to each set of reference points includes: a reference upper eyelid curve that characterizes the upper eyelid of the person to be inspected and a reference curve that characterizes the lower eyelid of the person to be inspected Refer to the lower eyelid curve;

In the grayscale edge map, for each first upper eyelid curve, the sum of the pixel values of the pixel points corresponding to the first upper eyelid curve is determined, wherein the first upper eyelid curve includes: each set of reference points The corresponding reference upper eyelid curve and the upper eyelid curve to be detected;

From the sum of the pixel values of the pixel points corresponding to each first upper eyelid curve, the sum with the largest value is determined, and the first upper eyelid curve corresponding to the largest sum is determined as a characterizing the person to be tested The target upper eyelid curve of the upper eyelid;

In the grayscale edge map, for each first lower eyelid curve, determine the sum of the pixel values of the pixel points corresponding to the first lower eyelid curve, where the first lower eyelid curve includes: each set of reference points The corresponding reference lower eyelid curve and the lower eyelid curve to be detected;

From the sum of the pixel values of the pixel points corresponding to each first lower eyelid curve, the sum with the largest value is determined, and the first lower eyelid curve corresponding to the largest sum is determined as a characterizing the person to be tested The target lower eyelid curve of the lower eyelid;

Based on the principle of mathematical integration, the target eyelid curve is integrated to determine the third curve length of the target upper eyelid curve and the fourth curve length of the target lower eyelid curve in the target eyelid curve; In the curve of the target lower eyelid, multiple reference eyelid points are determined;

Based on the third curve length of the target upper eyelid curve in the target eyelid curve, the fourth curve length of the target lower eyelid curve, the multiple reference eyelid points and the preset number of equal points, the target upper eyelid curve is selected from the target eyelid curve. In the eyelid curve and the target lower eyelid curve, the preset number of equal division points minus one equal upper eyelid point and the preset number of equal division points minus one equal lower eyelid point are respectively determined.
A marking device for key points of eyes, characterized in that it comprises:

The first obtaining module is configured to obtain a face image and a marked eyelid curve corresponding to each face image, wherein the face image is marked with marked corner points of the eyes and marked eyelid points of the upper and lower eyelids, each Annotating the eyelid curve includes: an upper eyelid curve representing the upper eyelid and a lower eyelid curve representing the lower eyelid generated based on the corresponding eyelid points and corner points of the eye;

The first determining module is configured to, for each marked eyelid curve, integrate the marked eyelid curve based on the principle of mathematical integration, and determine the first curve length of the marked upper eyelid curve and the marked lower eyelid curve in the marked eyelid curve The second curve length; respectively determine a plurality of eyelid points to be used from the marked upper eyelid curve and the marked lower eyelid curve;

The second determining module is configured to, for each marked eyelid curve, based on the first curve length marking the upper eyelid curve in the marked eyelid curve, the second curve length marking the lower eyelid curve, the multiple eyelid points to be used, and Preset the number of equal points, from the marked upper eyelid curve and marked lower eyelid curve from the marked eyelid curve, respectively determine the preset number of equal points minus 1 equal upper eyelid point and the preset number of equal points minus 1 Divide the lower eyelid points equally;

The third determining module is configured to determine the marked eye corner point, the equally divided upper eyelid point, and the equally divided lower eyelid point in each face image as the key eye points corresponding to each face image.
A training device for an eye key point detection model, characterized in that it comprises:

The second obtaining module is configured to obtain training data, where the training data includes the upper and lower eyelid equal eyelid points marked with the eyes and the eye images marked with the corner points of the eyes, and the calibration information corresponding to each eye image, the calibration information Including: the position information of the divided eyelid points and the marked eye corner points marked in the corresponding eye image. The divided eyelid points include: based on the marked eyelid curve corresponding to the face image and the principle of mathematical integration, the determined face image The equal division of the upper eyelid of the eye and the equal division of the lower eyelid point of the lower eyelid; the marked eyelid curve includes: generated based on the marked corner points of the eye marked in the corresponding face image and the marked eyelid points of the upper and lower eyelids , The marked upper eyelid curve representing the upper eyelid and the marked lower eyelid curve representing the lower eyelid, each eye image is an image of the area where the eye is cut out from the corresponding face image;

The input module is configured to input the eye image and the position information of the equally divided eyelid points and the marked eye corner points included in the calibration information corresponding to each eye image into the initial eye key point detection model to be trained for An eye key point detection model that detects the equally divided eyelid points and corner points of the upper and lower eyelids of the eyes in the image.