WO2020237941A1 - Procédé et appareil de détection d'état de personnel sur la base d'informations de caractéristiques de paupière - Google Patents

Procédé et appareil de détection d'état de personnel sur la base d'informations de caractéristiques de paupière Download PDF

Info

Publication number
WO2020237941A1
WO2020237941A1 PCT/CN2019/108074 CN2019108074W WO2020237941A1 WO 2020237941 A1 WO2020237941 A1 WO 2020237941A1 CN 2019108074 W CN2019108074 W CN 2019108074W WO 2020237941 A1 WO2020237941 A1 WO 2020237941A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
eyelid
face
point
position information
Prior art date
Application number
PCT/CN2019/108074
Other languages
English (en)
Chinese (zh)
Inventor
李源
陈荔伟
王晋玮
Original Assignee
初速度（苏州）科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 初速度（苏州）科技有限公司 filed Critical 初速度（苏州）科技有限公司
Publication of WO2020237941A1 publication Critical patent/WO2020237941A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/18—Eye characteristics, e.g. of the iris
    • G06V40/193—Preprocessing; Feature extraction
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/18—Eye characteristics, e.g. of the iris
    • G06V40/197—Matching; Classification

Definitions

  • the present invention relates to the technical field of video surveillance, in particular to a method and device for detecting a person's state based on eyelid feature information.
  • the process is generally: obtaining the face image collected for the target person, and detecting the eye state through pre-training
  • the model detects the face image and detects the open and closed state of the target person’s eyes, that is, whether the target person’s eyes are closed; according to the detection result, it is determined whether the target person is fatigued, and if the target person is detected If the eyes of is in the closed state, it is determined that the target person is fatigued and an alarm is issued.
  • the pre-trained eye state detection model is based on samples marked with closed eyes and open eyes Neural network model from image training.
  • the annotation standards for the closed state and open state of the eyes in the sample image cannot be unified, such as the annotations for half-open eyes Persons are marked as open, and some persons are marked as closed, which causes the pre-trained eye state detection model to blur the detection boundary between the closed state and the open state of the human eyes in the image, and the detection result is not accurate enough.
  • the present invention provides a method and device for detecting a person's status based on eyelid feature information, so as to determine the eyelid feature information of the human eye, and use the eyelid feature information of the human eye to achieve the accuracy of the detection result of the opening and closing state of the human eye Improve the performance, and improve the accuracy of the detection results of the current state of the personnel.
  • the specific technical solutions are as follows:
  • an embodiment of the present invention provides a method for detecting a person's state based on eyelid feature information, including:
  • the face feature points include the eyelid feature points of the upper and lower eyelids of the human eye
  • the target three-dimensional face model includes: The upper and lower eyelids of the human eye constructed by the eyelid feature points;
  • the eyelid space points at the designated positions of the upper and lower eyelids of the human eye in the target three-dimensional face model Based on the three-dimensional position information of the eyelid space points at the designated positions of the upper and lower eyelids of the human eye in the target three-dimensional face model and a preset projection matrix, it is determined that the eyelid space points at the designated position are on the face Projection position information of the projection point in the image;
  • the upper and lower eyelids are determined based on the two-dimensional position information of the eyelid feature points at the position corresponding to the designated position in the face image, the three-dimensional position information of the eyelid spatial points at the designated position, and the projection position information
  • the current state of the target person is determined.
  • the eyelid feature points at the position corresponding to the specified position include: a first center point at the center position of the upper eyelid and a first center point at the center position of the lower eyelid in the face image;
  • the eyelid space points at the designated position include: the third center point at the center position of the upper eyelid and the fourth center point at the center position of the lower eyelid in the target three-dimensional face model;
  • the two-dimensional position information of the eyelid feature points at the positions corresponding to the designated positions of the upper and lower eyelids, the three-dimensional position information of the eyelid spatial points at the designated positions, and the projection position information are used to determine the upper and lower eyelids.
  • the steps of the current opening and closing length between the eyelids include:
  • the distance between the first projection point and the second projection point is determined as the third distance, wherein the first projection A point is a projection point of the third center point in the face image, and the second projection point is a projection point of the fourth center point in the face image;
  • the product of the second distance and the first ratio is determined as the current opening and closing length between the upper and lower eyelids.
  • the step of detecting the two-dimensional position information of the facial feature points from the facial image includes:
  • the two-dimensional position information of facial feature points is detected from the face image, wherein the preset facial feature point detection model is: based on marking each part of the human face A model trained on the first sample image of facial feature points;
  • the eyelid feature points of the upper and lower eyelids of the human eye are detected from the human eye image, wherein the preset eyelid feature point detection model is: The model obtained by training the second sample image of the eyelid feature points of the upper and lower eyelids.
  • the human eye image includes a left eye image and a right eye image
  • the method further includes:
  • the mirror image and the unmirrored image are spliced to obtain a spliced image. If the left-eye image is mirrored, the unmirrored image is the right-eye image; if the right-eye image is mirrored, The image is mirrored, and the image that is not mirrored is the left-eye image;
  • the step of using a preset eyelid feature point detection model to detect the eyelid feature points of the upper and lower eyelids of the human eye from the human eye image includes:
  • Mirror processing is performed on the eyelid feature points of the upper and lower eyelids of the human eye in the mirror image to obtain the eyelid feature points after mirroring, so as to obtain the eyelid feature points of the upper and lower eyelids of the human eye in the human eye image.
  • the method before the step of performing mirror image processing on the left-eye image or the right-eye image to obtain a mirror image, the method further includes:
  • the image to be processed is subjected to normalization processing to obtain a normalized image to be processed, wherein the normalization processing is: making the line of two corner points in the image to be processed parallel to the coordinate axis of the preset image coordinate system, and The processed images are the left-eye image and the right-eye image;
  • the step of performing mirror image processing on the left eye image or the right eye image to obtain a mirror image includes:
  • the step of constructing a target three-dimensional face model corresponding to the target person based on the two-dimensional position information of the facial feature points of the face image and a preset three-dimensional face model includes:
  • a target three-dimensional face model corresponding to the target person is constructed.
  • the step of determining the current state of the target person based on the current opening and closing length includes:
  • the opening and closing length includes the current opening and closing length and the historical opening and closing length
  • the current state of the target person is determined based on the current opening and closing length and the total number of the historical opening and closing lengths and the first result number.
  • the step of determining the current state of the target person based on the current opening and closing length and the total number of the historical opening and closing lengths and the first result number can be achieved by Either way to achieve:
  • the second ratio is not greater than the preset ratio, determining that the current state of the target person is a non-fatigue state
  • the difference is greater than the preset difference, it is determined that the current state of the target person is a non-fatigue state
  • the difference is not greater than the preset difference, it is determined that the current state of the target person is a fatigue state.
  • the method further includes:
  • an alarm message is generated and sent.
  • an embodiment of the present invention provides a device for detecting a person's state based on eyelid feature information, including:
  • An obtaining module configured to obtain a face image containing the face of the target person
  • the detection module is configured to detect two-dimensional position information of facial feature points from the face image, where the facial feature points include the eyelid feature points of the upper and lower eyelids of the human eye;
  • the construction module is configured to construct a target three-dimensional face model corresponding to the target person based on the two-dimensional position information of the facial feature points of the face image and a preset three-dimensional face model, wherein the target three-dimensional
  • the face model includes: upper and lower eyelids of the human eye constructed based on the eyelid feature points;
  • the first determining module is configured to determine, based on the three-dimensional position information of the eyelid space points at the designated positions of the upper and lower eyelids of the human eye in the target three-dimensional face model and a preset projection matrix, the Projection position information of the projection point of the eyelid space point in the face image;
  • the second determining module is configured to be based on the two-dimensional position information of the eyelid feature points at the position corresponding to the specified position in the face image, the three-dimensional position information of the eyelid space point at the specified position, and the projection Position information, determining the current opening and closing length between the upper and lower eyelids;
  • the third determining module is configured to determine the current state of the target person based on the current opening and closing length.
  • the eyelid feature points at the position corresponding to the specified position include: a first center point at the center position of the upper eyelid and a first center point at the center position of the lower eyelid in the face image;
  • the eyelid space points at the designated position include: the third center point at the center position of the upper eyelid and the fourth center point at the center position of the lower eyelid in the target three-dimensional face model;
  • the second determining module is specifically configured as
  • the distance between the first projection point and the second projection point is determined as the third distance, wherein the first projection A point is a projection point of the third center point in the face image, and the second projection point is a projection point of the fourth center point in the face image;
  • the product of the second distance and the first ratio is determined as the current opening and closing length between the upper and lower eyelids.
  • the detection module includes:
  • the first detection unit is configured to detect two-dimensional position information of facial feature points from the face image based on a preset facial feature point detection model, wherein the preset facial feature point detection model is: A model trained based on the first sample image of facial feature points labeled with various parts of the human face;
  • the determining and intercepting unit is configured to determine and intercept the area where the eyes of the target person are located from the face image based on the two-dimensional position information of the facial feature points, as the eye image;
  • the second detection unit is configured to use a preset eyelid feature point detection model to detect the eyelid feature points of the upper and lower eyelids of the human eye from the human eye image, wherein the preset eyelid feature point detection
  • the model is a model trained based on the second sample image of the eyelid feature points of the upper and lower eyelids of a human eye.
  • the human eye image includes a left eye image and a right eye image
  • the detection module further includes:
  • the mirroring unit is configured to, before detecting the eyelid feature points of the upper and lower eyelids of the human eye from the human eye image using the preset eyelid feature point detection model, Perform mirror image processing on the right eye image to obtain a mirror image;
  • the splicing unit is configured to splice the mirror image and the unmirrored image to obtain a spliced image, wherein, if the left-eye image is mirrored, the unmirrored image is the right-eye image; If mirroring is performed on the right-eye image, the image that has not been mirrored is the left-eye image;
  • the second detection unit is specifically configured as:
  • Mirror processing is performed on the eyelid feature points of the upper and lower eyelids of the human eye in the mirror image to obtain the eyelid feature points after mirroring, so as to obtain the eyelid feature points of the upper and lower eyelids of the human eye in the human eye image.
  • the first detection module further includes:
  • the normalization unit is configured to perform normalization processing on the image to be processed before the mirror image processing is performed on the left-eye image or the right-eye image to obtain the mirror image, to obtain a normalized image to be processed, wherein the normalization
  • the processing is: making the line of two corner points in the image to be processed parallel to the coordinate axis of the preset image coordinate system, and the image to be processed is the left eye image and the right eye image;
  • the mirroring unit is specifically configured to perform mirroring processing on the image to be processed after being normalized to obtain a mirrored image.
  • the building module is specifically configured as:
  • a target three-dimensional face model corresponding to the target person is constructed.
  • the third determining module includes:
  • An obtaining unit configured to obtain the historical opening and closing length of the human eye of the target person determined within a preset time period
  • the comparison obtaining unit is configured to compare each opening and closing length with a preset length threshold to obtain a comparison result, wherein the opening and closing length includes the current opening and closing length and the historical opening and closing length;
  • the statistical unit is configured to calculate the number of first results that characterize the comparison results in which the opening and closing length is less than the preset length threshold;
  • the determining unit is configured to determine the current state of the target person based on the current opening and closing length and the total number of the historical opening and closing lengths and the first result number.
  • the determining unit is specifically configured as:
  • the second ratio is not greater than the preset ratio, determining that the current state of the target person is a non-fatigue state
  • the difference is greater than the preset difference, it is determined that the current state of the target person is a non-fatigue state
  • the difference is not greater than the preset difference, it is determined that the current state of the target person is a fatigue state.
  • the device further includes:
  • Generating a sending module configured to determine the current state of the target person based on the current opening and closing length
  • an alarm message is generated and sent.
  • the method and device for detecting human status based on eyelid feature information provided by the embodiments of the present invention can obtain a face image containing the face of a target person; the two-dimensional feature points of the face are detected from the face image.
  • the face feature points include the eyelid feature points of the upper and lower eyelids of the human eye; based on the two-dimensional location information of the face feature points of the face image and the preset three-dimensional face model, construct the target three-dimensional target corresponding to the target person
  • the face model where the target three-dimensional face model includes: the upper and lower eyelids of the human eye constructed based on eyelid feature points; the three-dimensional position information and predictions of the eyelid space points at the designated positions of the upper and lower eyelids of the human eye in the target three-dimensional face model Set the projection matrix to determine the projection position information of the projection point of the eyelid space point at the specified position in the face image; based on the two-dimensional position information at the position corresponding to the specified position in the face image, the eyelid space at the specified position
  • the three-dimensional position information of the point and the projection position information determine the current opening and closing length between the upper and lower eyelids; based on the current opening and closing length, the current state of the target person is determined.
  • the two-dimensional position information of the facial feature points including the eyelid feature points of the upper and lower eyelids of the human eye can be detected from the facial image, and the two-dimensional position information of the facial feature points including the face of the target person can be detected Points and preset three-dimensional face models to construct a target three-dimensional face model corresponding to the target person’s eye and upper and lower eyelids, that is, to construct the spatial information of the target person’s eye; and then to determine the target three-dimensional
  • the projection position information of the eyelid space points at the designated positions of the upper and lower eyelids in the face model in the face image is based on the spatial information of the human eye, that is, the three-dimensional position information of the eyelid space points at the designated positions of the upper and lower eyelids of the target person , And the two-dimensional position information of the eyelid feature points at the position corresponding to the specified position in the face image and the corresponding projection position information to determine the opening and closing length of the upper and lower eyelids,
  • combining the three-dimensional information and two-dimensional information of the upper and lower eyelids of the human eye can differentiate the three-dimensional information of the upper and lower eyelids of the human eye and the error of any information in the two-dimensional information, which can better improve the current opening and closing to a certain extent.
  • the accuracy of the length thereby improving the accuracy of the detection result of the current state of the personnel.
  • the detection model blurs the detection boundary between the closed state and the open state of the human eye in the image, which leads to the occurrence of insufficient detection results.
  • the feature information of the eyelid of the human eye can be determined, and the feature information of the eyelid can be used to improve the accuracy of the detection result of the open and closed state of the human eye, and improve the accuracy of the detection result of the current state of the person.
  • any product or method of the present invention does not necessarily need to achieve all the advantages described above at the same time.
  • the two-dimensional position information of the face feature points including the eyelid feature points of the upper and lower eyelids of the human eye can be detected from the face image, and based on the face feature points and presets in the face image containing the face of the target person
  • the 3D face model of the target person is constructed to construct the target 3D face model including the upper and lower eyelids of the target person’s eyes, that is, the spatial information of the target person’s eyes is constructed; and then the target 3D face model is determined
  • the projection position information of the projection point of the eyelid space point at the designated position of the upper and lower eyelids in the face image is based on the spatial information of the human eye, that is, the three-dimensional position information of the eyelid space point at the designated position of the upper and lower eyelids of the target person, and the face
  • the two-dimensional position information of the eyelid feature points at the position corresponding to the specified position in the image and the corresponding projection position information determine the opening and closing length of the upper and lower eyelids, which can be
  • the spatial distance between the upper and lower eyelids of the human eye with higher accuracy can be more accurately determined based on the spatial distance between the upper and lower eyelids of the human eye with higher accuracy
  • the current status of the target person Among them, combining the three-dimensional information and two-dimensional information of the upper and lower eyelids of the human eye can differentiate the three-dimensional information of the upper and lower eyelids of the human eye and the error of any information in the two-dimensional information, which can better improve the current opening and closing to a certain extent.
  • the accuracy of the length thereby improving the accuracy of the detection result of the current state of the personnel.
  • the detection model blurs the detection boundary between the closed state and the open state of the human eye in the image, which leads to the occurrence of insufficient detection results.
  • the feature information of the eyelid of the human eye can be determined, and the feature information of the eyelid can be used to improve the accuracy of the detection result of the open and closed state of the human eye, and improve the accuracy of the detection result of the current state of the person.
  • the two-dimensional position information of facial feature points is detected from the face image, and based on the two-dimensional position information of facial feature points, the human eyes in the face are cut out from the face image
  • the area where the human eye is located that is, the human eye image
  • use the preset eyelid feature point detection model to detect the eyelid feature points of the upper and lower eyelids of the human eye from the human eye image, which can improve the accuracy of the detected eyelid feature points, and then
  • the accuracy of the upper and lower eyelids of the human eye in the target three-dimensional face model constructed based on the eyelid feature points can be improved, so as to better improve the accuracy of the detection result of the target person's state.
  • the left-eye image and the right-eye image are corrected to obtain the corrected left-eye image and the corrected right-eye image, and then the corrected left-eye image or the corrected right-eye image is subjected to subsequent processing, so that To a certain extent, the detection burden of the preset eyelid feature point detection model can be reduced, and the detection result of eyelid feature points can be improved to a certain extent.
  • FIG. 1 is a schematic flowchart of a method for detecting a person's state based on eyelid feature information according to an embodiment of the present invention
  • FIG. 2 is a schematic flowchart of determining the current opening and closing length between the upper and lower eyelids of a human eye according to an embodiment of the present invention
  • Fig. 3 is a schematic structural diagram of a device for detecting a person's state based on eyelid feature information provided by an embodiment of the present invention.
  • the present invention provides a method and device for detecting a person's status based on eyelid feature information, so as to determine the eyelid feature information of a human eye, and use the eyelid feature information to improve the accuracy of the detection result of the open and close state of the human eye, and improve The accuracy of the detection results of the current status of the personnel.
  • the embodiments of the present invention will be described in detail below.
  • FIG. 1 is a schematic flowchart of a method for detecting a person's state based on eyelid feature information according to an embodiment of the present invention. The method can include the following steps:
  • the method for detecting a person's state based on the feature information of the eyelid can be applied to any type of electronic device based on computing capability.
  • the electronic device may be an image capture device, and the image capture device may perform the subsequent person based on eyelid feature information provided by the embodiment of the present invention after obtaining the face image containing the target person's face collected by itself. Status detection process.
  • the electronic device may be a non-image acquisition device.
  • the electronic device may communicate with one or more image acquisition devices.
  • the electronic device can obtain the face image collected by each image acquisition device in communication connection, and then implement the embodiment of the present invention for the face image collected by each image acquisition device In the provided human state detection process based on eyelid feature information, different image acquisition devices can shoot different people to obtain face images.
  • the image acquisition device can be set in the vehicle, and correspondingly, the target person is the driver of the vehicle, and the image acquisition device can photograph the driver’s face in the vehicle in real time, and collect information containing the driver.
  • the image of the face is then sent to the electronic device, and the electronic device obtains an image containing the driver’s face.
  • the obtained image may only contain the driver’s face.
  • the electronic device can directly use the obtained image as a human Face image; in addition, the obtained image may include the driver’s face while also including the vehicle’s seat or the driver’s body.
  • the electronic device After the electronic device obtains the image captured by the image capture device, it may be based on the prediction Suppose the face detection algorithm detects the image of the area where the face is located from the image, and cuts out the image of the area where the face is located from the image to obtain a face image containing the driver's face.
  • the preset face detection algorithm can be: eigenface method (Eigenface) and face detection algorithm based on neural network model
  • face detection algorithm based on neural network model can be: FasterR-CNN(FasterRegion-Convolutional Neural Networks, fast area-convolutional neural network) detection algorithm, this is all possible.
  • the embodiment of the present invention does not limit the specific type of the preset face detection algorithm.
  • the vehicle may be a private car, a truck, a bus, etc.
  • the embodiment of the present invention does not limit the vehicle type of the vehicle.
  • the image capture device can also monitor the passing vehicles on the road in real time.
  • the target person can be the target driver, and the electronic device can obtain multiple image capture devices to take pictures of the target driver.
  • the captured image containing the face of the target driver is one case, after the electronic device obtains the image containing the face of the target driver collected by the image acquisition device, it directly uses the image as the face image, and then executes the subsequent human state detection process of the eyelid feature information.
  • the electronic device After the electronic device obtains the image containing the face of the target driver collected by the image capture device, it detects the area where the face of the target driver is located based on a preset face detection algorithm from the image, and then The image of the area where the face of the target driver is located is cut out from the image, and a face image containing only the face of the target driver is obtained.
  • the image capture device can monitor indoor household personnel in real time.
  • the target person can be the target household person
  • the electronic device can obtain the target captured by the image capture device for shooting the target household person. Face image of the face of the householder.
  • the facial feature points include the eyelid feature points of the upper and lower eyelids of the human eye, and may also include facial feature points that characterize the locations of various parts of the face of the target person.
  • the various parts of the face may include the nose, lips, eyebrows, human eyes, jaw, cheeks, ears and other parts.
  • the facial feature points of each part of the face can respectively include: feature points in the face that characterize the position of the nose, such as nose wings, nose bridge, and tip of the nose; can also include feature points that characterize the position of the lips, such as lips
  • the feature points of the edge of the lip line can also include the feature points that characterize the location of the eyebrows, such as the feature points of the edge of the eyebrow; it can also include the feature points that characterize the location of the human eye, such as the corners of the eyes, the eye sockets Feature points and pupil feature points, etc.; can also include feature points that characterize the location of the mandible, such as feature points on the contour of the mandible, that is, feature points on the chin contour, etc.; can also include features that characterize the location of the ear Each feature point, such as each feature point on each contour of the ear.
  • any face feature point detection algorithm can be used to detect the face feature points of the target person's face from the face image, and determine the two-dimensional position information of the face feature points from the face image.
  • the aforementioned facial feature point detection algorithms can be model-based ASM (Active Shape Model) and AAM (Active Appearnce Model) algorithms, based on cascaded shape regression CPR (Cascaded pose regression) algorithms, and deep learning-based algorithms, etc.
  • the embodiment of the present invention can apply any algorithm in the related technology that can detect the facial feature points from the face of the face image to realize the detection of the facial feature points on the face of the target person.
  • S103 Based on the two-dimensional position information of the facial feature points of the facial image and the preset three-dimensional face model, construct a target three-dimensional face model corresponding to the target person.
  • the target three-dimensional face model includes: the upper and lower eyelids of a human eye constructed based on eyelid feature points.
  • a preset three-dimensional face model is prestored locally or in a storage device connected to the electronic device.
  • the electronic device determines the facial feature points of the face in the face image, it can be based on the preset three-dimensional face model And the two-dimensional position information of the facial feature points to construct a target three-dimensional face model corresponding to the target person.
  • 3DMM 3D Morphable Models, three-dimensional deformable models
  • 3DMM 3D Morphable Models, three-dimensional deformable models
  • the S103 may include:
  • a target three-dimensional face model corresponding to the target person is constructed.
  • the electronic device may receive a user selection instruction, where the user selection instruction carries a preset face position of the spatial point to be selected, and the electronic device may be based on the preset face position carried by the user selection instruction , From the preset three-dimensional face model, determine the spatial point at the preset face position as the spatial point to be processed.
  • the electronic device may pre-store the preset face position, and then the electronic device may read the preset face position from the corresponding storage location, and then, from the preset three-dimensional face model , The spatial point at the preset face position is determined as the spatial point to be processed.
  • the preset face position may be set based on the position of the face feature points of the face contained in the first sample image.
  • the preset three-dimensional face model can be expressed by the following formula (1):
  • S represents the preset three-dimensional face model
  • a id represents the shape information of the human face
  • a exp represents the expression information of the human face
  • ⁇ id represents the weight of the shape information of the human face, which can be called the shape weight
  • ⁇ exp The weight of the expression information representing the human face can be called the expression weight.
  • the electronic device can draw the represented three-dimensional face model based on the above formula (1), and the three-dimensional face model is composed of point clouds.
  • the electronic device may determine the spatial point at the preset face position from the drawn three-dimensional face model as the spatial point to be processed, and further, may continue to obtain spatial position information of the spatial point to be processed, that is, three-dimensional position information.
  • the electronic device After the electronic device determines the spatial point to be processed, it can project each spatial point to be processed into the face image based on the preset weak perspective projection matrix, that is, use the weak perspective projection matrix and the space of each spatial point to be processed
  • the position information determines the projection position information of the projection point of each spatial point to be processed in the face image. Based on the projection position information of the projection point of each spatial point to be processed and the two-dimensional position information of the facial feature point corresponding to each spatial point to be processed, a target three-dimensional face model corresponding to the target person is constructed.
  • the above process of constructing a target three-dimensional face model corresponding to the target person based on the projection position information of the projection point of each spatial point to be processed and the two-dimensional position information of the facial feature point corresponding to each spatial point to be processed may be Yes: Based on the projection position information of the projection point of each spatial point to be processed and the two-dimensional position information of the facial feature point corresponding to each spatial point to be processed, determine each spatial point to be processed and its corresponding facial feature point Based on the principle of least squares and the distance error of each spatial point to be processed and its corresponding facial feature point, the objective function is constructed. When the solution minimizes the function value of the objective function, a solution of the corresponding unknown quantity in the objective function is obtained based on the solution to obtain a target three-dimensional face model corresponding to the target person.
  • the preset weak perspective projection matrix can be expressed by the following formula (2):
  • s i2d represents the projection position information of the projection point of the i-th spatial point to be processed, where i can be an integer in [1, n], where n represents the number of spatial points to be processed, f represents the scale factor, and R ( ⁇ , ⁇ , ⁇ ) represents a 3*3 rotation matrix, ⁇ represents the rotation angle of the preset three-dimensional face model under the horizontal axis in the preset space rectangular coordinate system, and ⁇ represents the preset three-dimensional face The rotation angle of the model under the vertical axis in the preset space rectangular coordinate system, ⁇ represents the rotation angle of the preset three-dimensional face model under the vertical axis in the preset space rectangular coordinate system, and t 3d represents the translation vector; S i represents the spatial position information of the i-th spatial point to be processed, and the rotation matrix and translation vector are used to: convert the preset three-dimensional face model from the preset spatial rectangular coordinate system where it is located to the image acquisition device In the device coordinate system.
  • the objective function can be expressed by the following formula (3):
  • s i2dt represents the two-dimensional position information of the face feature point corresponding to the i-th space point to be processed
  • ⁇ represents the modulus of the vector
  • the vector represents: the face feature point corresponding to the i-th space point to be processed The distance error between the two-dimensional position information of and the projection position information of the projection point of the i-th spatial point to be processed.
  • the specific values of f, R( ⁇ , ⁇ , ⁇ ), t 3d , ⁇ id , and ⁇ exp can be continuously adjusted through an iterative method to minimize P or satisfy preset constraints.
  • the preset constraint condition may be that P is not greater than the preset distance error threshold.
  • S104 Based on the three-dimensional position information of the eyelid space points at the designated positions of the upper and lower eyelids of the human eyes in the target three-dimensional face model and the preset projection matrix, determine the projection point of the eyelid space points at the designated position in the face image Projection location information.
  • the preset projection matrix is: the projection matrix of the image acquisition device that collects the face image of the target person's face, and can project each eyelid space point in the upper and lower eyelids of the human eye in the target three-dimensional face model to In the face image, the projection position information of the projection points of each eyelid space point in the upper and lower eyelids of the human eye in the target three-dimensional face model on the face image is obtained.
  • the preset projection matrix may be the aforementioned preset weak perspective projection matrix.
  • the eyelid space points at the specified position may include: all eyelid space points of the upper and lower eyelids of the human eye in the target three-dimensional face model; or may include: the eyelid space points at the center of the upper eyelid in the target three-dimensional face model , As the first center point, and the eyelid space point at the center of the lower eyelid, as the second center point; or may include: the eyelid space point at any target bisecting point of the upper eyelid in the target three-dimensional face model And the eyelid space point at the target bisecting point of the lower eyelid, and so on.
  • the electronic device After the electronic device determines the target three-dimensional face model, it can determine the three-dimensional position information of each eyelid space point in the upper and lower eyelid of the human eye in the target three-dimensional face model, and then determine the eyelid at the designated position of the upper and lower eyelid of the human eye
  • the three-dimensional position information of the spatial point based on the three-dimensional position information of the eyelid spatial point at the designated position of the upper and lower eyelids of the human eye in the target three-dimensional face model and the preset projection matrix, the eyelid at the designated position in the target three-dimensional face model is determined
  • the projection position information of the projection point of the spatial point in the face image The projection position information of the projection point of the spatial point in the face image.
  • the preset projection matrix determines the eyelid feature points in the center of the upper eyelid of the human eye and the eyelid feature points in the center of the lower eyelid in the target three-dimensional face model, respectively, the projection points in the face image
  • the projection location information It is used for the subsequent calculation of the current opening and closing length between the upper and lower eyelid points of the human eye to reduce the calculation amount to a certain extent.
  • S105 Determine the current opening and closing length between the upper and lower eyelids based on the two-dimensional position information of the eyelid feature points at the position corresponding to the designated position in the face image, the three-dimensional position information of the eyelid space point at the designated position, and the projection position information.
  • the two-dimensional position information of the eyelid feature points at the position corresponding to the specified position in the face image is used, that is, the two-dimensional position information of the eyelid feature points at the positions corresponding to the specified positions of the upper and lower eyelids of the human eye in the face image , Determine the two-dimensional distance between the upper and lower eyelids of the human eye in the face image as the first two-dimensional distance.
  • the projection position information that is, the projection position information of the projection point of the eyelid space point at the specified position in the target three-dimensional face model in the face image
  • the two-dimensional distance between the projection points corresponding to the eyelid space point at the specified position is used as the second two-dimensional distance.
  • the three-dimensional distance between the upper and lower eyelids of the human eye in the target three-dimensional face model can be determined.
  • the current opening and closing length between the upper and lower eyelids of the human eye is determined. To a certain extent, the accuracy of the determined opening and closing length between the upper and lower eyelids of the human eye can be further improved.
  • the accuracy of the determined opening and closing length between the upper and lower eyelids of the human eye can be further improved.
  • the distance between the eyelid space points of the human eye in the target three-dimensional face model and the distance between the eyelid feature points of the human eye in the two-dimensional face image jointly determine the opening between the upper and lower eyelids of the human eye.
  • the closed length can differentiate the three-dimensional information of the human eye’s upper and lower eyelids and the error of any information in the two-dimensional information. To a certain extent, it can better improve the accuracy of the current opening and closing length, thereby improving the current status of the personnel. The accuracy of the test results.
  • the eyelid feature points at the position corresponding to the designated position include: a first center point at the center position of the upper eyelid and a second center point at the center position of the lower eyelid in the face image;
  • the eyelid space points of the position include: the third center point at the center of the upper eyelid and the fourth center point at the center of the lower eyelid in the target three-dimensional face model;
  • S105 may include:
  • S201 Based on the two-dimensional position information of the first center point and the two-dimensional position information of the second center point, determine the distance between the first center point and the second center point as the first distance.
  • S202 Based on the three-dimensional position information of the third center point and the three-dimensional position information of the fourth center point, determine the distance between the third center point and the fourth center point as the second distance.
  • S203 Determine the distance between the first projection point and the second projection point as the third distance based on the projection position information of the first projection point and the projection position information of the second projection point.
  • the first projection point is the projection point of the third center point in the face image
  • the second projection point is the projection point of the fourth center point in the face image
  • S204 Calculate the ratio of the first distance to the third distance as the first ratio.
  • S205 Determine the product of the second distance and the first ratio as the current opening and closing length between the upper and lower eyelids.
  • the eyelid points at the center of the upper and lower eyelids can better represent the open and closed state of the human eye to a certain extent, that is, the distance between the eyelid points at the center of the upper and lower eyelids can better represent the upper and lower eyelids the distance between.
  • the first center is determined based on the two-dimensional position information of the first center point at the center of the upper eyelid of the face image and the two-dimensional position information of the second center point at the center of the lower eyelid.
  • the distance between the point and the second center point is taken as the first distance.
  • the distance between the first center point and the second center point can be expressed as: Among them, d represents the first distance, (u 1 , v 1 ) represents the two position information of the first center point, and (u 2 , v 2 ) represents the two-dimensional position information of the second center point.
  • the distance between the third center point and the fourth center point can be expressed as: Among them, (x 1 , y 1 , z 1 ) represents the three-dimensional position information of the third center point, and (x 2 , y 2 , z 2 ) represents the three-dimensional position information of the fourth center point.
  • the ratio of the second distance and the third distance is calculated as the first ratio; and the product of the first ratio and the first distance is used as the current opening and closing length between the upper and lower eyelids of the human eye.
  • S106 Determine the current status of the target person based on the current opening and closing length.
  • the opening and closing state of the human eye of a person can represent the fatigue degree of the person to a certain extent, and the opening and closing state of the human eye can be measured by the opening and closing length between the upper and lower eyelids of the human eye Logo.
  • the distance between the upper and lower eyelids of a person’s eyes will be relatively small, while in a non-fatigue state, the distance between the upper and lower eyelids of the human eye will be relatively large.
  • the current status of the target person is determined.
  • the current state may include fatigue state and non-fatigue state.
  • the eyelid feature information of the upper and lower eyelids of any person of the target person such as the eyelid feature information of the upper and lower eyelids of the left eye or the eyelid feature information of the upper and lower eyelids of the right eye, to determine the current opening between the upper and lower eyelids Close the length, and then determine the current state of the target person.
  • the eyelid feature information may include the three-dimensional position information of the upper and lower eyelids of the human eye in the target three-dimensional face model, the two-dimensional position information of the upper and lower eyelids of the human eye in the face image, and the information of the human eye in the target three-dimensional face model in the face image.
  • the projection position information of the projection point of the upper and lower eyelids may include the three-dimensional position information of the upper and lower eyelids of the human eye in the target three-dimensional face model, the two-dimensional position information of the upper and lower eyelids of the human eye in the face image, and the information of the human eye in the target three-dimensional face model in the face image
  • it may be: for the eyelid feature information of the upper and lower eyelids of the two human eyes of the target person, such as the eyelid feature information of the upper and lower eyelids of the left eye and the right eye, determine the current opening and closing length between the upper and lower eyelids, and then , To determine the current status of the target personnel.
  • it can be used to determine the opening and closing length between the upper and lower eyelids of each person's eye according to the eyelid feature information of each eye of the target person, and then calculating the average value of the opening and closing lengths between the upper and lower eyelids of the two eyes , As the current opening and closing length between the upper and lower eyelids to determine the current state of the target person.
  • the two-dimensional position information of the face feature points including the eyelid feature points of the upper and lower eyelids of the human eye can be detected from the face image, and based on the facial features in the face image containing the face of the target person Points and preset three-dimensional face models to construct a target three-dimensional face model corresponding to the target person’s upper and lower eyelids, that is, to construct the spatial information of the target person’s eyes; and then to determine the target three-dimensional
  • the projection position information of the eyelid space points at the designated positions of the upper and lower eyelids in the face model in the face image is based on the spatial information of the human eye, that is, the three-dimensional position information of the eyelid space points at the designated positions of the upper and lower eyelids of the target person , And the two-dimensional position information of the eyelid feature points at the position corresponding to the specified position in the face image and the corresponding projection position information to determine the opening and closing length of the upper and lower eyelids, which can realize the combination of
  • combining the three-dimensional information and two-dimensional information of the upper and lower eyelids of the human eye can differentiate the three-dimensional information of the upper and lower eyelids of the human eye and the error of any information in the two-dimensional information, which can better improve the current opening and closing to a certain extent.
  • the accuracy of the length thereby improving the accuracy of the detection result of the current state of the personnel.
  • the detection model blurs the detection boundary between the closed state and the open state of the human eye in the image, which leads to the occurrence of inaccurate detection results.
  • the feature information of the eyelid of the human eye can be determined, and the feature information of the eyelid can be used to improve the accuracy of the detection result of the open and closed state of the human eye, and improve the accuracy of the detection result of the current state of the person.
  • the S102 may include:
  • the two-dimensional position information of the facial feature points is detected from the face image.
  • the preset facial feature point detection model is: based on the facial feature points of each part of the face The model trained on the first sample image.
  • the area where the eyes of the target person are located is determined and intercepted from the face image, as the eye image.
  • the eyelid feature points of the upper and lower eyelids of the human eye are detected from the human eye image.
  • the preset eyelid feature point detection model is: based on marking the eyelid feature points of the upper and lower eyelids of the human eye The model trained on the second sample image.
  • the preset facial feature point detection model is: a neural network model obtained by training based on a first sample image that annotates facial feature points of various parts of a human face.
  • the embodiment of the present invention may also include a process of training to obtain a preset facial feature point detection model.
  • the electronic device may first obtain an initial facial feature point detection model, and the initial facial feature point detection model Feature extraction layer and feature classification layer; obtain a first sample image, each first sample image includes a human face; obtain calibration information corresponding to each first sample image, where the calibration information includes the first sample image
  • the calibration position information of the facial feature points of the human face contained in, the calibration feature points include: facial feature points that characterize the locations of various parts of the face.
  • the electronic device inputs each first sample image into the feature extraction layer of the initial facial feature point detection model to obtain the image feature of each first sample image; input the image feature of each first sample image into the initial The feature classification layer of the facial feature point detection model obtains the current position information of the facial feature point in each first sample image; compares the current position information of the facial feature point in each first sample image with its corresponding calibration position information Matching; if the matching is successful, the initial facial feature point detection model is determined to converge, and the initial facial feature point detection model including the feature extraction layer and the feature classification layer is obtained, that is, the preset facial feature point detection model is obtained; if the matching is not successful , Adjust the parameters of the feature extraction layer and the feature classification layer, and return to execute the step of inputting each first sample image into the feature extraction layer of the initial facial feature point detection model to obtain the image features of each first sample image; Until the matching is successful, an initial facial feature point detection model including a feature extraction layer and a feature classification layer is obtained, which is a preset facial feature point detection model
  • the above process of matching the current position information of the facial feature points in each first sample image with the corresponding calibration position information may be: using a preset loss function to calculate the current position information of each facial feature point Determine whether the first loss value is less than the first preset loss threshold value between the corresponding calibration position information; if it is determined that the first loss value is less than the first preset loss threshold value, it is determined that the matching is successful. It can be determined that the initial facial feature point detection model converges, that is, it is determined that the training of the initial facial feature point detection model is completed, and the preset facial feature point detection model is obtained; if it is determined that the first loss value is not less than the first preset Loss threshold is determined to be unsuccessful.
  • each first sample image has a corresponding relationship with the current position information of the facial feature point
  • each first sample image has a corresponding relationship with the calibration position information of the facial feature point in the calibration information
  • the facial feature point There is a correspondence between the current position information and the calibration position information of the facial feature points in the calibration information.
  • the electronic device can detect the obtained face image based on the preset facial feature point detection model, and detect the facial feature points of the face in the face image.
  • the area where the human eye of the target person is located is determined and cut out from the face image, as the human eye image.
  • it can be based on the two-dimensional position information of each feature point representing the location of the human eye in the facial feature point, determining the smallest rectangular area containing the human eye of the target person, taking the rectangular area as the area of the human eye, and Cut out to get the human eye image. It may be that the images of the area where the target person is located are respectively intercepted for the two eyes of the target person to obtain the human eye image.
  • the eyelid feature points of the upper and lower eyelids of the human eye are detected from the human eye image.
  • the aforementioned preset eyelid feature point detection model is a neural network model obtained by training based on the second sample image marked with the eyelid feature points of the upper and lower eyelids of a human eye.
  • the training process of the preset eyelid feature point detection model refer to the training process of the aforementioned preset facial feature point detection model.
  • the first sample image that is different from the preset facial feature point detection model is an image that annotates the eyelid feature points of the upper and lower eyelids of a human eye
  • the calibration information corresponding to the second sample image includes the The calibration position information of the eyelid feature points of the upper and lower eyelids of the human eye.
  • the eyelid feature points of the upper and lower eyelids of the human eye marked by the second sample image may be eyelid feature points calibrated manually or through a specific calibration procedure.
  • the aforementioned preset facial feature point detection model and the preset eyelid feature point detection model may be a model combined with functions.
  • the third sample image required by the training to obtain the combined model is an image containing a human face
  • the calibration information corresponding to the third sample image contains the calibration position information of the eyelid feature points of the upper and lower eyelids of the human eye and facial features Calibration location information of the point.
  • the human eye image includes a left eye image and a right eye image
  • the method may further include:
  • the S102 may include:
  • Mirror processing is performed on the eyelid feature points of the upper and lower eyelids of the human eye in the mirror image to obtain the eyelid feature points after mirroring to obtain the eyelid feature points of the upper and lower eyelids of the human eye in the human eye image.
  • the human eye image includes: the image containing the left eye of the target person, called the left eye image; and the image containing the right eye of the target person, called the right eye image.
  • the left-eye image or the right-eye image may be mirrored to obtain a mirror image.
  • the mirror image and the unmirrored image are spliced to obtain the spliced image; the spliced image is input into the preset eyelid feature point detection model to use the preset eyelid feature point detection model to detect the mirror image from the spliced image
  • the preset eyelid feature point detection model can simultaneously detect the mirror image and the unmirrored image, which can shorten the detection time required to detect the eyelid feature points of the target person by using the preset eyelid feature point detection model.
  • the image that is not mirrored is the left-eye image; if the left-eye image is mirrored, the image that is not mirrored is the right-eye image.
  • Mirroring the left-eye image or the right-eye image can make the left-eye image mirror the right-eye image corresponding to the left-eye image, or make the right-eye image mirror the left-eye image corresponding to the right-eye image, to a certain extent Reduce the complexity of using the preset eyelid feature point detection model to detect the eyelid feature points of the target person.
  • the required second sample image may include the left eye image of the sample person and the left eye image obtained by mirroring the right eye image of the sample person. Or include the right eye image of the sample person's right eye image and the right eye image of the sample person's left eye image. If the second sample image required by the above-mentioned preset eyelid feature point detection model is obtained by training, it contains the left eye image of the sample person and the left eye image obtained by mirroring the right eye image of the sample person, and the subsequent, in the detection process , Mirror the right eye image of the target person.
  • the second sample image required by the above-mentioned preset eyelid feature point detection model contains the right eye image of the sample person and the right eye image obtained by mirroring the left eye image of the sample person, and the subsequent detection process , Mirror the left eye image of the target person.
  • the right eye image or left eye image of the sample person is mirrored. To a certain extent, it can also increase the training to obtain the above-mentioned preset eyelid feature point detection model.
  • the number of second sample images is obtained by training.
  • the above process of splicing the mirror image and the unmirrored image to obtain the spliced image can be: splicing the mirror image and the unmirrored image in the spatial dimension or the channel dimension, wherein the splicing of the spatial dimension can be To: stitch the mirror image and the unmirrored image left and right or up and down.
  • Left and right splicing can be: the right side of the mirrored image is spliced with the left side of the image that is not mirrored, and the left side of the mirrored image is spliced with the right side of the image that is not mirrored.
  • Top and bottom splicing may be: the upper side of the mirror image is spliced with the lower side of the image that is not mirrored, and the lower side of the mirror image is spliced with the upper side of the image that is not mirrored.
  • the method may further include:
  • the image to be processed is subjected to normalization processing to obtain the image to be processed after the correction, wherein the normalization processing is: making the line of the two corner points of the image to be processed parallel to the coordinate axis of the preset image coordinate system, and the image to be processed is the left Eye image and right eye image;
  • the step of performing mirror image processing on the left eye image or the right eye image to obtain a mirror image may include:
  • the head of the target person may be tilted.
  • the left-eye image and the right-eye image can be corrected first, that is, the connection between the two corner points of the left-eye image is parallel to the horizontal axis of the preset image coordinate system, and the two corner points of the right-eye image are connected.
  • the line is parallel to the horizontal axis of the preset image coordinate system; or, making the line between the two corner points of the left eye image parallel to the vertical axis of the preset image coordinate system, and making the line between the two corner points of the right eye image Parallel to the longitudinal axis of the preset image coordinate system, this is all possible.
  • mirror image processing may be performed on the left-eye image after normalization or the right-eye image after normalization to obtain a mirror image.
  • the preset image coordinate system may be the image coordinate system of the image acquisition device.
  • the S106 may include:
  • the opening and closing length includes the current opening and closing length and the historical opening and closing length
  • the current state of the target person is determined.
  • the time dimension information that is, the historical opening and closing length of the human eye, can be combined to determine the current state of the target person.
  • the electronic device can obtain the face image containing the face of the target person collected by the image acquisition device at the current moment.
  • the preset time length may be a time length preset by the user, or a time length independently set by the electronic device, both of which are possible.
  • the historical opening and closing length of the eyes of the target person determined within the preset time period may include: the historical opening and closing length of the eyes of the target person determined within the preset time period ahead of the current moment, that is, The historical opening and closing length of the human eye of the target person determined within the latest preset time period at the current moment.
  • the electronic device can store the historical opening and closing length of the human eye of the target person locally or in the storage device connected to it. After calculating the current opening and closing length of the human eye, the electronic device can download the corresponding storage location Obtain the historical opening and closing length of the target person’s eye.
  • the historical opening and closing length of the human eye of the target person is determined based on the face image before the face image collected by the image acquisition device for shooting the target person. The process of determining the historical opening and closing length of the target person's eyes is similar to the process of determining the current opening and closing length of the target person's eyes, and will not be repeated here.
  • the opening and closing length of the human eye can be determined more accurately, that is, the physical length of the opening and closing of the human eye. Furthermore, combined with the time dimension, the target can be monitored more flexibly and accurately. The current status of the person.
  • the electronic device can obtain the preset length threshold set in advance, and compare each opening and closing length, that is, the current opening and closing length and the historical opening and closing length with the preset length threshold respectively, to compare each opening and closing length with the preset length
  • the size of the threshold, the comparison result is obtained; further, the number of comparison results indicating that the opening and closing length is less than the preset length threshold is obtained by statistics, as the first result number; the subsequent, based on the total number of the current opening and closing length and the historical opening and closing length
  • the first result quantity determines the current status of the target person.
  • the step of determining the current state of the target person based on the current opening and closing length and the total number of historical opening and closing lengths and the first result number can be achieved by any of the following implementation methods Realization:
  • the second ratio is greater than the preset ratio, it is determined that the current state of the target person is a fatigue state
  • the second ratio is not greater than the preset ratio, it is determined that the current state of the target person is non-fatigue
  • the difference is not greater than the preset difference, it is determined that the current state of the target person is a fatigue state.
  • the preset ratio and the preset difference may be set by the staff according to experience values.
  • the first number can be directly compared with the preset number, and if the number of first results is greater than The preset number determines that the current state of the target person is a fatigue state; if the first result number is not greater than the preset number, the current state of the target person is determined to be a non-fatigue state.
  • the historical opening and closing length of the human eye of the target person determined within the preset time is 99; that is, the current opening and closing length and the historical opening and closing length are 100. If the statistics show that the opening and closing length is less than the preset length The first result number of the threshold comparison result is 80. At this time, it can be determined that the current state of the target person is the fatigue state.
  • the method may further include:
  • warning information can be generated , To remind the user that the target person is in a state of fatigue, so that the user can take corresponding measures for this situation to reduce the occurrence of car accidents caused by fatigue driving to a certain extent.
  • the driver can also be prompted to enter the automatic driving mode, or the driving mode control signal can be sent to control the vehicle to automatically enter the automatic driving mode, so as to reduce the fatigue caused by driving to a certain extent Of the car accident.
  • a household control signal of the household equipment can be generated and sent.
  • the household control signal can be to control the playback volume of the TV to decrease or turn off the TV; it can be: control The current setting temperature of the air conditioner is within the preset temperature range, and so on.
  • the embodiment of the present invention provides a person state detection device based on eyelid feature information, as shown in FIG. 3, which may include:
  • the obtaining module 310 is configured to obtain a face image containing the face of the target person
  • the detection module 320 is configured to detect the two-dimensional position information of the face feature points from the face image, where the face feature points include the eyelid feature points of the upper and lower eyelids of the human eye;
  • the construction module 330 is configured to construct a target three-dimensional face model corresponding to the target person based on the two-dimensional position information of the facial feature points of the face image and a preset three-dimensional face model, wherein the target The three-dimensional face model includes: upper and lower eyelids of the human eye constructed based on the eyelid feature points;
  • the first determining module 340 is configured to determine the designated position based on the three-dimensional position information of the eyelid space points at the designated positions of the upper and lower eyelids of the human eye in the target three-dimensional face model and a preset projection matrix Projection position information of the projection point of the eyelid space point in the face image;
  • the second determining module 350 is configured to be based on the two-dimensional position information of the eyelid feature points at the position corresponding to the specified position in the face image, the three-dimensional position information of the eyelid space point at the specified position, and the Projection position information to determine the current opening and closing length between the upper and lower eyelids;
  • the third determining module 360 is configured to determine the current state of the target person based on the current opening and closing length.
  • the two-dimensional position information of the face feature points including the eyelid feature points of the upper and lower eyelids of the human eye can be detected from the face image, and based on the facial features in the face image containing the face of the target person Points and preset three-dimensional face models to construct a target three-dimensional face model corresponding to the target person’s upper and lower eyelids, that is, to construct the spatial information of the target person’s eyes; and then to determine the target three-dimensional
  • the projection position information of the eyelid space points at the designated positions of the upper and lower eyelids in the face model in the face image is based on the spatial information of the human eye, that is, the three-dimensional position information of the eyelid space points at the designated positions of the upper and lower eyelids of the target person , And the two-dimensional position information of the eyelid feature points at the position corresponding to the specified position in the face image and the corresponding projection position information to determine the opening and closing length of the upper and lower eyelids, which can realize the combination of
  • combining the three-dimensional information and two-dimensional information of the upper and lower eyelids of the human eye can differentiate the three-dimensional information of the upper and lower eyelids of the human eye and the error of any information in the two-dimensional information, which can better improve the current opening and closing to a certain extent.
  • the accuracy of the length thereby improving the accuracy of the detection result of the current state of the personnel.
  • the detection model blurs the detection boundary between the closed state and the open state of the human eye in the image, which leads to the occurrence of inaccurate detection results.
  • the feature information of the eyelid of the human eye can be determined, and the feature information of the eyelid can be used to improve the accuracy of the detection result of the open and closed state of the human eye, and improve the accuracy of the detection result of the current state of the person.
  • the eyelid feature points at the position corresponding to the designated position include: the first center point at the center position of the upper eyelid and the first center point at the center position of the lower eyelid in the face image A center point;
  • the eyelid space point at the specified position includes: the third center point at the center of the upper eyelid and the fourth center point at the center of the lower eyelid in the target three-dimensional face model;
  • the second determining module 350 is specifically configured to
  • the distance between the first projection point and the second projection point is determined as the third distance, wherein the first projection A point is a projection point of the third center point in the face image, and the second projection point is a projection point of the fourth center point in the face image;
  • the product of the second distance and the first ratio is determined as the current opening and closing length between the upper and lower eyelids.
  • the detection module 320 includes:
  • the first detection unit (not shown in the figure) is configured to detect two-dimensional position information of facial feature points from the face image based on a preset facial feature point detection model, wherein the preset The facial feature point detection model is: a model trained based on the first sample image of the facial feature points of each part of the human face labeled;
  • the determining and intercepting unit (not shown in the figure) is configured to determine and intercept the area where the eyes of the target person are located from the face image based on the two-dimensional position information of the facial feature points, as the eyes image;
  • the second detection unit (not shown in the figure) is configured to use a preset eyelid feature point detection model to detect the eyelid feature points of the upper and lower eyelids of the human eye from the human eye image, wherein the The preset eyelid feature point detection model is: a model trained based on the second sample image marked with the eyelid feature points of the upper and lower eyelids of a human eye.
  • the human eye image includes a left eye image and a right eye image
  • the detection module 320 further includes:
  • the mirroring unit (not shown in the figure) is configured to detect the eyelid feature points of the upper and lower eyelids of the human eye from the human eye image using the preset eyelid feature point detection model. Performing mirror image processing on the left eye image or the right eye image to obtain a mirror image;
  • the splicing unit (not shown in the figure) is configured to splice the mirror image and the unmirrored image to obtain a spliced image.
  • the unmirrored image is Is the right-eye image
  • the image that has not been mirrored is the left-eye image
  • the second detection unit (not shown in the figure) is specifically configured as:
  • Mirror processing is performed on the eyelid feature points of the upper and lower eyelids of the human eye in the mirror image to obtain the eyelid feature points after mirroring, so as to obtain the eyelid feature points of the upper and lower eyelids of the human eye in the human eye image.
  • the first detection module 320 further includes:
  • the normalization unit (not shown in the figure) is configured to perform normalization processing on the image to be processed before the mirror image processing is performed on the left-eye image or the right-eye image to obtain the mirror image, to obtain the pending processing Image, wherein the correction processing is: making the line of two corner points in the image to be processed parallel to the coordinate axis of the preset image coordinate system, and the image to be processed is the left eye image and the right eye image;
  • the mirroring unit (not shown in the figure) is specifically configured to perform mirroring processing on the image to be processed after being normalized to obtain a mirrored image.
  • the construction module 330 is specifically configured as:
  • a target three-dimensional face model corresponding to the target person is constructed.
  • the third determining module 360 includes:
  • An obtaining unit (not shown in the figure), configured to obtain the historical opening and closing length of the human eye of the target person determined within a preset time period;
  • the comparison obtaining unit (not shown in the figure) is configured to compare each opening and closing length with a preset length threshold to obtain a comparison result, wherein the opening and closing length includes the current opening and closing length and the history Opening and closing length
  • a statistical unit (not shown in the figure), configured to obtain a number of first results that represent comparison results whose opening and closing lengths are less than the preset length threshold;
  • the determining unit (not shown in the figure) is configured to determine the current state of the target person based on the current opening and closing length and the total number of the historical opening and closing lengths and the first result number.
  • the determining unit is specifically configured as:
  • the second ratio is not greater than the preset ratio, determining that the current state of the target person is a non-fatigue state
  • the difference is greater than the preset difference, it is determined that the current state of the target person is a non-fatigue state
  • the difference is not greater than the preset difference, it is determined that the current state of the target person is a fatigue state.
  • the device further includes:
  • a generating and sending module (not shown in the figure), configured to determine that the current state of the target person is a fatigue state after the current state of the target person is determined based on the current opening and closing length, Generate and send alarm information.
  • the foregoing device embodiment corresponds to the method embodiment, and has the same technical effect as the method embodiment.
  • the device embodiment is obtained based on the method embodiment, and the specific description can be found in the method embodiment part, which will not be repeated here.
  • modules in the device in the embodiment may be distributed in the device in the embodiment according to the description of the embodiment, or may be located in one or more devices different from this embodiment with corresponding changes.
  • the modules of the above-mentioned embodiments can be combined into one module, or further divided into multiple sub-modules.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Ophthalmology & Optometry (AREA)
  • Health & Medical Sciences (AREA)
  • Multimedia (AREA)
  • Human Computer Interaction (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Health & Medical Sciences (AREA)
  • Evolutionary Computation (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Artificial Intelligence (AREA)
  • General Engineering & Computer Science (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Evolutionary Biology (AREA)
  • Data Mining & Analysis (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)

Abstract

Des modes de réalisation de la présente invention concernent un procédé et un appareil de détection d'état de personnel basés sur des informations de caractéristiques de paupière. Le procédé consiste à : obtenir une image de visage contenant le visage d'une personne cible ; détecter des informations de position bidimensionnelle d'un point de caractéristique de visage à partir de l'image de visage ; construire un modèle de visage tridimensionnel cible sur la base d'un modèle de visage tridimensionnel prédéfini ; déterminer des informations de position de projection d'un point d'espace de paupière à une position spécifiée dans l'image de visage sur la base d'une matrice de projection ; déterminer la longueur d'ouverture et de fermeture en cours entre les paupières supérieure et inférieure sur la base des informations de position bidimensionnelle du point de caractéristique de paupière à une position correspondant à la position spécifiée dans l'image de visage, des informations de position tridimensionnelle du point d'espace de paupière à la position spécifiée, et des informations de position de projection ; et déterminer l'état en cours de la personne cible sur la base de la longueur d'ouverture et de fermeture en cours, de façon à déterminer les informations de caractéristique de paupière des yeux humains. La précision du résultat de détection de l'état d'ouverture et de fermeture des yeux humains est améliorée à l'aide des informations de caractéristiques de paupière, et la précision du résultat de détection de l'état en cours de la personne est améliorée.
PCT/CN2019/108074 2019-05-29 2019-09-26 Procédé et appareil de détection d'état de personnel sur la base d'informations de caractéristiques de paupière WO2020237941A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201910457032.7A CN112016347B (zh) 2019-05-29 2019-05-29 一种基于眼睑特征信息的人员状态检测方法及装置
CN201910457032.7 2019-05-29

Publications (1)

Publication Number Publication Date
WO2020237941A1 true WO2020237941A1 (fr) 2020-12-03

Family

ID=73500863

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/108074 WO2020237941A1 (fr) 2019-05-29 2019-09-26 Procédé et appareil de détection d'état de personnel sur la base d'informations de caractéristiques de paupière

Country Status (2)

Country Link
CN (1) CN112016347B (fr)
WO (1) WO2020237941A1 (fr)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113256586A (zh) * 2021-05-28 2021-08-13 深圳数联天下智能科技有限公司 人脸图像的模糊判断方法、装置、设备和介质
CN116115878A (zh) * 2023-02-22 2023-05-16 王华虎 一种情景式营造犯困氛围的智能化快速催眠方法及装置

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2008069171A1 (fr) * 2006-12-04 2008-06-12 Aisin Seiki Kabushiki Kaisha Dispositif de détection de l'œil, procédé de détection de l'œil et programme
CN102098955A (zh) * 2008-05-28 2011-06-15 有效调控运输公司 检测微入睡事件的方法和设备
CN104881955A (zh) * 2015-06-16 2015-09-02 华中科技大学 一种驾驶员疲劳驾驶检测方法及系统
US20160171321A1 (en) * 2014-12-15 2016-06-16 Aisin Seiki Kabushiki Kaisha Determination apparatus and determination method
CN108460345A (zh) * 2018-02-08 2018-08-28 电子科技大学 一种基于人脸关键点定位的面部疲劳检测方法

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104091150B (zh) * 2014-06-26 2019-02-26 浙江捷尚视觉科技股份有限公司 一种基于回归的人眼状态判断方法

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2008069171A1 (fr) * 2006-12-04 2008-06-12 Aisin Seiki Kabushiki Kaisha Dispositif de détection de l'œil, procédé de détection de l'œil et programme
CN102098955A (zh) * 2008-05-28 2011-06-15 有效调控运输公司 检测微入睡事件的方法和设备
US20160171321A1 (en) * 2014-12-15 2016-06-16 Aisin Seiki Kabushiki Kaisha Determination apparatus and determination method
CN104881955A (zh) * 2015-06-16 2015-09-02 华中科技大学 一种驾驶员疲劳驾驶检测方法及系统
CN108460345A (zh) * 2018-02-08 2018-08-28 电子科技大学 一种基于人脸关键点定位的面部疲劳检测方法

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
ZHANG, BO: "Driver Drowsiness Detection in Complicated Conditions Based on Computer Vision", DOCTORAL DISSERTATION, 1 June 2015 (2015-06-01), pages 1 - 152, XP009524799 *

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113256586A (zh) * 2021-05-28 2021-08-13 深圳数联天下智能科技有限公司 人脸图像的模糊判断方法、装置、设备和介质
CN116115878A (zh) * 2023-02-22 2023-05-16 王华虎 一种情景式营造犯困氛围的智能化快速催眠方法及装置

Also Published As

Publication number Publication date
CN112016347B (zh) 2022-06-24
CN112016347A (zh) 2020-12-01

Similar Documents

Publication Publication Date Title
CN107506693B (zh) 畸变人脸图像校正方法、装置、计算机设备和存储介质
CN109690553A (zh) 执行眼睛注视跟踪的系统和方法
US11043011B2 (en) Image processing method, apparatus, terminal, and storage medium for fusing images of two objects
CN104616438B (zh) 一种用于疲劳驾驶检测的打哈欠动作检测方法
US11068704B2 (en) Head pose and distraction estimation
CN107358152B (zh) 一种活体识别方法和系统
TW201643761A (zh) 活體檢測方法和裝置以及身分認證方法和裝置
WO2020237940A1 (fr) Procédé et dispositif de détection de fatigue basés sur une identification d'état d'œil humain
CN110826521A (zh) 驾驶员疲劳状态识别方法、系统、电子设备和存储介质
JP2003015816A (ja) ステレオカメラを使用した顔・視線認識装置
US20200242336A1 (en) Method for selecting images in video of faces in the wild
WO2020237941A1 (fr) Procédé et appareil de détection d'état de personnel sur la base d'informations de caractéristiques de paupière
WO2022199545A1 (fr) Procédé et appareil de traitement d'image, dispositif informatique et support de stockage
WO2020252969A1 (fr) Procédé et appareil de marquage de point clé d'œil, et procédé et appareil d'entraînement pour modèle de détection de point clé d'œil
WO2021197466A1 (fr) Procédé, appareil et dispositif de détection de globe oculaire, et support de stockage
WO2024051345A1 (fr) Procédé et appareil d'identification de ligne de visée d'un conducteur, véhicule et support de stockage
JP2021531601A (ja) ニューラルネットワーク訓練、視線検出方法及び装置並びに電子機器
JPWO2018078857A1 (ja) 視線推定装置、視線推定方法及びプログラム記録媒体
US10964046B2 (en) Information processing apparatus and non-transitory computer readable medium storing information processing program for estimating face orientation by using an omni-directional camera
CN105095885A (zh) 一种人眼状态的检测方法和检测装置
WO2023071190A1 (fr) Procédé et appareil de détection de vivacité, dispositif informatique et support de stockage
CN114894337A (zh) 一种用于室外人脸识别测温方法及装置
CN112036217B (zh) 一种基于嘴部信息的人员状态检测方法及装置
CN114022514A (zh) 一种融合头部姿态和眼球跟踪的实时视线推断方法
CN112084820B (zh) 一种基于头部信息的人员状态检测方法及装置

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19930748

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19930748

Country of ref document: EP

Kind code of ref document: A1

122 Ep: pct application non-entry in european phase

Ref document number: 19930748

Country of ref document: EP

Kind code of ref document: A1