JP2011128966A - Face feature point detection device and sleepiness detection device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a face feature point detection device and a sleepiness detection device facilitated in practical use capable of grasping an expression with high accuracy. <P>SOLUTION: By imaging the face image of a driver 3 by an imaging device 10, a sleepiness detection device 20 first generates a 2D reference face and a 2D template in an individual information model for use to obtain feature points on the face of the driver 3 (S1). Next, feature points at an awakening state are obtained (S2), and a sleepiness level is estimated by detecting the feature points at timing to detect the sleepiness (S3). According to the sleepiness level, a signal is output to operate a nap prevention device (S4). The acquisition of the feature points is performed by imaging the face image of the driver 3 by the imaging device 10, and by fitting the image concerned by deforming the 2D reference face and the 2D template so as to obtain two-dimensional coordinates of the feature points from the imaged image. Next, three-dimensional coordinates are obtained by be fir to a 3D reference face which is a common model. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a face feature point detection device that acquires a position of a feature point on a face from a photographed image of a subject, and a drowsiness detection device using the face feature point detection device.

  2. Description of the Related Art Conventionally, a technique has been proposed in which a subject's face is photographed and changes in the positions of feature points on the face and the orientation of the face are captured from the photographed image. For example, a face image of a target person (vehicle driver) is photographed, and a 2D (two-dimensional) model having a form and texture information consisting of a large number of feature points, and a 3D (three-dimensional) model consisting only of a form An apparatus has been proposed that detects that a subject is dozing by fitting these models to captured images and capturing changes in the subject's face (see Patent Document 1).

JP 2009-45418 A

  In the device disclosed in Patent Document 1 described above, as a 2D model and a 3D model, a model customized for each individual (individual model) created based on various individual image data, or created based on image data of a plurality of persons One of the general-purpose models (common model) to be used is used.

  When a personal model is used, facial expressions can be detected with very high accuracy for an individual who has provided image data, but the detection accuracy is low when a person other than the individual is targeted. Therefore, in order to detect facial expressions with high accuracy, it is necessary to create a model for each person whose facial expressions are to be measured, and the scenes that can be put to practical use are limited.

  On the other hand, when a common model is used, it is not necessary to create a detailed model for each subject, and a model created in advance can be used. However, since the common model is a model based on the average face, although it can detect better for many subjects compared to the case of using the individual model, As the difference from the correct face increases, the detection accuracy deteriorates. For many subjects, for example, it is impossible to capture a change in facial expression with high accuracy enough to determine drowsiness, for example.

  The present invention has been made in view of the above-described problems, and an object of the present invention is to provide a facial feature point detection device and a drowsiness detection device that can be easily put into practical use and can capture facial expressions with high accuracy. is there.

  The invention according to claim 1, which has been made to solve the above-described problem, is a face feature point detection that detects a movement of three-dimensional coordinates of a plurality of predetermined feature points on a face from a photographed image. Device.

  This face feature point detection device has means for acquiring a photographed image showing the face of the subject. This means may acquire a captured image from an imaging device such as a camera, or may acquire a captured image stored in a storage device or the like. Note that the above-described feature points are measurement points that can specify positions on the face, such as the corners of the eyes, the corners of the eyes, the corners of the mouth, and the eyebrows. By knowing the movement of this feature point, it is possible to detect changes in facial expressions, that is, changes in physical condition such as sleepiness and emotions.

  Further, the face feature point detection device extracts two-dimensional coordinates of a plurality of predetermined feature points on the face from a reference image that is an acquired photographed image, and generates a reference 2D face model composed of the two-dimensional coordinates. create. The reference 2D face model here may be a data group having only two-dimensional coordinate information of feature points, or may further have information of a graphic in which feature points are arranged and visualized. Then, a modified 2D face model is created by moving the two-dimensional coordinates of a plurality of feature points from the reference 2D face model.

  The face feature point detection apparatus creates a deformed image obtained by deforming the reference image according to the created deformed 2D face model. Since the reference 2D model is based on the reference image, the feature points naturally correspond to the feature points of the reference image. The reference image is deformed so as to maintain the correspondence between the image and the feature point in accordance with the change of the reference 2D model to the deformed 2D model, and a deformed image is created.

  The face feature point detection apparatus performs fitting on the acquired determination image, which is a captured image, and the above-described deformed image by adjusting the movement of the two-dimensional coordinates of the feature points in the deformed image. Then, the two-dimensional coordinates of the plurality of feature points in the deformed 2D face model corresponding to the deformed image when the determination image and the deformed image are matched are defined 2D coordinates that are the two-dimensional coordinates of the plurality of feature points in the determination image. decide. The term “adaptation” as used herein means a state in which the determination image and the deformed image match with a predetermined accuracy.

In this way, the face feature point detection apparatus detects the two-dimensional coordinates of the feature points on the face in the acquired determination image through the processing so far.
Next, the face feature point detection apparatus creates a modified 3D face model composed of three-dimensional coordinates obtained by moving predetermined three-dimensional coordinates corresponding to the plurality of feature points described above. The modified 3D face model here may be a data group having only the three-dimensional coordinate information of the feature points, or may further have graphic information obtained by arranging and visualizing the feature points.

Subsequently, converted 2D coordinates are created by converting the three-dimensional coordinates of the deformed 3D face model into two-dimensional coordinates using a conversion formula. The conversion formula here is, for example, a perspective projection matrix.
Then, the determined 2D coordinates that have already been determined and the created converted 2D coordinates are fitted by moving the three-dimensional coordinates to the deformed 3D face model or adjusting the conversion formula described above. In addition, by adjusting the conversion formula, the face direction of the deformed 3D face model can be changed and converted into two-dimensional coordinates.

  Then, the three-dimensional coordinates of the plurality of feature points in the modified 3D face model corresponding to the transformed 2D coordinates that match the confirmed 2D coordinates are determined as the three-dimensional coordinates of the plurality of feature points in the determination image. The term “adaptation” here means a state in which the confirmed 2D coordinate and the converted 2D coordinate coincide with each other with a predetermined accuracy.

That is, the face feature point detection apparatus detects the three-dimensional coordinates of the feature points on the face in the acquired determination image through the processing so far.
In the face feature point detection apparatus configured in this way, a two-dimensional model of two-dimensional coordinates and a reference image is created using information acquired from a subject at a predetermined timing, so that a plurality of objects The detection accuracy of two-dimensional coordinates (determined 2D coordinates) is higher than when a common model created from a person is used. Accordingly, the detection accuracy of the face feature point detection apparatus as a whole is increased, so that changes in the facial expression of the subject can be captured with high accuracy.

  In addition, since three-dimensional models such as predetermined three-dimensional coordinates and conversion formulas are not customized to the person to be photographed, various images of the person to be photographed are used to create a three-dimensional model. Thus, there is no need to create a model, and the person to be photographed only needs to photograph the front face before detecting the facial feature points. Therefore, it can be used in various situations without imposing a burden on the subject.

Thus, the face feature point detection apparatus described above is easy to put into practical use and can capture facial expressions with high accuracy.
By the way, regarding the movement of the three-dimensional coordinates when creating the deformed 3D face model, the moving direction is determined for each feature point, and the deformed 3D face model is created by adjusting only the movement amount. If there are individual differences in the shape and movement direction of the face of the person being photographed, the coordinates of the feature points can only represent the amount of movement as the amount of change. Can be easily compared.

  Note that the above-described reference image is a captured image that is captured to create a model (reference 2D face model) that serves as a reference for determining feature points, and the above-described determination image is the actual movement of feature points. It is the picked-up image used as the determination object which determines this. Therefore, by acquiring the reference image in advance and acquiring the determination image when determining the movement of the feature point, the operation of the feature point can be determined in real time.

  According to a second aspect of the present invention, in the facial feature point detection device according to the first aspect, when moving the three-dimensional coordinates for fitting, a rule for determining a predetermined positional relationship between feature points is satisfied. The feature point is that the three-dimensional coordinates of the feature points are moved.

  In the face feature point detection apparatus configured as described above, the feature point does not move to a position outside the rule, so that the deformed 3D face model does not have an unnatural shape. Therefore, it is possible to eliminate useless fitting processing using a deformed 3D model having an unnatural shape, and it is possible to suppress a detection result with low accuracy falling into a local solution.

In addition, as said rule, it is possible to prescribe | regulate the positional relationship between feature points like the face feature point detection apparatus of Claims 3-6.
The face feature point detection apparatus according to claim 3 is characterized in that, as the above-described rule, the feature point indicating the eye position is determined to be located below the feature point indicating the eyebrow position. . The feature point indicating the eye position may be a feature point indicating the eye itself, or may be a feature point indicating the eyelid, the corner of the eye, the top of the eye, or the like.

  The face feature point detection apparatus according to claim 4 is characterized in that, as the above-described rule, the feature point indicating the position of the mouth is positioned below the feature point indicating the position of the nose. And

  Further, in the face feature point detection apparatus according to claim 5, the rule described above is set such that the feature point indicating the position of the upper eyelid is positioned above the feature point indicating the position of the lower eyelid. It is characterized by that.

  Further, in the face feature point detecting device according to claim 6, as the rule described above, the feature point indicating the position of the left mouth corner is determined to be located to the left of the feature point indicating the right mouth corner. It is characterized by that.

In addition to the rules described above, for example, the distance between the eyebrows, the distance between the mouth corners, the distance between the lower lip and the chin can be considered.
The invention described in claim 7 is a drowsiness detection apparatus comprising the face feature point detection apparatus according to any one of claims 1 to 6 and means for determining drowsiness.

  The means for determining drowsiness determines three-dimensional coordinates (A) of a plurality of feature points determined by the face feature point detection device based on a determination image shot when the subject is awake, and drowsiness. The degree of drowsiness is determined by comparison with the three-dimensional coordinates (B) of a plurality of feature points determined by the face feature point detection device based on the determination image captured at the power timing.

  In the drowsiness detection device configured as described above, drowsiness is detected based on the feature points detected by the face feature point detection device, so that drowsiness can be detected with high accuracy and the subject is not burdened. It can be used in various situations.

  The invention according to claim 8 is the drowsiness detection device according to claim 7, wherein the means for determining drowsiness compares the three-dimensional coordinates (A) with the three-dimensional coordinates (B), and compares a plurality of feature points. The amount of movement is calculated, and the amount of sleepiness is determined by comparing the amount of movement with a plurality of data shown below.

  The data described above is the amount of movement from the three-dimensional coordinates of a plurality of feature points in a state in which the subject is awake to the three-dimensional coordinates of the plurality of feature points in a state in which the subject is photographed. The degree of sleepiness of the subject to be photographed in a state of drowsiness is associated. Note that the number of persons to be photographed, which is a target for acquiring data, may be one, but preferably a plurality of persons.

  With the drowsiness detection device configured as described above, since the degree of drowsiness is determined by comparing the movement amount stored in the data with the detected movement amount, the individual difference in the position of the feature point in the awake state is determined. Since there is no need to consider, the degree of sleepiness can be determined with high accuracy.

  The invention described in claim 9 is the drowsiness detection device according to claim 7 or 8, further comprising means for measuring the heart rate of the subject. And when the subject is awake, the average value of the heart rate measured by the heart rate measuring means is in a predetermined range, and based on the determination image taken at that time The three-dimensional coordinates of the feature points are detected, and the coordinates are used as the feature points at awakening.

  The drowsiness detection device configured as described above can determine the arousal state based on the heart rate. Therefore, the determination image captured when not in the awake state is not processed as the determination image at the time of awakening, so that drowsiness can be detected with high accuracy.

Side view of the dozing prevention system Block diagram showing the configuration of the drowsiness detection device The flowchart which shows the processing procedure of the dozing prevention processing Figure showing feature points on the face Flow chart showing processing procedure of personal adaptation processing Figure showing an individual expression model A flowchart showing the procedure of the reference value acquisition process Flow chart showing processing procedure of sleepiness estimation processing Illustration of sleepiness estimation algorithm Flow chart showing processing procedure of facial expression feature amount detection processing The flowchart which shows the process sequence of 3D facial expression correction processing The figure which shows the state whose distance between eyelids is 0 or less The figure which shows the state where the distance between upper and lower eyelids is larger than the distance between the corner of the eye The figure which shows the state in which the distance of an eyebrow head and an upper eyelid is larger than 1/2 of the distance between temples The figure which shows the state in which the feature point of an eyebrow is located below the upper eyelid The figure which shows the state where the upper lip is located below the lower lip The figure which shows the state where the distance of a chin position and a lower lip is smaller than 1/2 of the distance of an eye head and an eye corner The figure which shows the state where the distance of a nose and an upper lip is 0 or less The figure which shows the state whose distance between mouth corners is 0 or less The figure which shows the state where the eyebrows actually moved The figure which shows the state which the eyebrows moved when the common three-dimensional model of a present Example was used The figure which shows the state which the eyebrows moved when the common three-dimensional model of a present Example was used

Embodiments of the present invention will be described below with reference to the drawings. However, the embodiment described below is merely an example, and it goes without saying that the present invention can be implemented in various forms other than the following examples.
[Example]
(1) Overall Configuration FIG. 1 shows a dozing prevention system 1 according to this embodiment. The dozing prevention system 1 is a system that is mounted on a vehicle 2 and detects drowsiness of a driver 3 (person to be photographed) to prevent dozing, and includes an imaging device 10, a drowsiness detection device 20, and an accident caused by dozing operation. It comprises an alarm device 30 that performs an operation for prevention, a neck air conditioner 40, a seat belt vibration device 50, and a brake control device 60.

  Among these, the photographing device 10 continuously photographs the face of the driver 3 of the vehicle 2, and is arranged toward the driver 3 on the dashboard 4 of the vehicle 2. An image showing the front face of the camera is taken at a predetermined time interval (for example, 1/30 s).

  The drowsiness detection device 20 is a plurality of feature points on the face of the driver 3 based on the image photographed by the photographing device 10 (measurement points that can specify the position on the face, and hereinafter the feature points are these measurement points. And drowsiness level is determined in 6 levels of levels 0-5. The configuration of the drowsiness detection device 20 is shown in FIG. The drowsiness detection device 20 includes a control unit 21, an image capture board 22, a hard disk drive 23, and an in-vehicle LAN communication unit 24.

  The control unit 21 is a well-known microcomputer including a CPU, ROM, RAM, EEPROM, etc. (not shown), and executes various processes based on programs stored in the ROM.

  The image capture board 22 acquires the image data of the face of the driver 3 photographed by the photographing device 10 (hereinafter, also simply referred to as a photographed image) from the photographing device 10 and temporarily stores it. The latest image is provided to the control unit 21 in accordance with the command.

The HDD 23 has a storage area for storing data such as a personal information model described later and learning data used for sleepiness level estimation.
Based on the output from the control unit 21, the in-vehicle LAN communication unit 24 performs an operation for preventing a doze on the alarm device 30, the neck air conditioner 40, the seat belt vibration device 50, and the brake control device 60. Outputs a control signal.

  The alarm device 30 includes a display and a speaker (not shown). This alarm device 30 displays a message that prompts the driver 3 to take a nap of driving according to the degree of drowsiness determined by the control unit 21 (for example, when it is determined that the drowsiness level is 1 or 2, “break early”. ”Let's take it”, “Caution” when sleepiness level 3 is determined, “Stop driving” when sleepiness levels 4 and 5 are determined, and the like. The displayed content is output by voice through a speaker.

  Moreover, the neck air conditioner 40 is arrange | positioned at the headrest in the driver | operator's 3 sheet | seat, and when it determines with the drowsiness level 1-5 by the control part 21, at the driver | operator's 3 neck. Blow.

  The seat belt vibration device 50 is provided in the seat belt retracting mechanism, and vibrates the seat belt when the control unit 21 determines that the drowsiness level is 3 to 5.

The brake control device 60 is a device that automatically operates the brake. When the control unit 21 determines that the drowsiness level is 5, the brake is operated to forcibly stop the vehicle running or sequentially decelerate the vehicle.
(2) Processing by Drowsiness Detection Device 20 Hereinafter, various processing executed by the control unit 21 included in the drowsiness detection device 20 will be described.
(2.1) Dozing Prevention Process The processing procedure of the dozing prevention process will be described based on the flowchart shown in FIG. This process is continuously executed while the vehicle 2 is activated and the dozing prevention system 1 is activated.

  In this processing, first, personal adaptation processing is performed (step 1; hereinafter, steps are simply referred to as S). Here, a part of the personal information model used when acquiring the feature points on the face of the driver 3 is created based on the photographed image of the driver 3. Details of the personal adaptation process and the personal information model will be described later.

  Next, a reference value acquisition process is performed (S2). Here, using the personal information model created in S1, feature points in a state where the driver 3 is not drowsy (that is, an awake state) are acquired. The feature points on the face are shown in FIG. The feature point is coordinate information of a predetermined feature point on the face, and in this embodiment, the upper eyelids 101L and 101R, the lower eyelids 102L and 102R, the eyes 103L and 103R, the corners 104L and 104R, the eyebrows 105L, 105R, upper eyebrows 106L and 106R, buttocks 107L and 107R, noses 108L and 108R, upper lip 109, lower lip 110, corners 111L and 111R, and contours 112 to 116 are defined. Note that the feature points are not limited to those described above, and can be set to various points as long as changes in facial expressions accompanying sleepiness appear.

Note that acquisition / detection of feature points refers to acquisition / detection of coordinates corresponding to feature points. Details of this reference value acquisition process will be described later.
Next, sleepiness estimation processing is performed (S3). Here, the amount of movement is calculated by comparing the feature point in the awake state acquired in S2 and the feature point acquired at the time of executing S3, and the amount of movement is compared with the learning data that is stored in advance. Are calculated in 6 levels from level 0 to level 5. Further, a reliability A that is a value indicating the measurement accuracy of the feature point and a reliability B that is a value indicating the determination accuracy of the drowsiness level are also calculated. Details of this sleepiness estimation process and learning data will be described later.

  Next, based on the drowsiness level and the reliability levels A and B determined in S3, a signal for causing the alarm device 30, the neck air conditioner 40, the seat belt vibration device 50, and the brake control device 60 to perform an operation is output. (S4).

  Here, when it is determined that the sleepiness level is 0, no operation is performed on any of the above-described apparatuses. When it is determined that the sleepiness level is 1 or 2, the alarm device 30 and the neck air conditioner 40 are operated. When the sleepiness level 3 is determined, the alarm device 30, the neck air conditioner 40, and the seat belt vibration device 50 are operated. When it is determined that the drowsiness level is 4 or 5, all the devices are operated. However, when both the reliability levels A and B are lower than the predetermined threshold value, the operation of the brake control device 60 is performed. Not performed.

After S4, the process returns to S3, and sleepiness is estimated again.
(2.2) Personal Adaptation Processing The personal adaptation processing procedure will be described based on the flowchart shown in FIG. This process is executed in S1 of the dozing prevention process (FIG. 3).

In this process, first, a photographed image of the driver 3 photographed by the photographing device 10 is acquired from the image capture board 22 (S10).
Next, a region that is a front face (portion corresponding to FIG. 6B) is detected from the captured image acquired in S10 (S12). In this embodiment, the detection is performed using the haar-like feature, but a method using a face image template may be used.

  Next, feature points are detected from the front face detected in S12 (S14). Here, each feature point is detected by fitting using a common 2D facial expression model (AAM: Active Appearance Model), and its two-dimensional coordinates are acquired. Note that the feature points may be detected using edge feature points or Gabor feature values instead of the fitting by AAM.

  Next, it is determined whether or not the feature point detected in S14 shows an abnormal value (S16). Here, it is determined whether or not the distance between the predetermined feature points indicates an abnormal value exceeding a predetermined threshold value, and if either is an abnormal value (S16: YES), it is acquired in S10. It is determined that the image data is not a front face, and the process returns to S10 to acquire the image data again. The distance between the predetermined feature points mentioned here corresponds to, for example, the opening degree of the eye (distance between the upper eyelid and the lower eyelid), the opening degree of the mouth (distance between the upper and lower lips), and the like. . On the other hand, if none of them is an abnormal value (S16: NO), the process proceeds to S18.

  Next, a personal expression model is created (S18). The personal expression model includes a 2D reference face shown in FIG. 6A, a 2D template shown in FIG. 6B, a 2D expression vector shown in FIG. 6C, a 3D reference face shown in FIG. It consists of a 3D expression vector shown in (e) and a 2D transformation matrix (not shown).

The 2D reference face is a face shape model composed of a plurality of feature points having two-dimensional coordinates.
The 2D template is a face image corresponding to a 2D reference face area.
The 2D expression vector is a two-dimensional vector that changes the position of the feature point of the 2D reference face, and is set for each feature point.

The 3D reference face is a face shape model including a plurality of feature points having three-dimensional coordinates.
The 3D expression vector is a three-dimensional vector that changes the position of the feature point of the 3D reference face, and is set for each feature point.

The 2D conversion matrix is a perspective projection matrix used when converting each feature point of the 3D reference face into two-dimensional coordinates, and corresponds to the conversion formula in the present invention.
Note that the 2D reference face described above is a 2D facial expression because fitting is performed by changing the facial expression (adjusting the coordinates of feature points) in a state combined with the 2D template (see FIG. 6F). The direction of the vector is determined in advance so as to follow the change in facial expression (movement of facial muscles). Similarly, the direction of the 3D facial expression vector is determined along the change in facial expression.

  In S18, a 2D reference face is created based on the feature points detected in S14, and the front face detected in S12 is set as a 2D template. The 2D reference face created here corresponds to the reference 2D face model in the present invention, and the 2D template set here corresponds to the reference image in the present invention.

  The 2D expression vector is customized according to the created 2D reference face. The 2D reference face and the 2D expression vector have a general-purpose model having a correlation with each other, and the 2D expression vector of the general-purpose model is changed according to the change from the 2D reference face to the newly created 2D reference face in the general-purpose model. And a new 2D expression vector is set.

The 3D reference face, 3D expression vector, and 2D conversion matrix have general-purpose data in advance. After this S18, this process ends.
(2.3) Reference Value Acquisition Processing A procedure for the reference value acquisition processing will be described based on the flowchart shown in FIG. This process is executed in S2 of the dozing prevention process (FIG. 3).

In this process, first, counting of elapsed time is started (S30).
Next, facial expression feature amount detection processing is performed (S32). Here, the face of the driver 3 is photographed, and based on the photographed image of the face, three-dimensional coordinates of a plurality of feature points on the face of the driver 3 (three-dimensional coordinates of the plurality of feature points are referred to as three-dimensional feature points hereinafter). Is acquired and stored. Details of the facial expression feature amount detection processing will be described later.

  Next, it is determined whether or not 5 minutes have elapsed since the elapsed time started in S30 (S34). If five minutes have not elapsed (S34: NO), the process returns to S32 to newly acquire a three-dimensional feature point set. If 5 minutes have elapsed, the process proceeds to S36.

  Next, only the three-dimensional feature point set with high measurement accuracy is selected from the plurality of three-dimensional feature point sets acquired in S32 (S36). The accuracy of the measurement is calculated simultaneously with the calculation of the three-dimensional feature point set as a value indicating the degree of coincidence when the captured image and the personal expression model are fitted in the expression feature amount detection process of S32. Here, a three-dimensional feature point set whose value is equal to or greater than a predetermined threshold is selected.

  Next, using the three-dimensional feature point set selected in S36, an average value for each feature point is calculated to determine an average three-dimensional coordinate for each feature point (hereinafter also simply referred to as a reference value). Then, the calculated reference value is stored, and then this process is terminated.

In addition, the process of S30-S34 aims at acquiring the average three-dimensional coordinate of each feature point in an arousal state by acquiring a some 3D feature point set over a fixed time. Therefore, as long as a large number of 3D feature point sets can be acquired, the time may be set to a time other than 5 minutes. In such a case, the process may proceed to S36.
(2.4) Drowsiness Estimation Processing The procedure of sleepiness estimation processing will be described based on the flowchart shown in FIG. This process is executed in S3 of the dozing prevention process (FIG. 3).

In this process, first, counting of elapsed time is started (S40).
Next, facial expression feature amount detection processing is performed (S42). Here, the face of the driver 3 is photographed, and a three-dimensional feature point set is acquired and stored based on the photographed image of the face. Details of the facial expression feature amount detection processing will be described later.

  Next, it is determined whether or not 5 seconds have elapsed since the start of counting the elapsed time in S40 (S44). If 5 seconds have not elapsed (S44: NO), the process returns to S42, and a new three-dimensional feature point set is acquired. If 5 seconds have elapsed (S44: YES), the process proceeds to S46.

  Next, a three-dimensional feature point set with high measurement accuracy is selected from the plurality of three-dimensional feature point sets acquired in S42, and a ratio of the three-dimensional feature point set with high accuracy is calculated (S46). ). The measurement accuracy is calculated as a value simultaneously with the calculation of the three-dimensional feature point set, similar to the measurement accuracy in S36 described above. Here, a three-dimensional feature point set whose value is equal to or greater than a predetermined threshold is selected, and the ratio of the three-dimensional feature point set equal to or greater than the predetermined threshold is stored as the reliability A.

Next, the average value for each feature point is calculated using the three-dimensional feature point set selected in S46, and the average value of the three-dimensional coordinates for each feature point is calculated (S48).
Next, a standard value is calculated (S50). Specifically, the average value of the three-dimensional coordinates calculated in S48 is compared with the reference value acquired in S38 of the reference value acquisition process, and the movement amount for each feature point is calculated. The movement amount is stored as a standard value.

  Next, sleepiness estimation is performed (S52). Here, the drowsiness level is calculated by the drowsiness estimation algorithm using the standard value calculated in S50. The sleepiness estimation algorithm will be described with reference to FIG.

  First, the driver data 201 shown in the figure will be described. The values (x1, y1, z1, x2, y2, z2,...) Shown in the driver data 201 are obtained by arranging the standard values of the feature points calculated in S50. Numerical values (1, 2,...) attached to x, y, z are numerical values for identifying feature points, and (x1, y1, z1), (x2, y2, z2),. , Yi, zi) indicate standard values at the respective feature points.

In order to perform sleepiness estimation, first, the driver data 201 is compared with the learning data 202 stored in advance in the HDD 23.
The learning data 202 is a collection of a plurality of sets 205 of drowsiness levels 203 and standard values 204 that are paired with the drowsiness levels 203. The standard value 204 in each set 205 corresponds to S1 (personal adaptation processing), S2 (reference value acquisition processing) of dozing prevention processing, and S30 to S50 of sleepiness estimation processing for a person who is a sample of the learning data 202. It is calculated by performing processing. That is, it is calculated by the same method as the driver data 201.

  The sleepiness level 203 is a value obtained by sensory evaluation of the degree of sleepiness of the person when executing S30 to S50 in the sleepiness estimation process. The learning data 202 has a plurality of sets 205 regarding a plurality of persons.

  After comparing the driver data 201 with the standard values 204 of all the sets 205 in the learning data 202, the top K sets 205 having high similarity to the driver data 201 are extracted. Then, the most sleepy level in the extracted set 205 is estimated to be the sleepiness level of the driver 3. In the case of FIG. 9, the sleepiness level 0 is estimated to be “0” because the set of sleepiness level 0 is the most in the top K sets having high similarity. For example, the K-NN method may be used for the process of estimating the drowsiness level by comparing the driver data 201 and the learning data 202.

  The matching accuracy of the sleepiness estimation, that is, the ratio of the number of sets having the sleepiness level that is the largest among the total number (K) of the extracted sets is stored as the reliability B. Thereafter, this process is terminated.

In addition, the process of S40-S44 of this process is counting 5 seconds short compared with 5 minutes of S30-S34 of the reference value acquisition process mentioned above. Here, although it takes time to acquire a plurality of three-dimensional feature point sets in order to acquire the average three-dimensional coordinates of each feature point, it is necessary to immediately detect sleepiness when the driver 3 develops sleepiness. Therefore, in order to estimate sleepiness at short time intervals, the time is shorter than 5 minutes. Similarly to the case of S34, a time setting other than 5 seconds may be set, or the number of times the three-dimensional feature point set is acquired is counted, and the process proceeds to S46 when the predetermined number of times is exceeded. May be.
(2.5) Facial Expression Feature Quantity Detection Processing A processing procedure of facial expression feature quantity detection processing will be described based on the flowchart shown in FIG. This process is executed in S32 of the reference value acquisition process (FIG. 7) and S42 of the sleepiness estimation process (FIG. 8).

  In this process, first, a photographed image photographed by the photographing apparatus 10 is acquired from the capture board (S60). The captured image acquired here corresponds to the determination image in the present invention.

  Next, the 2D reference face created in S18 of the personal adaptation process (FIG. 5) is transformed by moving each feature point using the 2D facial expression vector, and a face shape model composed of each feature point after the transformation. A 2D facial expression A is created (S62). The 2D facial expression A corresponds to the modified 2D face model in the present invention.

  Next, the 2D template created in S18 of the personal adaptation process is modified so as to match the 2D facial expression A created in S62 (S64). When the 2D reference face and the 2D template are combined, as shown in FIG. 6F, the 2D template is deformed in accordance with the movement of the feature point when the 2D reference face is transformed into the 2D expression A. This deformed 2D template corresponds to a deformed image in the present invention.

  Next, the image of the part (face part) corresponding to the 2D reference face in the captured image acquired in S60 and the 2D template deformed in S64 are fitted (S66), and the error is digitized. For this digitization, a method using a luminance difference or a method using a normalized cross-correlation can be considered.

  Next, it is determined whether the captured image matches the 2D template (S68). If the error acquired in S66 exceeds a predetermined threshold value, it is determined that they do not match (S68: NO), and the deformation amount of the 2D reference face is reset (S70). Specifically, the magnitude of the 2D facial expression vector that reduces the error is determined for each feature point using the steepest descent method, Newton method, or the like. Then, after the 2D facial expression A created in S62 is discarded, the process returns to S62, and the 2D reference face is deformed using the reset 2D facial expression vector.

  On the other hand, if the error is equal to or smaller than the predetermined threshold value in S68, it is determined that the photographed image matches the 2D template (S68: YES), and the feature of the image photographed by the 2D facial expression A created in S62 is determined. Determine that it represents a point. Thereafter, the process proceeds to S72. The determined 2D facial expression A corresponds to the determined 2D coordinate in the present invention.

  Subsequently, the 3D reference face shown in FIG. 6 (d) is deformed by moving it using the 3D expression vector shown in FIG. 6 (e), and the 3D expression is a face shape model composed of each feature point after the deformation. Is created (S72). In creating the 3D facial expression, the movement amount of the 3D facial expression vector is determined so as to approach the face with reference to the 2D reference face created in S18 of the personal adaptation process. This 3D facial expression corresponds to the modified 3D face model in the present invention.

  Next, when the 3D facial expression created in S72 has an unnatural shape as a face structure, 3D facial expression correction processing is performed to correct the deviation (S74). By this 3D facial expression correction processing, a rule for determining the positional relationship of the feature points is given to the deformation of the 3D reference face in S72 to suppress an unnatural shape. This has the effect of suppressing falling into a local solution in the error minimization problem using the steepest descent method or Newton method in S82 and S84 described later. Details of this 3D facial expression correction process will be described later.

  Next, using the 2D conversion matrix, a 2D facial expression B obtained by converting the 3D facial expression created in S72 into two-dimensional coordinates is calculated (S76). The 2D facial expression B corresponds to the converted 2D coordinates in the present invention.

Next, the 2D facial expression B and the 2D facial expression A calculated in S76 are fitted (S78).
Next, it is determined whether or not the 2D facial expression A and the 2D facial expression B match (S80). Here, the distance for each feature point is measured as an error, and if the arithmetic average of the errors exceeds a predetermined threshold, it is determined that they do not match (S80: NO), and the error is reduced. The subsequent processes of S82 and S84 are performed as described above.

  First, the deformation amount of the 3D reference face is reset (S82). Specifically, the size of the 3D facial expression vector that reduces the error is determined for each feature point using the steepest descent method, Newton method, or the like. Thereby, each feature point of the 3D reference face can be moved to reduce the difference in expression between the 3D expression and the photographed image.

  Next, the 2D conversion matrix is updated (S84). Specifically, a 2D conversion matrix with a small error is determined using a steepest descent method, a Newton method, or the like. This makes it possible to reduce the error by adjusting the face direction when making the 3D reference face two-dimensional, and to reduce the difference in the face direction between the 3D facial expression and the captured image. After S84, the 2D facial expression B is discarded, and the 2D facial expression B is created again in S72 to S76.

In S80, if the average value of errors is equal to or less than a predetermined threshold value, it is determined that they match (S80: YES), and the 3D facial expression corrected in S74 represents the feature point of the captured image. The three-dimensional coordinates (three-dimensional feature point set) are stored (S86). Thereafter, this process is terminated.
(2.6) 3D Facial Expression Correction Processing The processing procedure of 3D facial expression correction processing will be described based on the flowchart shown in FIG. This process is executed based on the 3D facial expression created in S72 immediately before in S74 of the facial expression feature amount detection process (FIG. 10). In the following description, the feature points located on both the left and right sides are described only on the left side.

  In this process, first, it is determined whether or not the distance between eyelids is 0 or less (S100). The state where the distance between the eyelids is 0 or less is a state where the upper eyelid 101L is located below the lower eyelid 102L as shown in FIG. 12, and at this time, the upper eyelid 101L is located from the position of the lower eyelid 102L. The upward distance to the position of 0 is 0 or minus.

  If the distance between the eyelids is not less than or equal to 0 (S100: NO), the process proceeds to S104. If the distance between the eyelids is 0 or less (S100: YES), the lower eyelid 102L, the eyes 103L, and the eyes 104L are changed to predetermined positions based on the position of the upper eyelid 101L (S102). Thereafter, the process proceeds to S104.

  Next, as shown in FIG. 13, it is determined whether or not the distance L1 between the upper and lower eyelids is larger than the distance L2 between the corners of the eyes and the eyes (S104). This prevents the distance between the upper and lower eyelids from being too far apart.

  If L1> L2 is not satisfied (S104: NO), the process proceeds to S108. If L1> L2 (S104: YES), the lower eyelid 102L, the eye 103L, and the eye corner 104L are changed to predetermined positions based on the position of the upper eyelid 101L (S106). Thereafter, the process proceeds to S108.

  Next, as shown in FIG. 14, it is determined whether or not the distance L3 between the eyebrow head 105L and the upper eyelid 101L is larger than one half of the distance L4 between the temples (between the contours 112 and 116) (S108). ). Thereby, it is prevented that the distance between eyebrows is separated too much.

  If L3> (L4 / 2) is not satisfied (S108: NO), the process proceeds to S112. If L3> (L4 / 2) (S108: YES), the eyebrow head 105L, the eyebrow upper end 106L, and the eyebrow butt 107L are changed to predetermined positions based on the position of the upper eyelid 101L (S110). Thereafter, the process proceeds to S112.

  Next, it is determined whether or not the distance between the eyebrow feature points (eyebrow head 105L, eyebrow upper end 106L, eyebrow butt 107L) and upper eyelid 101L is 0 or less (S112). The state where the distance between the eyebrows and the upper eyelid is 0 or less is a state in which the feature point of the eyebrows is located below the upper eyelid 101L as shown in FIG. The upward distance from the position to the eyebrow feature point is 0 or minus.

  If the distance between the eyebrows and the upper eyelid is not less than 0 (S112: NO), the process proceeds to S116. If the distance between the eyebrows and the upper eyelid is 0 or less (S112: YES), the eyebrow head 105L, the upper eyebrow upper end 106L, and the eyebrows upper end 107L are changed to predetermined positions based on the position of the upper eyelid 101L (S114). . Thereafter, the process proceeds to S116.

  Next, it is determined whether or not the distance between lips is 0 or less (S116). The state where the distance between the lips is 0 or less is a state where the upper lip 109 is located below the lower lip 110 as shown in FIG. The distance to go is 0 or minus.

  If the distance between lips is not less than or equal to 0 (S116: NO), the process proceeds to S120. If the distance between the lips is 0 or less (S116: YES), the positions of the upper lip 109 and the lower lip 110 are switched (S118). Thereafter, the process proceeds to S120.

  Next, as shown in FIG. 17, it is determined whether or not the distance L5 between the jaw position (contour 114) and the lower lip 110 is smaller than one half of the distance L6 between the eye 103L and the outer corner 104L ( S120). This prevents the lower lip from falling too much.

  If L5 <(L6 / 2) is not satisfied (S120: NO), the process proceeds to S124. If L5 <(L6 / 2) (S120: YES), the lower lip 110 is changed to a predetermined position based on the jaw position (contour 114) (S122). Thereafter, the process proceeds to S124.

  Next, it is determined whether or not the distance between the nose 108L and the upper lip 109 is 0 or less (S124). The state where the distance between the nose 108L and the upper lip 109 is 0 or less is a state where the nose 108L is positioned below the upper lip 109 as shown in FIG. 18, and the distance from the position of the upper lip 109 to the nose 108L. The upward distance is 0 or minus.

  If the distance between the nose 108L and the upper lip 109 is not less than 0 (S124: NO), the process proceeds to S128. If the distance between the nose 108L and the upper lip 109 is 0 or less (S124: YES), the upper lip 109 is changed to a predetermined position based on the position of the nose 108L (S126). Thereafter, the process proceeds to S128.

  Next, it is determined whether the distance between the mouth corners is 0 or less (S128). The state where the distance between the mouth corners is equal to or less than 0 is a state where the left mouth corner 111L is positioned to the right of the right mouth corner 111R as shown in FIG. 19, and is located to the right from the left mouth corner 111L to the right mouth corner 111R. The distance to go is 0 or minus.

If the distance between the mouth corners is not less than or equal to 0 (S128: NO), this process ends. If the distance between the mouth corners is 0 or less (S128: YES), the positions of the left mouth corner 111L and the right mouth corner 111R are switched (S130). Thereafter, this process is terminated.
(3) Effects of the Invention In the dozing prevention system 1 configured as described above, the 2D reference face and the 2D template in the personal expression model are created using information acquired from the driver 3 by the personal adaptation process. Therefore, the detection accuracy of feature points can be increased, and as a result, the detection accuracy of sleepiness can be increased.

  Also, since the 3D reference face and the 2D conversion matrix are not customized to the driver 3, there is no need to create a model by creating various images of the driver 3, and the driver 3 does not have any special features. There is no need to take action. Therefore, it is possible to detect drowsiness while reducing the burden on the driver 3 for creating the personal facial expression model. Such a dozing prevention system 1 is easy to put into practical use as compared with a configuration in which the driver 3 is photographed to create a 3D reference face and a 2D conversion matrix.

Further, the 3D reference face and the 2D conversion matrix are common models created in advance, and the moving direction of the 3D facial expression vector is fixed.
First, FIGS. 20A and 20B show the actual position and operation of the eyebrows of the two persons A and B, whose feature points are to be detected. FIG. 20A shows the eyebrow position at the time of awakening, and FIG. 20B shows the eyebrow position that rises when the person has drowsiness (drowsiness).

  A and B are different in both the distance between the eyebrows and the distance between the eyebrows during awakening and somnolence. The direction of movement when the eyebrows are raised is also different. When using a 3D reference face, a 2D transformation matrix, and a 3D expression vector customized for each person as in the past, based on the eyebrow position before moving and the individual difference in the moving direction based on the position and amount of movement of the eyebrow Therefore, there is a problem that it is difficult to judge changes in facial expression uniformly.

  On the other hand, when a common model is used as in the present embodiment, as shown in FIG. 21, the eyebrows of the common 3D facial expression move within a predetermined range by a 3D facial expression vector whose direction is determined. Therefore, the positions of the eyebrows A and B are determined as any position obtained by moving the eyebrows in the common 3D reference face in a predetermined direction.

  In addition, as shown in FIG. 22, although there are individual differences in the eyebrow position during awakening and the eyebrow position during somnolence, whether or not the eyebrows have moved is based on the amount of eyebrow movement between awakening and somnolence. Can be judged.

  In this way, even if there are individual differences in the shape and movement direction of the target person's face, changes in facial expressions can be handled using only the movement amount of the feature point as a parameter, so the initial position and movement direction of the feature point, etc. Therefore, it is possible to easily determine and compare facial expression changes.

  Furthermore, since the data acquired by performing the process similar to the dozing prevention system 1 of a present Example with respect to a some subject is used as learning data, the determination precision of a drowsiness level can be improved.

Further, in the dozing prevention system 1 having the above-described configuration, the 3D facial expression is not unnaturally shaped by performing the 3D facial expression correction process because the feature point does not move to a position outside the rule. Therefore, useless fitting processing that uses an unnatural 3D expression can be eliminated, and output of detection results with low accuracy can be suppressed.
(4) Modifications While the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and can take various forms as long as they belong to the technical scope of the present invention. Needless to say.

  For example, in the above-described embodiment, the configuration in which the reference value acquisition process for acquiring the feature point of the awake state is performed immediately after the personal adaptation process is illustrated, but it may be performed at other timings. For example, a heart rate measuring means such as a measurement sensor for measuring the heart rate of the driver 3 is attached to a part of the vehicle 2 such as a handle or a seat that comes into contact (approaching) with the heart rate measured by the measurement sensor. The driver 3 may be determined to be in an awake state when the average value is within a predetermined range, and the reference value acquisition process may be executed when the driver 3 is in an awake state.

  In the dozing prevention system 1 configured as described above, the arousal state can be determined based on the heart rate. Therefore, since the captured image captured when not in the awake state is not processed as the captured image when awakening, drowsiness can be detected with high accuracy.

  In the above embodiment, the 2D reference face and the 2D template are exemplified based on the captured image acquired in the personal adaptation process. However, only the 2D reference face and the 2D template are acquired in advance. The structure to be used may be used. For example, the data may be stored in the HDD 23 and read when the driver 3 drives, or the driver 3 may have a storage medium storing the data. In the driving, the data may be read from the storage medium.

  Further, in the 3D facial expression correction process in S74 of the facial expression feature amount detection process (FIG. 10), the rule for preventing an unnatural shape is not limited to the rule shown in FIG. For example, it is conceivable to limit the distance between the left and right mouth corners to be equal to or less than half of the distance between the temples.

In addition, when the reference value of the distance to be compared is other than 0 (S104, S108, S120), the distance between the feature points serving as the reference value is not limited to that described in the above embodiment. For example, in S104, the rule is that the distance between the eye corners and the eyes is smaller than the distance so that the distance between the eyelids is not too far, but the distance between the left and right eyes may be used as a reference. The distance used as the reference value is preferably a distance that is not easily affected by changes in facial expressions.
(5) Correspondence Relationship The processing of S10 and 60 executed by the control unit 21 corresponds to the processing by the image acquisition unit of the present invention, the processing of S18 corresponds to the processing by the reference 2D face model creation unit, and the processing of S62 Corresponds to the processing by the 2D deformation means, the processing of S64 corresponds to the processing by the deformation image creation means, the processing of S66 to 70 corresponds to the processing by the two-dimensional coordinate determination means, and the processing of S72 and 74 is the 3D deformation means. The processing of S76 corresponds to the processing by the coordinate conversion means, the processing of S78 to 84 corresponds to the processing by the three-dimensional coordinate determination means, and the processing of S50 and S52 corresponds to the processing by the sleepiness determination means. .

DESCRIPTION OF SYMBOLS 1 ... Dozing prevention system, 2 ... Vehicle, 3 ... Driver, 4 ... Dashboard, 10 ... Imaging device, 20 ... Drowsiness detection device, 21 ... Control part, 22 ... Image capture board, 23 ... Hard disk drive, 24 ... Inside the car LAN communication unit, 30 ... alarm device, 40 ... neck air conditioner, 50 ... seat belt vibration device, 60 ... brake control device, 101 ... upper eyelid, 102 ... lower eyelid, 103 ... eye, 104 ... eye corner, 105 ... eyebrow 106 ... upper end of eyebrows, 107 ... buttocks, 108 ... nose, 109 ... upper lip, 110 ... lower lip, 111 ... corner of mouth, 112 to 116 ... contour, 201 ... driver data, 202 ... learning data, 203 ... drowsiness level, 204 ... standard value, 205 ... set

Claims (9)

Image acquisition means for acquiring a captured image showing the face of the person to be imaged;
A reference 2D face for extracting two-dimensional coordinates of a plurality of predetermined feature points on a face from a reference image that is a captured image acquired by the image acquisition means and creating a reference 2D face model composed of the two-dimensional coordinates. Model creation means;
2D deforming means for creating a deformed 2D face model by moving the two-dimensional coordinates of the plurality of feature points from the reference 2D face model created by the reference 2D face model creating means;
Deformed image creating means for creating a deformed image obtained by deforming the reference image according to the deformed 2D face model created by the 2D deforming means;
The determination image, which is a captured image acquired by the image acquisition unit, and the modified image are fitted by adjusting the movement of the two-dimensional coordinates by the 2D deformation unit, and the determination image and the modified image are obtained. Two-dimensional that determines two-dimensional coordinates of the plurality of feature points in the deformed 2D face model corresponding to the deformed image when matched as fixed two-dimensional coordinates that are two-dimensional coordinates of the plurality of feature points in the determination image Coordinate determination means;
3D deforming means for creating a deformed 3D face model composed of three-dimensional coordinates obtained by moving predetermined three-dimensional coordinates corresponding to the plurality of feature points;
Coordinate conversion means for creating converted 2D coordinates obtained by converting the three-dimensional coordinates of the deformed 3D face model created by the 3D deforming means into two-dimensional coordinates using a conversion formula;
The fixed 2D coordinates determined by the two-dimensional coordinate determining means and the converted 2D coordinates are fitted by adjusting the movement of the three-dimensional coordinates by the 3D deforming means and / or the conversion formula, and the fixed 3D coordinate determination means for determining, as the 3D coordinates of the plurality of feature points in the determination image, the three-dimensional coordinates of the plurality of feature points in the deformed 3D face model corresponding to the transformed 2D coordinates that match 2D coordinates When,
A face feature point detection apparatus. The face feature point detection apparatus according to claim 1, wherein the 3D deforming unit moves the three-dimensional coordinates of the feature points so as to satisfy a rule that defines a predetermined positional relationship between the feature points. 3. The face feature point detection device according to claim 2, wherein the rule is defined such that the feature point indicating the position of the eye is positioned below the feature point indicating the position of the eyebrows. . The face according to claim 2 or 3, wherein the rule is set so that the feature point indicating the position of the mouth is located below the feature point indicating the position of the nose. Feature point detection device. 5. The rule according to claim 2, wherein the rule is defined such that the feature point indicating the position of the upper eyelid is positioned above the feature point indicating the position of the lower eyelid. The facial feature point detection apparatus according to claim 1. 6. The rule according to claim 2, wherein the rule is defined such that the feature point indicating the position of the left mouth corner is positioned to the left of the feature point indicating the right mouth corner. The facial feature point detection apparatus according to claim 1. The facial feature point detection apparatus according to any one of claims 1 to 6,
The sleep feature is determined based on the three-dimensional coordinates (A) of the plurality of feature points in the determination image determined by the face feature point detection device based on the determination image captured when the subject is awake. Sleepiness determination that determines the degree of sleepiness by comparing with the three-dimensional coordinates (B) of the plurality of feature points in the determination image determined by the facial feature point detection device based on the determination image captured at power timing And a drowsiness detection device. The drowsiness determining means includes
The amount of movement from the three-dimensional coordinates of the plurality of feature points in a state in which the subject to be photographed is awake to the three-dimensional coordinates of the plurality of feature points in a state in which the subject to be photographed is sleepy, and the sleepiness A plurality of data that associates the degree of sleepiness of the subject in the state with the degree of sleepiness,
The amount of sleepiness is determined by comparing the three-dimensional coordinates (A) and the three-dimensional coordinates (B), calculating the movement amounts of the plurality of feature points, and comparing the movement amounts with the data. The drowsiness detection device according to claim 7. Equipped with a heart rate measuring means for measuring the heart rate of the subject,
The time of awakening is a time when the average value of the heart rate measured by the heart rate measuring means is within a predetermined range. Sleepiness detection device.