JP4458146B2 - Sleepiness determination apparatus and program - Google Patents

Description

  The present invention relates to a drowsiness determination device, a program, and a drowsiness determination method for determining the degree of drowsiness based on the movement of a subject's face.

2. Description of the Related Art Conventionally, in order to prevent a driver of a vehicle from taking a nap, an apparatus for photographing a driver's face with a camera mounted in the vehicle and processing the obtained image to determine the degree of sleepiness has been proposed. .
In such a device, it is common to identify the driver's eyes from the obtained images, detect the eye opening, line of sight, etc., and determine whether or not they are dozing based on them It is. For example, there has been proposed an apparatus that detects the opening degree of eyes by continuously detecting the opening degree of the eyes to detect that the subject is dozing (see Patent Document 1).
JP-A-11-66304

  However, eye movements often change depending on the environment in which the vehicle is traveling. For example, when the driver's eyes are exposed to sunlight, an operation of narrowing the eyes than usual may be performed regardless of sleepiness. In addition, the confirmation frequency of the driver's mirror differs between the expressway and the city regardless of sleepiness.

  Although eye movements are directly related to the subject's sleepiness, as described above, there are many disturbance factors that impede the determination of the degree of drowsiness. It was necessary to take measures.

  The present invention has been made in view of such problems, and a drowsiness determination device, a program, and a drowsiness determination method capable of determining drowsiness without being affected by disturbance factors that hinder the determination of the degree of drowsiness. The purpose is to provide.

The drowsiness determination device according to claim 1, which is made to solve the above-described problem, includes a distance between left and right mouth corners, a distance between eyebrows and eyes, a head tilt angle, and a distance between a midpoint of the eyebrows and a midpoint of the eyes. , Distance between eyebrows, distance between midpoint of lower nose and midpoint of eye, distance between midpoint of mouth and midpoint of eye, distance between lower midpoint of lower lip and midpoint of eye, upper eyelid and lower Acquire at least two of the facial expression information, such as the distance between the eyelids, the distance between the lower right and left ends of the nose, and the distance between the upper lip upper end and the midpoint of the eye, and determine the facial expression information acquired at awakening and sleepiness by comparing the facial expression information obtained in should do the timing, with the respectively obtained facial expression information, and determines the presence or absence of drowsiness per the facial expression information, O and there sleepiness to the number of the determined facial expression information as the ratio of the number of the determined facial expression information is large, it is determined that the strong degree of drowsiness. The criteria for determining the presence or absence of sleepiness are the distance between the left and right mouth corners, the distance between the upper and lower eyelids, and the distance between the left and right lower nose. While it is judged that there is drowsiness when the value of facial expression information acquired at awakening is smaller, the distance between eyebrows and eyes, head tilt angle, distance between midpoint of eyebrows and midpoint of eyes, distance between eyebrows, The distance between the midpoint of the lower nose and the midpoint of the eye, the distance between the midpoint of the mouth and the midpoint of the eye, the distance between the lower lip and the midpoint of the eye, and the distance between the upper lip and the midpoint of the eye Is determined to be drowsy when the value of the facial expression information acquired at the timing when sleepiness should be determined is greater than the value of the facial expression information acquired at awakening.
The drowsiness determination device according to claim 2, which is made to solve the above-described problem, includes a distance between left and right mouth corners, a distance between eyebrows and eyes, a head tilt angle, a middle point between eyebrows and a middle point of eyes. Distance, distance between eyebrows, distance between midpoint of lower nose and midpoint of eye, distance between midpoint of mouth edge and midpoint of eye, distance between lower midpoint of lower lip and midpoint of eye, upper eyelid And at least two of the facial expression information, such as the distance between the lower eyelid, the distance between the left and right sides of the nose, the distance between the upper lip upper edge and the middle point of the eye, and the facial expression information acquired at awakening and sleepiness by comparing the facial expression information obtained by the timing to determine, whether based on each acquired facial expression information, corresponding to one of a plurality of stages in which the degree of drowsiness per the facial expression information is predetermined determines the average value of the step that indicates the degree of該眠air also in the determination the facial expression information Determining the degree of drowsiness based on the mode. The degree of sleepiness is determined by the distance between the left and right mouth corners, the distance between the upper and lower eyelids, and the distance between the left and right sides of the nose. When it is smaller than the value of facial expression information that is sometimes obtained, drowsiness is detected, and the degree of drowsiness is determined by the magnitude of the difference between these values, while the eyebrow-eye distance, head tilt angle, The distance between the middle point of the eye, the distance between the eyebrows, the distance between the middle point of the lower nose and the middle point of the eye, the distance between the middle point of the mouth edge and the middle point of the eye, and the middle point of the lower lip and the middle point of the eye The distance between the upper lip and the middle point of the upper lip is the timing at which sleepiness should be determined when the value acquired at the timing when sleepiness should be determined is greater than the value acquired at awakening. Sleep due to the magnitude of the difference between the acquired value and the value acquired at awakening To determine the degree of.

  Usually, when the degree of sleepiness changes, the facial expression changes. FIG. 1 shows eleven facial muscles that change facial expressions. The arrows in the figure indicate the direction of movement of the skin when each facial muscle contracts. When the degree of sleepiness changes, the facial muscles move due to relaxation and contraction of each facial muscle, and the facial expression changes. Moreover, when sleepiness becomes strong, it becomes difficult to keep the head upright, and the head tilts.

Therefore, the degree of drowsiness can be determined by measuring the distance between facial feature points and the inclination of the head reflecting the movement of facial muscles associated with sleepiness.
In the case of the drowsiness determination device configured as described above, the distance between the left and right mouth corners, the distance between the eyebrows and the eyes, the head tilt angle, the distance between the middle point of the eyebrow and the middle point of the eye, the distance between the eyebrows, and the middle of the lower nose The distance between the point and the middle point of the eye, the distance between the middle point of the mouth and the middle point of the eye, the distance between the lower edge of the lower lip and the middle point of the eye, the distance between the upper eyelid and the lower eyelid, between the left and right lower ends of the nose The degree of drowsiness can be determined based on any two or more facial expression information among the facial expression information such as the distance between the upper lip upper end and the middle point of the upper eye. Therefore, information other than eye information can be used to determine the degree of sleepiness. Therefore, the degree of sleepiness can also be determined when affected by disturbance factors that hinder the acquisition of eye information.

  In the above-described sleepiness determination apparatus, the degree of sleepiness is determined by combining at least two or more determination results of the presence or absence or degree of sleepiness based on each of the facial expression information described above. Each information described above has a different degree of sleepiness in which a change occurs, as will be described later in [Test for confirming tendency of sleepiness expression feature to sleepiness degree]. For example, the distance between the left and right mouth corners changes with mild sleepiness, the distance between the eyebrows and the eyes changes with moderate sleepiness, and the head tilt angle changes with strong sleepiness. Therefore, by combining the information and determining the degree of sleepiness, the degree of sleepiness can be determined in multiple stages, so the degree of sleepiness can be compared with the determination of sleepiness based on a single piece of information. More detailed determination can be made.

The drowsiness determination device configured as in claim 1 determines the degree of drowsiness according to the ratio of the facial expression information determined to be drowsiness that is determined to have drowsiness. Therefore, the result of sleepiness determination based on all facial expression information can be reflected in the determination result. As a result, even if certain facial expression information is erroneously determined to be drowsy due to a change in facial expression that does not involve drowsiness, the determination of the degree of sleepiness is performed in combination with the determination result by other facial expression information, It is possible to suppress erroneous determination of the degree of sleepiness.
Further, the drowsiness determination device configured as in claim 2 determines a level of drowsiness based on each facial expression information, and the drowsiness level is determined based on an average value or mode value of the level of drowsiness. Determine the degree.
With the drowsiness determination device configured as described above, the degree of drowsiness can be determined based on higher-order information in which the degree of drowsiness of each facial expression information is classified into a plurality of stages. be able to.

In addition, since the drowsiness determination device configured as in claim 1 or 2 can detect that the facial expression has changed by comparison with awakening, the presence / absence or degree of drowsiness is appropriately determined. be able to.
The specific method for acquiring the facial expression information described above is not particularly limited. For example, it is conceivable to detect each facial expression information described above by attaching electrodes to the face and detecting the weak electricity generated in proportion to the magnitude of the movement of the facial expression muscle described above.

Further, as a method for determining drowsiness by a method other than the method described above, a method of detecting each facial expression information described above by photographing the face of the subject and performing image processing on the face image is conceivable. For that purpose, it may be configured as drowsiness determination apparatus according to claim 5.

  The drowsiness determination device detects position / angle information indicating at least a position of a component of a face or a tilt angle of a head based on image data indicating a face image, and based on the position / angle information, the facial expression It is configured to obtain information.

  With the sleepiness determination device configured as described above, the degree of sleepiness can be determined based on the image data indicating the face image captured by the imaging unit. Therefore, the degree of drowsiness can be easily determined without directly attaching the electrodes for detecting each facial expression information described above to the subject who determines the degree of drowsiness.

Further, the drowsiness determination device described in claim 5 may be used by being mounted on a vehicle provided with photographing means, like the drowsiness determination device described in claim 6 . In this case, the position / angle information described above is detected based on the image data indicating the face image captured by the image capturing means, the facial expression information is acquired based on the position / angle information, and sleepiness is based on the information. It is good to comprise so that the degree of may be determined.

Since the drowsiness determination device configured as described above can be mounted on a vehicle and used, the degree of drowsiness can be determined for a vehicle driver.
In addition, the apparatus which performs the operation | movement which prevents a driver's doze according to the determination result of the degree of sleepiness is provided in a vehicle, and a driver's doze can be prevented by cooperating with the apparatus. For example, it is conceivable that the drowsiness determination device cooperates with a car navigation system to perform screen display and audio output according to the determination result of the degree of sleepiness. Further, a device that blows air to the driver, a device that vibrates a seat belt, a device that controls a brake, or the like may be operated according to the determination result of the degree of sleepiness.

Moreover, you may determine the degree of sleepiness by methods, such as the sleepiness determination apparatus of Claim 3 .
This drowsiness determination device acquires at least two of the distance between the left and right mouth corners, the distance between the eyebrows and the eyes, and the head tilt angle, and determines that there is drowsiness based on the head tilt angle. Is defined as Condition A, Condition B is determined to be drowsy based on the distance between the eyebrows and eyes, and Condition C is determined to be drowsy based on the distance between the left and right mouth corners. And this drowsiness determination apparatus determines the degree of drowsiness based on these conditions.

  Specifically, when the condition A is satisfied, the sleepiness is determined to be strong. Further, when the condition B is satisfied and the condition A is not satisfied, it is determined that the sleepiness is moderate sleepiness lower than the intense sleepiness. In addition, when the condition C is satisfied and the condition A and the condition B are not satisfied, it is determined that the sleepiness is milder than that of the moderate sleepiness. If none of the conditions A, B, and C is satisfied, it is determined that there is no sleepiness.

As described above, the distance between the left and right mouth corners changes with mild sleepiness, the distance between the eyebrows and the eyes changes with moderate sleepiness, and the head tilt angle changes with strong sleepiness. appear. Therefore, in the case of a drowsiness determination device that determines the degree of drowsiness by such a method, the facial expression information acquisition unit acquires information on any of the left and right mouth corner distances, the eyebrow and eye distances, and the head tilt angle. The level of drowsiness can be determined in four stages: intense drowsiness, moderate drowsiness, mild drowsiness, and no drowsiness. In addition, when the facial expression information acquisition means acquires two of the above-mentioned information, the degree of sleepiness is determined in three stages: strong sleepiness, moderate sleepiness, mild sleepiness, and no sleepiness. Can be determined.
By the way, the specific method of judging the presence or absence or degree of sleepiness based on the acquired facial expression information is not particularly limited. For example, a method of determining drowsiness based on whether the distance and angle values indicated in the acquired facial expression information are within a predetermined reference range is conceivable. Further, as a configuration for determining drowsiness by other methods, as described in claim 4, comparing facial expression information acquired at awakening with facial expression information acquired at a timing when sleepiness should be determined A configuration is possible in which the presence or absence of drowsiness is determined and the degree of drowsiness is determined from the combination of the results.
With the drowsiness determination device configured as described above, it is possible to detect that the facial expression has changed by comparison with awakening, and therefore it is possible to appropriately determine whether or not there is drowsiness.

A program according to a seventh aspect is a program for causing a computer system to execute the processing procedure for causing the facial expression information acquiring unit and the drowsiness determining unit according to any one of the first to sixth aspects to function. is there.

A computer system controlled by such a program can constitute a part of the drowsiness determination device according to any one of claims 1 to 6 .
Note that the above-described program is composed of an ordered sequence of instructions suitable for processing by a computer system, and is provided to a sleepiness determination device and a user who uses this device via various recording media and communication lines. Is.

[Example 1]
(1.1) Overall Configuration The dozing prevention system 1 according to the first embodiment is used by being mounted on a vehicle 2, and as illustrated in FIG. 2, an imaging device 10 that captures a face image of the driver 3, A drowsiness determination device 20 that determines the degree of drowsiness based on the face image, and a plurality of devices (alarm device 30, neck air conditioner 40) that perform operations for preventing accidents due to drowsy driving according to the determined drowsiness level , A seat belt vibration device 50, and a brake control device 60).

  Among these, the imaging device 10 is arranged toward the driver 3 in the front direction of the driver 3 in the vehicle 2, and the face image of the front of the driver 3 is displayed at a predetermined time interval (for example, 1/30 s). ).

Further, as shown in FIG. 3, the drowsiness determination device 20 includes a control unit 21, a reception unit 22, an input unit 23, a storage unit 24, an output unit 25, and the like.
The control unit 21 of the sleepiness determination device 20 controls the entire sleepiness determination device 20 according to a program stored in the storage unit 24.

  The control unit 21 stores the image data in a built-in memory (hereinafter referred to as “built-in memory”) each time image data indicating a face image photographed by the photographing apparatus 10 is received from the receiving unit 22. . Thereafter, based on the image data, a control unit main process, awakening data collection process, a driving data collection process, a facial expression information acquisition process, and a drowsiness determination process, which will be described later, are performed. In the sleepiness determination process, the sleepiness level is determined as one of sleepiness level 0 (no sleepiness), sleepiness level 1 (mild sleepiness), sleepiness level 2 (medium sleepiness), and sleepiness level 3 (strong sleepiness).

The input unit 23 includes a start button for starting a control unit main process, which will be described later, and an end button for ending all processes and operations of the dozing prevention system 1.
The storage unit 24 has a storage area for a program for controlling the control unit 21 and a storage area for storing various data to be described later as storage areas for storing data.

  The output unit 25 prevents the following drowsy prevention from being given to the alarm device 30, the neck air conditioner 40, the seat belt vibration device 50, and the brake control device 60 according to the degree of sleepiness determined by the control unit 21. Outputs a signal for operation.

  Among the devices described above, the alarm device 30 includes a display and a speaker. The warning device 30 displays a message that prompts the driver 3 to take a nap of driving according to the degree of drowsiness determined by the control unit 21 (for example, when it is determined that the drowsiness level is 1, “take a break early”). ”Sho”, “Caution” when drowsiness level 2 is determined, “Stop driving” when drowsiness level 3 is determined, and the like, Outputs sound by speaker.

  Moreover, the neck air conditioner 40 is arrange | positioned at the headrest in the driver | operator's 3 sheet | seat, and when it determines with the drowsiness level 1-3 being one of the drowsiness levels 1-3 by the driver | operator's 3 head, Blow.

  The seat belt vibration device 50 is provided in a seat belt retracting mechanism, and vibrates the seat belt when the control unit 21 determines that it is the drowsiness level 2 or the drowsiness level 3.

The brake control device 60 is a device that automatically operates the brake. When it is determined that the drowsiness level 3 is determined by the control unit 21, the brake is operated to forcibly stop the vehicle running or sequentially decelerate the vehicle.
(1.2) Processing by Control Unit 21 of Drowsiness Determination Device 20 Various processes executed by the control unit 21 included in the drowsiness determination device 20 of the dozing prevention system 1 according to the first embodiment will be described below.
(1.2.1) Control Unit Main Process The processing procedure of the control unit main process performed by the control unit 21 will be described below with reference to FIG. This control unit main process is started by pressing a start button provided in the input unit 23.

  In this control unit main process, first, an awakening data collection process for collecting information on the face of the driver 3 at the time of awakening is performed (step 1; hereinafter, steps are referred to as S). Here, L1b, L2b, αb, and θb are collected as information on the face at the time of awakening, which is a reference for determining the sleepiness of the driver 3. The awakening data collection process will be described later in detail. Here, L1 is the distance information of the left and right mouth corners, L2 is the distance information of the eyebrows and the eyes, α is the tilt angle information of the head back and forth, and θ is The tilt angle information of the head to the left and right.

  Next, a driving data collection process for collecting information on the face of the driver 3 is performed (S2). Here, L1d, L2d, αd, and θd are collected as information on the face of the driver 3. This operation data collection process will be described in detail later.

Next, a sleepiness determination process for determining the degree of sleepiness of the driver 3 is performed (S3).
Here, first, by comparing L1b, L2b, αb, θb calculated in S1 with L1d, L2d, αd, θd calculated in S2, sleepiness based on L1, L2, α, θ respectively. Judgment is made.

  Based on the combination of the determination results, the sleepiness level is determined as one of sleepiness level 0 (no sleepiness), sleepiness level 1 (mild sleepiness), sleepiness level 2 (medium sleepiness), and sleepiness level 3 (strong sleepiness) To do. This sleepiness determination process will be described in detail later.

  Next, it is checked whether or not the end button of the input unit 23 has been pressed (S4). If the end button has not been pressed (S4: NO), the process returns to S2. On the other hand, if the end button is pressed (S4: YES), this process ends.

Note that this control unit main process also ends when the power unit such as the engine in the vehicle 2 stops and the control of the entire vehicle ends.
(1.2.2) Awakening Data Collection Processing A processing procedure of awakening data collection processing by the control unit 21 will be described below with reference to FIG. This awakening data collection process is executed in S1 of the control unit main process.

  In this awakening data collection process, first, data is initialized (S21). Here, the deletion of the image data stored in the internal memory of the control unit 21 and the deletion of various data stored in the storage unit 24 are performed at this time.

Next, the variable i is set to 0 (S22).
Next, facial expression information acquisition processing for detecting information on the face of the driver 3 is performed (S23). Here, L1, L2, α, and θ are calculated a predetermined number of times (m times) based on the image data indicating the face image captured by the imaging device 10 and stored in the storage unit 24. At this time, the storage unit 24 stores m pieces of L1, L2, α, and θ.

  Note that L1 is the distance between the left and right mouth corners. L2 is the distance between the center point of the left eye head and the left eye corner and the center of the left eyebrow (hereinafter simply referred to as “distance between eyebrows”), and increases when sleepiness is felt. Further, α is an angle indicating the front-back inclination of the head. Further, θ is an angle indicating the tilt of the head to the left and right. The facial expression information acquisition process will be described in detail later.

Next, based on the m pieces of L1, L2, α, θ calculated in S23, the representative values L1a, L2a, αa, θa are calculated (S24).
Here, first, all L1, L2, α, and θ stored in the storage unit 24 are read. Then, based on the frequency distribution in each of L1, L2, α, and θ, for L1 whose value decreases as sleepiness increases, a value corresponding to 5% of the cumulative relative frequency from the minimum value is calculated as L1a. On the other hand, for L2, α, and θ that increase in value due to increased sleepiness, values corresponding to 5% of the cumulative relative frequency from the maximum value are calculated as L2a, αa, and θa.

  Then, the values of L1a, L2a, αa, θa calculated in this way are stored in the storage unit 24, and L1, L2, α, θ stored in the storage unit 24 are deleted. The storage unit 24 can store a plurality of the above-described values of L1a, L2a, αa, and θa, and is stored in the storage unit 24 every time the above-described values are calculated in S24. The number of each value will increase by one.

  Thereafter, the variable i is incremented (i = i + 1) (S25). If the variable i is less than the predetermined number n (S26: YES), the process returns to S23. On the other hand, if the variable i is not less than the predetermined number n (S26: NO), the process proceeds to S27. At this time, the storage unit 24 stores n L1a, L2a, αa, and θa.

Next, L1b, L2b, αb, and θb used in sleepiness determination processing described later are calculated (S27).
In S27, all of L1a, L2a, αa, and θa stored in the storage unit 24 in S24 are read, and respective normal distributions are created. Assuming that the average value in these normal distributions is μ and the standard deviation is σ, for L1a corresponding to L1 that decreases as sleepiness increases, a value corresponding to μ−2σ is calculated as L1b. On the other hand, for L2a, αa, and θa corresponding to L2, α, and θ that increase as sleepiness increases, values corresponding to μ + 2σ are calculated as L2b, αb, and θb, respectively.

Then, the values of L1b, L2b, αb, and θb calculated in this way are stored in the storage unit 24, and L1a, L2a, αa, and θa stored in the storage unit 24 are deleted.
Then, this process is terminated and the process returns to the control unit main process.
(1.2.3) Driving Data Acquisition Process The processing procedure of the driving data collection process by the control unit 21 will be described below with reference to FIG. This awakening data collection process is executed in S2 of the control part main process.

  In this driving data collection processing, first, facial expression information acquisition processing for detecting information on the face of the driver 3 is performed (S31). Here, m L1, L2, α, θ are acquired in the same manner as in S23 of the awakening data collection process. The facial expression information acquisition process will be described later in detail.

Next, the representative values L1d, L2d, αd, θd are calculated based on L1, L2, α, θ calculated in S31 (S32).
Here, first, all L1, L2, α, and θ stored in the storage unit 24 are read. Then, based on the frequency distribution in each of L1, L2, α, and θ, for L1 whose value decreases as sleepiness increases, a value corresponding to 5% of the cumulative relative frequency from the minimum value is calculated as L1d. On the other hand, for L2, α, and θ that increase as sleepiness increases, a value corresponding to 5% of the cumulative relative frequency from the maximum value is calculated as L2d, αd, and θd.

Then, the values of L1d, L2d, αd, and θd calculated in this way are stored in the storage unit 24, and L1, L2, α, and θ stored in the storage unit 24 are deleted.
Then, this process is terminated and the process returns to the control unit main process.
(1.2.4) Facial Expression Information Acquisition Processing A processing procedure of facial expression information acquisition processing by the control unit 21 will be described below with reference to FIG. This facial expression information acquisition process is executed in S23 in the awakening data collection process and in S31 in the driving data collection process.

In this facial expression information acquisition process, first, a variable j is set to 0 (S41).
Next, image data indicating a face image photographed by the photographing apparatus 10 is acquired from the receiving unit 22 (S42).

Next, based on the image data acquired in S42, position / angle information indicating the position / angle of the components of the face is detected (S43).
Here, as shown in FIG. 8, the left eye (x ₁ , y ₁ , z ₁ ), the right eye (x ₂ , y ₂ , z ₂ ), the left eye corner (x ₃ , y ₃ , z ₃ ), left the inner end of eyebrow _{(x 4, y 4, z} 4), Migimayuatama _{(x 5, y 5, z} 5), the left eyebrow center upper _{(x 6, y 6, z} 6), the left nostril outer end (right nostril opposite ) (X ₇ , y ₇ , z ₇ ), right nostril outer end (end opposite to the left nostril) (x ₈ , y ₈ , z ₈ ), left mouth corner (x ₉ , y _9, z _9), right corner of the mouth _{(x 10, y 10, z} 10), the lower lip lower end _{(x 11, y 11, z} 11), the upper eyelid _{(x 12, y 12, z} 12), lower lid ( x ₁₃ , y ₁₃ , z ₁₃ ), the position of each facial feature point on the upper lip upper end (x ₁₄ , y ₁₄ , z ₁₄ ), the head tilt angle information α and the head tilt angle to the left and right Information θ is detected. Note that α and θ described above are detected as positive values regardless of whether the head is tilted forward or backward or left and right.

  The position / angle information described above is detected from image data representing a face image, but it is assumed that it can be processed using an image taken with a single camera in consideration of practical use, and face feature points are detected from that image. Then, the method of detecting the distance between them is adopted.

  Face feature points are generally detected by detecting the two-dimensional coordinates on the image using shape extraction or pattern matching, but in this embodiment, the distance between feature points can be accurately determined even when the face orientation changes. Since it needs to be detected, the face feature points are detected as three-dimensional coordinates. Specifically, using AAM (Active Appearance Model), the three-dimensional coordinates of the feature points are detected from the face image photographed by one camera, and the distance between the face feature points as the feature amount is calculated from the coordinates. To do. AAM uses a two-dimensional model (see FIG. 9 (a)) having a form and texture information consisting of a large number of feature points, and a three-dimensional model (see FIG. 9 (b)) consisting only of a form. This is a method for detecting the three-dimensional coordinates of the feature points and the angle of inclination of the head by fitting these models.

  Next, based on the detection result in S43, the distance information L1 of the left and right mouth corners and the distance information L2 of the eyebrows and the eyes are calculated by the calculation formula described later, and the storage unit 24 together with α and θ detected in S43. (S44).

Of the information described above, L1 is the distance between the left and right mouth corners, and is calculated by the following equation.
L1 = [(x ₉ −x ₁₀ ) ² + (y ₉ −y ₁₀ ) ² + (z ₉ −z ₁₀ ) ² ] ^0.5
L2 is the distance between the eyebrows and the eyes (the distance between the center of the left eye head and the left eye corner and the upper end of the left eyebrow center), and is calculated by the following equation.

L2 = [(x ₆ − (x ₁ + x ₃ ) / 2) ² + (y ₆ − (y ₁ + y ₃ ) / 2) ² + (z ₆ − (z ₁ + z ₃ ) / 2) ² ] ^0.5
Then, the values of α and θ detected in L1, L2 and S43 calculated in this way are stored in the storage unit 24. A plurality of the above-described values can be stored in the storage unit 24, and each time L1, L2 is calculated in S44, each of L1, L2, α, θ stored in the storage unit 24 is stored. The number will increase one by one.

After S44, the variable j is incremented (j = j + 1) (S45). If the variable j is less than the predetermined number m (S46: YES), the process returns to S42. On the other hand, if the variable j is not less than the predetermined number m (S46: NO), this process ends.
(1.2.5) Drowsiness Determination Processing A processing procedure of drowsiness determination processing by the control unit 21 will be described below with reference to FIG. This sleepiness determination process is executed in S3 of the control unit main process.

In this sleepiness determination process, first, variables L1f, L2f, αf, and θf are set to 0 (S61).
Next, the value of L1b calculated in S27 of the awakening data collection process is compared with the value of L1d calculated in S32 of the driving data collection process (S62).

  At this time, if the value of L1d is equal to or smaller than the value of L1b (S62: YES), that is, if the distance between the left and right mouth corners is smaller than that at awakening, the variable L1f is set to 1 (S63). The process proceeds to S64. On the other hand, if the value of L1d is not less than or equal to the value of L1b (S62: NO), the process proceeds to S64.

Next, the value of L2b calculated in S27 of the awakening data collection process is compared with the value of L2d calculated in S32 of the driving data collection process (S64).
At this time, if the value of L2d is equal to or greater than the value of L2b (S64: YES), that is, if the distance between the eyebrows is larger than that at awakening, the variable L2f is set to 1 (S65), The process proceeds to S66. On the other hand, if the value of L2d is not equal to or greater than the value of L2b (S64: NO), the process proceeds to S66.

Next, the value of αb calculated in S27 of the awake data collection process is compared with the value of αd calculated in S32 of the data collection process during driving (S66).
At this time, if the value of αd is equal to or larger than the value of αb (S66: YES), that is, if the inclination of the head back and forth is larger than that at awakening, the variable αf is set to 1 ( (S67), the process proceeds to S68. On the other hand, if the above condition is not satisfied (S66: NO), the process proceeds to S68 without changing the variable αf.

Next, the value of θb calculated in S27 of the awakening data collection process is compared with the value of θd calculated in S32 of the driving data collection process (S68).
At this time, if the value of θd is equal to or larger than the value of θb (S68: YES), that is, if the inclination of the head to the left and right is larger than that at awakening, the variable θf is set to 1 ( (S69), the process proceeds to S70. On the other hand, if the above condition is not satisfied (S68: NO), the process proceeds to S70 without changing the variable θf.

  Next, if the variables L1f, L2f, αf, and θf are all 0 (S70: YES), it is determined that the sleepiness level is 0 (no sleepiness) (S71), and the process proceeds to S77. On the other hand, if none of the variables L1f, L2f, αf, and θf is 0 (S70: NO), the process proceeds to S72.

  Next, if the variables L2f, αf, and θf are all 0 (S72: YES), it is determined that the sleepiness level is 1 (mild sleepiness) (S73), and the process proceeds to S77. On the other hand, if none of the variables L2f, αf, and θf is 0 (S72: NO), the process proceeds to S74.

  Next, if both the variables αf and θf are 0 (S74: YES), it is determined that the sleepiness level is 2 (medium sleepiness) (S75), and the process proceeds to S77. On the other hand, if neither of the variables αf and θf is 0 (S74: NO), it is determined that the sleepiness level is 3 (intensity sleepiness) (S76).

  Next, based on the determined degree of sleepiness, the output unit 25 is caused to output a signal for causing the alarm device 30, the neck air conditioner 40, the seat belt vibration device 50, and the brake control device 60 to perform an operation (S77). ). Here, when it is determined that the sleepiness level is 0, no operation is performed on any of the above-described apparatuses. When it is determined that the drowsiness level is 1, the alarm device 30 and the neck air conditioner 40 are operated. When it is determined that the sleepiness level is 2, the alarm device 30, the neck air conditioner 40, and the seat belt vibration device 50 are operated. When it is determined that the sleepiness level is 3, all the devices are operated.

Next, L1d, L2d, αd, and θd are deleted from the storage unit 24 (S78). Then, this process is terminated and the process returns to the control unit main process.
(1.3) Effects With the sleepiness determination device 20 according to the first embodiment configured as described above, the degree of sleepiness is determined based on the distance between the left and right mouth corners, the distance between the eyebrows and the eyes, and the head tilt angle. can do. Therefore, eye information such as the opening degree of the eyes is not required for determining the degree of sleepiness. Therefore, because it is not affected by disturbance factors that obstruct the acquisition of eye information, even if the degree of sleepiness cannot be determined based only on eye information such as eye opening, pupil diameter, and line of sight, The degree of sleepiness can be determined.

  Further, with the drowsiness determination device 20 described above, the degree of drowsiness can be determined based on image data indicating a face image captured by the imaging device 10. Therefore, it is possible to easily determine the degree of drowsiness without directly attaching the electrodes for detecting the distance between the left and right mouth corners, the distance between the eyebrows and the eyes, the head tilt angle, etc. to the subject to determine the degree of drowsiness. Can do.

  Further, the distance between the left and right mouth corners changes due to mild sleepiness, the distance between the eyebrows and the eyes changes due to moderate sleepiness, and the head tilt angle changes due to intense sleepiness. Therefore, the drowsiness determination apparatus 20 that determines the degree of drowsiness by such a method can determine the degree of drowsiness in four stages of intense drowsiness, medium drowsiness, mild drowsiness, and no drowsiness.

  Moreover, since the dozing prevention system 1 configured as described above can be mounted on the vehicle 2 and used, the degree of drowsiness can be determined for the driver 3 of the vehicle 2.

Moreover, in the dozing prevention system 1 mentioned above, the drowsiness determination apparatus 20 cooperates with the warning device 30, and performs the screen display and audio | voice output according to the determination result of the sleepiness degree. Further, the neck air conditioner 40, the seat belt vibration device 50, and the brake control device 60 operate according to the determination result of the degree of sleepiness. For this reason, it is possible to appropriately prevent the driver 3 from falling asleep according to the degree of drowsiness, and to prevent an accident caused by a drowsy driving.
(1.4) Correspondence In the embodiment described above, S43 and S44 in FIG. 7 are facial expression information acquisition means in the present invention, the sleepiness determination processing in FIG. 10 is the sleepiness determination means in the present invention, and the photographing apparatus 10 These are the photographing means in the present invention.
[Example 2]
(2.1) Overall Configuration The dozing prevention system 1 according to the second embodiment is basically the same as the first embodiment. However, since the process by the control unit 21 is partially changed, the changed process is shown below.
(2.2) Processing by Control Unit 21 Hereinafter, various processing executed by the control unit 21 included in the drowsiness determination device 20 of the dozing prevention system 1 according to the second embodiment will be described. The second embodiment is different from the first embodiment in the processing methods of the control unit main process, the awakening data collection process, the driving data collection process, and the drowsiness determination process.
(2.2.1) Control Unit Main Process A processing procedure of the control unit main process in the second embodiment will be described below with reference to FIG. This control unit main process is started by pressing a start button provided in the input unit 23.

  In this control unit main process, first, an awakening data collection process 2 for collecting information on the face of the driver 3 at the time of awakening is performed (S101). Here, n pieces of L1a, L2a, αa, and θa are respectively acquired as information on the face at the time of awakening, which is a reference for determining the sleepiness of the driver 3. The awakening data collection process in the second embodiment will be described in detail later.

  Next, driving data collection processing for acquiring information on the face of the driver 3 is performed (S102). Here, n pieces of L1d, L2d, αd, and θd are acquired as information on the face of the driver 3, respectively. Details of the operation data collection processing in the second embodiment will be described later.

Next, sleepiness determination processing for determining the degree of sleepiness of the driver 3 is performed (S103).
Here, a corresponding t-test is performed based on n each of L1a, L2a, αa, and θa calculated in S101 and n each of L1d, L2d, αd, and θd calculated in S102. Based on the test result, the sleepiness based on the distance information L1 on the left and right mouth corners, the distance information L2 on the eyebrows and the eyes, the tilt angle information α before and after the head, and the tilt angle information θ on the left and right of the head. The degree of drowsiness is determined by combining the determination results. Details of the drowsiness determination processing in the second embodiment will be described later.

  Next, it is checked whether or not the end button of the input unit 23 has been pressed (S104). If the end button has not been pressed (S104: NO), the process returns to S102. On the other hand, if the end button has been pressed (S104: YES), this process ends.

Note that this control unit main process also ends when the power unit such as the engine in the vehicle 2 stops and the control of the entire vehicle ends.
(2.2.2) Awakening Data Collection Processing A processing procedure for awakening data collection processing in the second embodiment will be described below with reference to FIG.

  In this awake data collection process, the processes of S121 to S126 are the same as S21 to S26 in the awake data collection process of the first embodiment. If the variable i is not less than the predetermined number n in S126 (S126: NO), the process ends without performing the process corresponding to S27 of the awakening data collection process of the first embodiment, and the control unit main process Return to.

In the first embodiment, L1b, L2b, αb, and θb are stored in the storage unit 24 when the awakening data collection process is completed. In the second embodiment, the storage unit 24 24 is different in that n pieces of L1a, L2a, αa, and θa are stored.
(2.2.3) Operation Data Collection Process Hereinafter, a process procedure of the operation data collection process in the second embodiment will be described with reference to FIG. This operation time data collection process is executed in S102 of the control unit main process.

In this operation data collection process, first, a variable k is set to 0 (S141).
Next, the facial expression information acquisition process described in the first embodiment is performed to calculate m L1, L2, α, θ (S142).

  Next, based on the m L1, L2, α, θ calculated in S142, the representative values L1d, L2d, αd, θd are calculated (S143). Here, the representative value is calculated by performing the same process as S32 of the operation data collection process of the first embodiment.

  The values of L1d, L2d, αd, and θd calculated in this way are stored in the storage unit 24. A plurality of the above-described values can be stored in the storage unit 24. Each time the above-described value is calculated in S143, the number of each value stored in the storage unit 24 increases by one. It will be done.

Thereafter, the variable k is incremented (k = k + 1) (S144). If the variable k is less than the predetermined number n (S145: YES), the process returns to S142. On the other hand, if the variable k is not less than the predetermined number n (S145: NO), this process is terminated and the process returns to the control part main process. At this time, the storage unit 24 stores n L1d, L2d, αd, and θd.
(2.2.4) Drowsiness Determination Processing A procedure of sleepiness determination processing according to the second embodiment will be described below with reference to FIG. This sleepiness determination process is executed in S103 of the control unit main process.

In this drowsiness determination process, variables L1f, L2f, αf, and θf are first set to 0 (S161).
Next, a corresponding t-test is performed based on the n values of L1a calculated in S124 of the awake data collection process and the n values of L1d calculated in S143 of the driving data collection process. (S162). In this sleepiness determination process, the significance level of the corresponding t-test is 5%.

  At this time, if the average of L1d is significantly smaller than the average of L1a (S162: YES), that is, if the distance between the left and right mouth corners is smaller than that at awakening, the variable L1f is set to 1. (S163), the process proceeds to S164. On the other hand, if the average of L1d is not significantly smaller than the average of L1a (S162: NO), the process proceeds to S164.

  Next, a corresponding t-test is performed based on n L2a values calculated in S124 of the awakening data collection process and n L2d values calculated in S143 of the driving data collection process. (S164).

  At this time, if the average of L2d is significantly larger than the average of L2a (S164: YES), that is, if the distance between eyebrows is larger than that at awakening, the variable L2f is set to 1 ( In step S165, the process proceeds to step S166. On the other hand, if the average of L1d is not significantly larger than the average of L1a (S164: NO), the process proceeds to S166.

  Next, a corresponding t-test is performed based on n values of αa calculated in S124 of the awake data collection process and n values of αd calculated in S143 of the data collection process during driving. (S166).

  At this time, if the average of αd is significantly larger than the average of αa (S166: YES), that is, if the inclination of the head back and forth is larger than that at awakening, the variable αf is set to 1. The setting is made (S167), and the process proceeds to S168. On the other hand, if the above condition is not satisfied (S166: NO), the process proceeds to S168 without changing the variable αf.

  Next, a corresponding t-test is performed based on the n values of θa calculated in S124 of the awake data collection process and the n values of θd calculated in S143 of the driving data collection process. (S168).

  At this time, if the average of θd is significantly larger than the average of θa (S168: YES), that is, if the inclination of the head to the left and right is larger than that at awakening, the variable θf is set to 1. The setting is made (S169), and the process proceeds to S170. On the other hand, if the above condition is not satisfied (S168: NO), the process proceeds to S170 without changing the variable θf.

  Next, if the variables L1f, L2f, αf, and θf are all 0 (S170: YES), it is determined that the sleepiness level is 0 (no sleepiness) (S171), and the process proceeds to S177. On the other hand, if none of the variables L1f, L2f, αf, and θf is 0 (S170: NO), the process proceeds to S172.

  Next, if the variables L2f, αf, and θf are all 0 (S172: YES), it is determined that the sleepiness level is 1 (mild sleepiness) (S173), and the process proceeds to S177. On the other hand, if none of the variables L2f, αf, and θf is 0 (S172: NO), the process proceeds to S174.

  Next, if the variables αf and θf are both 0 (S174: YES), it is determined that the sleepiness level is 2 (medium sleepiness) (S175), and the process proceeds to S177. On the other hand, if neither of the variables αf and θf is 0 (S174: NO), it is determined that the sleepiness level is 3 (intensity sleepiness) (S176).

  Next, based on the determined degree of sleepiness, the output unit 25 is caused to output a signal for causing the alarm device 30, the neck air conditioner 40, the seat belt vibration device 50, and the brake control device 60 to perform an operation (S177). ). Here, the same process as S77 of the sleepiness determination process of the first embodiment is performed.

Next, L1d, L2d, αd, and θd are deleted from the storage unit 24 (S178). Then, this process is terminated and the process returns to the control unit main process.
(2.3) Effects Also in the drowsiness determination device 20 or the dozing prevention system 1 described in the second embodiment, the same effects as in the first embodiment can be obtained. Furthermore, in the second embodiment, in the sleepiness determination process, the presence / absence of sleepiness (S170, S172, S174) is determined by a corresponding t-test, so that the determination result is more reliable. Therefore, erroneous determination can be reduced as compared with the first embodiment described above.
(2.4) Correspondence In the embodiment described above, S43 and S44 in FIG. 7 are facial expression information acquisition means in the present invention, and the sleepiness determination processing in FIG. 14 is the sleepiness determination means in the present invention. These are the photographing means in the present invention.
[Example 3]
(3.1) Overall Configuration The dozing prevention system 1 according to the third embodiment is basically the same as the first embodiment. However, since the process by the control unit 21 is different, the process will be described below.
(3.2) Processing by Control Unit 21 Hereinafter, various types of processing executed by the control unit 21 included in the drowsiness determination device 20 of the dozing prevention system 1 according to the third embodiment will be described.
(3.2.1) Control Unit Main Process A processing procedure for the control unit main process in the third embodiment will be described below with reference to FIG. This control unit main process is started by pressing a start button provided in the input unit 23.

  In this control unit main processing, first, facial expression information acquisition processing is performed (S201). Here, based on the face image of the driver 3 photographed by the photographing apparatus 10, the distance between the left and right mouth corners (L1), the distance between the eyebrows and the eyes (L2), and the midpoint between the middle of the eyebrow and the middle of the eye as facial expression information. Distance (L3), distance between eyebrows (L4), distance between midpoint of lower nose and midpoint of eyes (L5), distance between midpoint of mouth end and midpoint of eyes (L6), lower lip The distance between the lower end and the middle point of the eye (L7), the distance between the upper eyelid and the lower eyelid (L8), the distance between the left and right sides of the nose (L9), the distance between the upper lip upper end and the middle point of the eye (L10), The facial expression information of the tilt angle (α) of the head to the front and rear and the tilt angle (θ) of the head to the left and right is calculated and stored in the storage unit 24. The facial expression information acquisition process will be described later in detail. The facial expression information calculated in S201 is accumulated in the storage unit 24 every time S201 is executed.

  Next, it is checked whether or not a specified time has passed (S202). If a specified time (for example, 5 minutes) has not elapsed (S202: NO), the process returns to S201 after waiting for a predetermined time (for example, 5 seconds). On the other hand, if the specified time has elapsed (S202: YES), the process proceeds to S203.

  Next, awakening information that is a reference for determining sleepiness is calculated (S203). Here, all the facial expression information stored in the storage unit 24 a plurality of times in S201 is read out. Then, a normal distribution is created based on the read facial expression information. At this time, since the values of L1, L8, and L9 become smaller as sleepiness increases, the value of the awakening information that serves as a reference is set to be smaller. Therefore, for L1, L8, and L9, when the average value is μ and the standard deviation is σ, the value corresponding to μ−2σ is stored in the storage unit 24 as the awakening information of L1, L8, and L9. On the other hand, for L2 to L7, L10, α, and θ, values corresponding to μ + 2σ in the normal distribution created by each are stored in the storage unit 24 as information on awakening of each of L2 to L7, L10, α, and θ.

Next, facial expression information acquisition processing is performed (S204). Here, each facial expression information is calculated and stored in the storage unit 24 as in S201 described above.
Next, sleepiness determination processing for determining the degree of sleepiness of the driver 3 is performed (S205). Here, the awakening information calculated in S203 and the facial expression information calculated in S204 are read from the storage unit 24 and compared to determine the degree of sleepiness. This drowsiness determination process will be described in detail later.

Thereafter, the facial expression information calculated in S204 is deleted from the storage unit 24, and the process returns to S204.
This process ends when the end button of the input unit 23 is pressed and when the power unit such as the engine in the vehicle 2 stops and the control of the entire vehicle ends.
(3.2.2) Facial Expression Information Acquisition Processing Hereinafter, the processing procedure of facial expression information acquisition processing according to the third embodiment will be described with reference to FIG. This facial expression information acquisition processing is executed in S201 and S204 in the control unit main processing.

In this facial expression information acquisition process, first, image data indicating a face image photographed by the photographing apparatus 10 is obtained from the receiving unit 22 (S221).
Next, based on the image data acquired in S221, position / angle information indicating the position / angle of the facial component is detected (S222). Here, similarly to S43 in the facial expression information acquisition process of the first embodiment, the left eye (x ₁ , y ₁ , z ₁ ), the right eye (x ₂ , y ₂ , z ₂ ), the left eye corner, using AAM. _{(x 3, y 3, z} 3), Hidarimayuatama _{(x 4, y 4, z} 4), Migimayuatama _{(x 5, y 5, z} 5), the left eyebrow center upper _{(x 6, y 6, z} 6 ), Left nostril outer end (end located on the side opposite to the right nostril) (x ₇ , y ₇ , z ₇ ), right nostril outer end (end located on the side opposite to the left nostril) (x ₈ , y ₈ , z ₈ ), left mouth corner (x ₉ , y ₉ , z ₉ ), right mouth corner (x ₁₀ , y ₁₀ , z ₁₀ ), lower lip lower end (x ₁₁ , y ₁₁ , z ₁₁ ), upper eyelid (x ₁₂ , y ₁₂ , z ₁₂ ), lower eyelids (x ₁₃ , y ₁₃ , z ₁₃ ), upper lip upper end (x ₁₄ , y ₁₄ , z ₁₄ ), and tilt angle information α before and after the head and Tilt angle information θ to the left and right of the head is detected. Note that α and θ described above are detected as positive values regardless of whether the head is tilted forward or backward or left and right.

Next, the facial expression information L1 to L10 described above are calculated based on the detection result in S222 (S223).
In the facial expression information described above, L1 is the distance between the left and right mouth corners (the distance between the left mouth corner and the right mouth corner), and is calculated by the following formula based on the left mouth corner and the right mouth corner.

L1 = [(x ₉ −x ₁₀ ) ² + (y ₉ −y ₁₀ ) ² + (z ₉ −z ₁₀ ) ² ] ^0.5
L2 is the distance between the eyebrows and the eyes (the distance between the center of the left eye head and the left eye corner and the upper end of the left eyebrow center), and is calculated by the following equation.

L2 = [(x ₆ − (x ₁ + x ₃ ) / 2) ² + (y ₆ − (y ₁ + y ₃ ) / 2) ² + (z ₆ − (z ₁ + z ₃ ) / 2) ² ] ^0.5
L3 is a distance between the midpoint of the eyebrow and the midpoint of the eye (the distance between the center of the left eyebrow and the right eyebrow and the center of the left eye and the right eye), and is calculated by the following equation. .

L3 = [(((x ₄ + x ₅ ) / 2− (x ₁ + x ₂ ) / 2) ² + ((y ₄ + y ₅ ) / 2− (y ₁ + y ₂ ) / 2) ² + ((z ₄ + z ₅ ) / 2- (z ₁ + z ₂ ) / 2) ² ] ^0.5
L4 is the distance between the eyebrows (the distance between the left eyebrow and the right eyebrow), and is calculated by the following equation.

L4 = [(x ₄ −x ₅ ) ² + (y ₄ −y ₅ ) ² + (z ₄ −z ₅ ) ² ] ^0.5
L5 is the distance between the midpoint of the nose lower end and the midpoint of the eye (the distance between the left nostril outer end and the right nostril outer end center, and the left and right eye center). Calculated by the formula.

L5 = [((x ₇ + x ₈ ) / 2− (x ₁ + x ₂ ) / 2) ² + ((y ₇ + y ₈ ) / 2− (y ₁ + y ₂ ) / 2) ² + ((z ₇ + z ₈ ) / 2- (z ₁ + z ₂ ) / 2) ² ] ^0.5
L6 is the distance between the midpoint of the mouth edge and the midpoint of the eye (the distance between the center of the left and right mouth corners and the center of the left and right eye corners), and is calculated by the following equation. The

L6 = [((x ₉ + x ₁₀ ) / 2− (x ₁ + x ₂ ) / 2) ² + ((y ₉ + y ₁₀ ) / 2− (y ₁ + y ₂ ) / 2) ² + ((z ₉ + z ₁₀ ) / 2- (z ₁ + z ₂ ) / 2) ² ] ^0.5
L7 is the distance between the lower lip lower end and the middle point of the eye (the distance between the lower lip lower end and the center of the left and right eyes), and is calculated by the following equation.

_{L7 = [(x 11 - (} x 1 + x 2) / 2) 2 + (y 11 - (y 1 + y 2) / 2) 2 + (z 11 - (z 1 + z 2) / 2) 2] 0.5
L8 is the distance between the upper eyelid and the lower eyelid, and is calculated by the following equation.

_{L8 = [(x 12 -x 13} ) 2 + (y 12 -y 13) 2 + (z 12 -z 13) 2] 0.5
L9 is the distance between the left and right ends of the nose (the distance between the left nostril outer end and the right nostril outer end), and is calculated by the following equation.

_{L9 = [(x 7 -x 8} ) 2 + (y 7 -y 8) 2 + (z 7 -z 8) 2] 0.5
Instead of the distance between the outer ends of the nostrils, the distance between the outer ends of the left and right nose wings may be used as L9. In that case, it is preferable that the position of the outer end of each of the right and left nose wings is detected in S222 described above.

L10 is the distance between the upper lip upper end and the middle point of the eye (the distance between the upper lip upper end and the center of the left and right eyes), and is calculated by the following equation.
_{L10 = [(x 14 - (} x 1 + x 2) / 2) 2 + (y 14 - (y 1 + y 2) / 2) 2 + (z 14 - (z 1 + z 2) / 2) 2] 0.5
The values of α and θ detected in L1 to L10 and S223 calculated in this way are stored in the storage unit 24. Then, this process is terminated and the process returns to the control unit main process.
(3.2.3) Drowsiness Determination Processing A processing procedure of sleepiness determination processing according to the third embodiment will be described below with reference to FIG. This drowsiness determination process is executed in S205 of the control unit main process.

  When this drowsiness determination process is started, it is first determined whether or not there is drowsiness in each facial expression information (S241). Here, the awakening information of each of L1 to L10, α, and θ calculated in S203 of the control unit main process is compared with each of the facial expression information L1 to L10, α, and θ calculated in S204 to determine the presence or absence of drowsiness. To do.

  Table 1 shows the correspondence between facial expression information and awakening information and the presence or absence of sleepiness.

The presence or absence of drowsiness is determined according to the correspondence relationship in Table 1.

Next, it is checked whether the number of facial expression information determined to be drowsy among the facial expression information determined in S241 is 7 or more (S242).
If the number of facial expression information determined to be drowsy is 7 or more (S242: YES), the drowsiness level is determined to be 3 (intensity drowsiness) (S243), and the process proceeds to S249. On the other hand, if it is not 7 or more (S242: NO), the process proceeds to S244.

Next, it is checked whether the number of facial expression information determined to be drowsy is 5 or more (S244).
If the number of facial expression information determined to be drowsy is 5 or more (S244: YES), the drowsiness level is determined to be 2 (medium drowsiness) (S245), and the process proceeds to S249. On the other hand, if it is not 5 or more (S244: NO), the process proceeds to S246.

  Next, it is checked whether the number of facial expression information determined to be drowsy is 3 or more (S246). If the number of facial expression information determined to be drowsy is 3 or more (S246: YES), the drowsiness level is determined to be 1 (mild drowsiness) (S247). On the other hand, if it is not 3 or more (S246: NO), it is determined that the sleepiness level is 0 (no sleepiness) (S248). Thereafter, the process proceeds to S249.

Next, based on the determined degree of sleepiness, the output unit 25 is caused to output a signal for causing the alarm device 30, the neck air conditioner 40, the seat belt vibration device 50, and the brake control device 60 to perform an operation (S249). ). Here, the same process as S77 of the sleepiness determination process of the first embodiment is performed. Then, this process is terminated and the process returns to the control unit main process.
(3.3) Effect Even in the dozing prevention system 1 described in the third embodiment, similarly to the first and second embodiments, even if the degree of sleepiness cannot be determined based on only eye information, The degree of drowsiness can be easily determined without directly attaching an electrode or the like for detecting the facial expression movement to the subject who determines the degree of drowsiness.

Furthermore, in the present embodiment, drowsiness is determined based on a large amount of facial expression information as compared with the first and second embodiments, so that the determination result is more reliable.
(3.4) Correspondence In the embodiment described above, S222 and S223 in FIG. 16 are facial expression information acquisition means in the present invention, and the sleepiness determination processing in FIG. 17 is the sleepiness determination means in the present invention. These are the photographing means in the present invention.
[Example 4]
(4.1) Overall Configuration The dozing prevention system 1 in the fourth embodiment is basically the same configuration as in the third embodiment, and the control unit main process and facial expression information acquisition process are the same processes as in the third embodiment. Instead of the sleepiness determination processing in the third embodiment, the sleepiness determination processing described below is executed.
(4.2) Sleepiness Determination Processing The sleepiness determination processing executed by the control unit 21 of the dozing prevention system 1 in the fourth embodiment will be described below with reference to FIG. This process is executed in S205 of the control unit main process of the third embodiment.

  In this sleepiness determination process, first, it is determined whether or not the face inclination is larger than the awakening information (S261). Specifically, the awakening information of α and θ calculated in S203 of the control unit main process is compared with the facial expression information α and θ calculated in S204, respectively, and either of the facial expression information α and θ is awakening. If it is larger than the information (S261: YES), the sleepiness level is determined to be 3 (strong sleepiness) (S262), and the process proceeds to S268. On the other hand, if neither the facial expression information α or θ is larger than the awakening information (S261: NO), the process proceeds to S263.

  Next, it is determined whether or not the distance between the eyebrows and the eyes is larger than the awakening information (S263). Specifically, the L2 awakening information calculated in S203 of the control unit main process is compared with the facial expression information L2 calculated in S204, and if the facial expression information L2 is larger than the awakening information (S263: YES), sleepiness The level is determined to be 2 (medium sleepiness) (S264), and the process proceeds to S268. On the other hand, if the facial expression information L2 is not larger than the awakening information (S263: NO), the process proceeds to S265.

  Next, it is determined whether the distance between the mouth corners is smaller than the awakening information (S265). Specifically, the awakening information of L1 calculated in S203 of the control unit main process is compared with the facial expression information L1 calculated in S204, and if the facial expression information L1 is smaller than the awakening information (S265: YES), sleepiness The level is determined to be 1 (mild drowsiness) (S266). On the other hand, if the facial expression information L1 is not smaller than the awakening information (S265: NO), the sleepiness level is determined to be 0 (no sleepiness) (S267). Thereafter, the process proceeds to S268.

Next, based on the determined degree of sleepiness, the output unit 25 is caused to output a signal that causes the alarm device 30, the neck air conditioner 40, the seat belt vibration device 50, and the brake control device 60 to perform an operation (S268). ). Here, the same process as S77 of the sleepiness determination process of the first embodiment is performed. Then, this process is terminated and the process returns to the control unit main process.
(4.3) Effects Even in the dozing prevention system 1 described in the fourth embodiment, as in the first to third embodiments, even if the degree of sleepiness cannot be determined based on only eye information, The degree of drowsiness can be easily determined without directly attaching an electrode or the like for detecting the movement of facial expression to the subject who determines the degree of drowsiness.
[Modification]
As mentioned above, although the Example of this invention was described, this invention is not limited to each said Example at all, and as long as it belongs to the technical scope of this invention, it cannot be overemphasized that various forms can be taken. .

  For example, in each of the above-described embodiments, the configuration for acquiring the facial motion based on the image data indicating the facial image captured by the imaging device 10 has been illustrated. However, the specific configuration is not particularly limited as long as the facial motion can be acquired.

For example, by attaching electrodes to the face and detecting the weak electricity generated in proportion to the magnitude of the force when the facial muscles move, acquire the loose mouth, eyebrows, head tilt, etc. Also good.
Moreover, in each said Example, although the process by the drowsiness determination apparatus 20 illustrated the structure started by pressing the start button with which the input part 23 was equipped, the structure started by satisfying other conditions It is also good.

  For example, a configuration in which processing is started at the same time when a power unit such as an engine in the vehicle 2 is activated, or a vehicle speed sensor that measures the traveling speed of the vehicle is provided in the vehicle, and the vehicle speed sensor has a predetermined speed (for example, A configuration is conceivable in which the processing is started when it is detected that the speed exceeds 30 km / h).

  Further, in each of the above-described embodiments, the configuration in which both the front and rear head tilt angle information α and the left and right head tilt angle information θ are acquired as information on the head tilt is illustrated. However, it may be configured to acquire only one of them.

  In each of the above embodiments, the distance between the left and right mouth corners, the distance between the eyebrows and the eyes, the head tilt angle, the distance between the midpoint of the eyebrows and the midpoint of the eyes, the distance between the eyebrows, the midpoint of the bottom of the nose, The distance between the middle point of the eye, the distance between the middle point of the mouth and the middle point of the eye, the distance between the lower lip and the middle point of the eye, the distance between the upper eyelid and the lower eyelid, the distance between the left and right sides of the nose The configuration of determining the degree of sleepiness based on the expression information of any one of the distance between the upper lip upper end and the middle point of the eye is illustrated. However, it is also possible to detect gaze and the like, detect the presence or absence of drowsiness based on them, and determine the degree of drowsiness by combining the detection result and the information described above.

  Further, in the third embodiment, as described in the sleepiness determination process, the presence or absence of sleepiness is determined based on each of the 12 facial expression information, and the degree of sleepiness is determined according to the number of facial expression information determined to be sleepy. The structure to determine was illustrated. However, the facial expression information for determining the presence or absence of sleepiness is not limited to 12, and the number serving as the threshold value of the sleepiness level is not limited to 7, 5, and 3.

  The drowsiness determination process described above may be configured to determine not the presence or absence of drowsiness but the degree of drowsiness based on each facial expression information. In this case, the degree of sleepiness may be determined by an average of the degree of sleepiness, or may be determined by a mode value of the degree of sleepiness. It should be noted that the degree of sleepiness may be determined based on how far away from the awakening information.

Further, in the fourth embodiment, the configuration in which the degree of sleepiness is determined based on the head tilt angle (face tilt), the distance between the eyebrows and the eyes, and the distance between the left and right mouth corners, but other facial expression information is used. The degree of sleepiness may be determined. The comparison between the awakening information and the facial expression information may be a comparison by significance determination using a statistical test.
[Study to confirm the tendency of sleepy facial expression features to the degree of sleepiness]
Below, the test for confirming the tendency with respect to the degree of sleepiness of the sleepiness facial expression feature mentioned above is explained.

  In this test, first, the changes in the face from the awake state to the doze state were continuously photographed. The evaluator classifies the subject's sleepiness level into levels 0 to 5 by sensory evaluation by looking at the captured video. At each sleepiness level, the distance between the left and right mouth corners, the distance between the eyebrows, and the front and rear heads. Head tilt, left and right head tilt, distance between midpoint of eyebrow and midpoint of eye, distance between browhead, distance between midpoint of lower nose and midpoint of eye, midpoint of mouth and inside of eyes The distance between the points, the distance between the lower lip and the middle point of the eye, the distance between the upper eyelid and the lower eyelid, the distance between the outer edge of the left nostril and the outer edge of the right nostril, and the distance between the upper edge of the upper lip and the middle point of the eye Measurement was performed about 10 times.

  The level of sleepiness is level 0 when it seems to be totally sleepless, level 1 when it is somewhat sleepy, level 2 when it is sleepy, level 3 when it is quite sleepy, level 4 when it is unbearably sleepy ， Sleeping state was set to level 5.

  19A to 19D are graphs showing measurement results of the distance between the left and right mouth corners, the distance between the eyebrows, the front and rear head tilt, and the left and right head tilt among the above-described items. Each graph shows the relative distance when the horizontal axis represents drowsiness level and the vertical axis represents 10 between the left and right eyes.

  The part surrounded by a thick frame is a part where, in each measurement result, a corresponding t-test is performed on the measurement result of drowsiness level 0 and the measurement result of other levels, and it is determined that there is a significant difference. is there. The distance between mouth corners and head inclination were tested at a significance level of 5%, and the distance between eyebrows was tested at a significance level of 1%.

  As can be seen from FIG. 19, the distance between the corners of the mouth is significantly different from drowsiness level 1, the distance between the eyebrows is significantly different from drowsiness level 3, and the head tilt is significant from drowsiness level 5. A difference appeared.

  As described above, the degree of drowsiness in which changes occur in the mouth looseness, the eyebrows, and the head tilt are different. Therefore, by combining these facial expression features, it is possible to determine the degree of sleepiness separately in more stages than when they are used alone.

  Also, the distance between the middle point of the eyebrow and the middle point of the eye, the distance between the eyebrows, the distance between the middle point of the bottom of the nose and the middle point of the eye, the distance between the middle point of the mouth and the middle point of the eye, the lower lip FIG. 20 is a graph showing the measurement results of the distance between the lower end and the middle point of the eye, the distance between the upper eyelid and the lower eyelid, the distance between the outer edge of the left nostril and the outer edge of the right nostril, and the distance between the upper edge of the upper lip and the middle point of the eye. To FIG. 20 to 24 and 27, each distance increases as sleepiness increases, and FIGS. 25 and 26 illustrate that the distance decreases as sleepiness increases. Therefore, it can be seen that the degree of drowsiness can be determined by the method as in each of the embodiments described above.

Illustration showing facial muscles Side view of the dozing prevention system Block diagram showing the drowsiness determination device The flowchart which shows the procedure of the control part main process of Example 1. The flowchart which shows the procedure of the data collection process at the time of awakening of Example 1. The flowchart which shows the procedure of the data collection processing at the time of Example 1 The flowchart which shows the procedure of the facial expression information acquisition process of Example 1. Diagram showing facial feature points Diagram showing 2D model and 3D model of AAM The flowchart which shows the procedure of the drowsiness determination process of Example 1. The flowchart which shows the procedure of the control part main process of Example 2. FIG. The flowchart which shows the procedure of the data collection process at the time of awakening of Example 2. The flowchart which shows the procedure of the operation time data collection process of Example 2. The flowchart which shows the procedure of the sleepiness determination process of Example 2. The flowchart which shows the procedure of the control part main process of Example 3. The flowchart which shows the procedure of the facial expression information acquisition process of Example 3. The flowchart which shows the procedure of the drowsiness determination processing of Example 3 The flowchart which shows the procedure of the sleepiness determination processing of Example 4 Results of a test to confirm the tendency of sleepiness facial features to the degree of sleepiness The relationship between the middle point of the eyebrow and the middle point of the eye and the sleepiness level Relationship between distance between eyebrows and sleepiness level The relationship between the midpoint of the lower nose and the midpoint of the eyes and the sleepiness level Relationship between midpoint of mouth edge and midpoint of eyes and drowsiness level The relationship between the distance between the lower lip lower edge and the middle point of the eye and the sleepiness level Relationship between distance between upper and lower eyelids and sleepiness level Relationship between distance between left and right nostril outer edges and right nostril outer edge and drowsiness level The relationship between the distance between the upper lip upper edge and the midpoint of the eye and the sleepiness level

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 ... Dozing prevention system, 2 ... Vehicle, 3 ... Driver, 10 ... Imaging device, 20 ... Drowsiness determination device, 21 ... Control part, 22 ... Reception part, 23 ... Input part, 24 ... Memory | storage part, 25 ... Output part , 30 ... alarm device, 40 ... neck air conditioner, 50 ... seat belt vibration device, 60 ... brake control device.

Claims (7)

Facial expression information acquisition means for acquiring at least two of facial expression information A to K;
Acquired by the facial expression information acquisition means by comparing the facial expression information acquired at the time of awakening by the facial expression information acquisition means and the facial expression information acquired at a timing when sleepiness should be determined by the facial expression information acquisition means about each of the facial expression information, to determine the presence or absence of drowsiness per the facial expression information, as the ratio of the number of the facial expression information it is determined that there drowsiness on the number of the facial expression information the decision is large, drowsiness Drowsiness judging means for judging that the degree is strong ,
The criteria for determining the presence or absence of sleepiness in the facial expression information A to K is that the facial expression information acquired at the time of awakening is the value of the facial expression information acquired at the timing when sleepiness should be determined. On the other hand, when the expression information B, C, D, E, F, G, H, and K is determined to be drowsiness, the value of the expression information acquired at the timing when sleepiness should be determined is determined. A drowsiness determination device characterized by determining drowsiness when the value of the facial expression information acquired at awakening is greater.
(A) Distance between left and right mouth corners (B) Distance between eyebrows and eyes (C) Head tilt angle (D) Distance between midpoint of eyebrows and midpoint of eyes (E) Distance between eyebrows (F) Lower nose Distance between the middle point of the eye and the middle point of the eye (G) Distance between the middle point of the mouth edge and the middle point of the eye (H) Distance between the lower edge of the lower lip and the middle point of the eye (I) Between the upper eyelid and the lower eyelid Distance (J) Distance between the right and left ends of the nose (K) Distance between the upper lip upper end and the middle point of the eye Facial expression information acquisition means for acquiring at least two of facial expression information A to K;
Acquired by the facial expression information acquisition means by comparing the facial expression information acquired at the time of awakening by the facial expression information acquisition means and the facial expression information acquired at a timing when sleepiness should be determined by the facial expression information acquisition means Determining the degree of sleepiness corresponding to each facial expression information based on the determined facial expression information, and indicating the degree of sleepiness in the determined facial expression information. Drowsiness determining means for determining the degree of drowsiness based on an average value or a mode value ,
For the expression information A, I, and J, the degree of sleepiness is determined when the value of the expression information acquired at the timing when sleepiness should be determined is smaller than the value of the expression information acquired at awakening. While there is drowsiness, the degree of drowsiness is determined by the magnitude of the difference between these values, while the facial expression information B, C, D, E, F, G, H, K is acquired at the timing when drowsiness should be determined. When the value of the facial expression information is greater than the value of the facial expression information acquired at awakening, it is assumed that there is sleepiness, and the degree of sleepiness is determined based on the magnitude of the difference between the values. .
(A) Distance between left and right mouth corners (B) Distance between eyebrows and eyes (C) Head tilt angle (D) Distance between midpoint of eyebrows and midpoint of eyes (E) Distance between eyebrows (F) Lower nose Distance between the middle point of the eye and the middle point of the eye (G) Distance between the middle point of the mouth edge and the middle point of the eye (H) Distance between the lower edge of the lower lip and the middle point of the eye (I) Between the upper eyelid and the lower eyelid Distance (J) Distance between the right and left ends of the nose (K) Distance between the upper lip upper end and the middle point of the eye   Facial expression information acquisition means for acquiring at least two facial expression information out of the facial expression information of the distance between the left and right mouth corners, the distance between the eyebrows and the eyes, and the head tilt angle;
  Drowsiness determining means for determining the degree of sleepiness based on each of the facial expression information acquired by the facial expression information acquiring means,
  The drowsiness determining means includes
  The condition A is determined to be drowsy based on the head tilt angle, the condition B is determined to be drowsy based on the distance between the eyebrows and eyes, and the person is drowsy based on the distance between the left and right mouth corners. Is determined to be Condition C,
  When the condition A is satisfied, the degree of sleepiness is determined as intense sleepiness,
  When the condition B is satisfied and the condition A is not satisfied, the degree of sleepiness is determined as medium sleepiness having a lower sleepiness level than the intense sleepiness,
  When the condition C is satisfied and the condition A and the condition B are not satisfied, the degree of drowsiness is determined as mild drowsiness having a lower degree of drowsiness than the moderate drowsiness,
  When none of the conditions A, B, and C is satisfied, it is determined that there is no sleepiness
  A drowsiness determination device characterized by the above.   The drowsiness determining means determines the presence or absence of drowsiness by comparing the facial expression information acquired at the time of awakening and the facial expression information acquired at a timing when sleepiness should be determined. Based on the degree of sleepiness
  The drowsiness determination device according to claim 3.   The facial expression information acquisition means detects position / angle information indicating at least the position of a component of the face or the tilt angle of the head based on image data indicating a face image, and based on the position / angle information, Get facial expression information
  The drowsiness determination device according to any one of claims 1 to 4, wherein   It is used by being mounted on a vehicle,
  The facial expression information acquisition means detects the position / angle information based on the image data photographed by photographing means provided in the vehicle.
  The drowsiness determination device according to claim 5.   The program for making a computer system perform the process sequence for functioning as a facial expression information acquisition means and sleepiness determination means in any one of Claim 1 to 6.