WO2023095229A1 - Wakefulness estimation device and wakefulness estimation method - Google Patents

Abstract

The present invention comprises: a facial determination information acquisition unit (11) that acquires facial determination information indicating whether a person is making a face with a furrowed brow; a luminance detection unit (12) that detects the luminance in a captured image capturing a range in which the face of the person should appear; and an estimation unit (13) that estimates the level of wakefulness of the person on the basis of the facial determination information acquired by the facial determination information acquisition unit (11) and the luminance detected by the luminance detection unit (12).

Description

Arousal level estimation device and arousal level estimation method

The present disclosure relates to an arousal level estimation device and an arousal level estimation method.

When people become drowsy and their arousal level decreases, they may wrinkle their eyebrows. Therefore, conventionally, the distance between feature points representing the inner corners of both eyes and the inner corners of the eyebrows detected based on an image obtained by imaging a person's face is equal to or less than a predetermined threshold. There is known a technique for detecting a state of increased sleepiness and estimating a person's drowsiness state from the frequency of detection of the state (for example, Patent Document 1).

Japanese Patent Application Laid-Open No. 2008-212298

A person's face with wrinkled brows is not limited to the case of sleepiness and decreased alertness. People may also wrinkle their eyebrows when they feel glare. Therefore, it is difficult to distinguish whether the person's arousal level has decreased or whether the person feels dazzling only from the expression that the person has wrinkled the eyebrows.
In the conventional technology represented by the technology disclosed in Patent Document 1, it is difficult to distinguish whether the person's arousal level has decreased or the person feels dazzling only from the expression that the person has wrinkled the eyebrows. , there is a possibility of erroneously inferring that a person's arousal has decreased.

The present disclosure has been made in order to solve the above problems, and when a person is in a state of wrinkling their eyebrows, it is assumed that the surrounding situation is a situation in which the person feels dazzling. It is an object of the present invention to provide an arousal level estimation device that prevents erroneous estimation of a decrease in a person's arousal level by estimating a person's arousal level by taking into consideration.

The arousal level estimation device according to the present disclosure includes a face determination information acquisition unit that acquires face determination information indicating whether or not a person has a face with wrinkles between the eyebrows, and a range in which the person's face should be captured. and an estimation unit for estimating the degree of arousal of a person based on the face determination information acquired by the face determination information acquisition unit and the brightness detected by the brightness detection unit. It is a thing.

According to the present disclosure, by estimating the degree of arousal of a person in consideration of whether the surrounding situation is a situation where the person is assumed to feel dazzling when the person is in a state of wrinkling the eyebrows. , it is possible to prevent misestimation of a person's arousal reduction.

1 is a diagram showing a configuration example of an arousal level estimation device according to Embodiment 1; FIG. 4 is a flowchart for explaining the operation of the wakefulness estimation device according to Embodiment 1; FIG. 3 is a flowchart for explaining details of a patient face determination process by a face determination unit in step ST5 of FIG. 2; FIG. FIG. 3 is a flowchart for explaining the details of arousal level estimation processing by an estimation unit in step ST8 of FIG. 2; FIG. 4 is a flow chart for explaining the calculation operation of the distance between reference feature points by the reference distance calculation unit in Embodiment 1. FIG. 6A and 6B are diagrams showing an example of the hardware configuration of the wakefulness estimation device according to Embodiment 1. FIG. 1 is a diagram showing a configuration example of an arousal level estimation device provided with a personal authentication function in Embodiment 1. FIG. 4 is a flowchart for explaining the operation of the awakening level estimation device 1 having a personal authentication function in Embodiment 1. FIG.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings.
Embodiment 1.
The arousal level estimation device according to Embodiment 1 estimates a person's arousal level based on a captured image in which at least the face of the person is captured. In particular, when a person makes a face with wrinkles between the eyebrows, the arousal level estimating device detects whether the face is due to a decrease in the person's arousal level or because the person feels that the person is dazzled. It distinguishes whether it is a face or not, and estimates the degree of arousal of a person.
In Embodiment 1, a face with wrinkles between the eyebrows is referred to as a "bearing face". A "bearing face" is a so-called frowning face with wrinkled eyebrows and narrowed eyes.
Further, in Embodiment 1, as an example, the person whose awakening level is estimated by the awakening level estimation device is the driver of the vehicle.
The result of estimating whether the driver's wakefulness has decreased, in other words, whether the driver is drowsy, estimated by the wakefulness estimating device, is used, for example, to detect dozing off of the driver.

FIG. 1 is a diagram showing a configuration example of an arousal level estimation device 1 according to Embodiment 1. As shown in FIG.
Arousal level estimation device 1 includes face determination device 2 , face determination information acquisition unit 11 , luminance detection unit 12 , and estimation unit 13 .
Arousal level estimation device 1 is connected to imaging device 3 . The imaging device 3 is mounted on a vehicle (not shown) and installed so as to be capable of imaging at least the range in which the driver's face should exist. For example, the imaging device 3 may be shared with a so-called DMS (Driver Monitoring System) installed for the purpose of monitoring the interior of the vehicle. The imaging device 3 is a visible light camera or an infrared camera. The imaging device 3 outputs the captured image to the face determination device 2 included in the wakefulness estimation device 1 .

The face determination device 2 acquires the captured image and determines whether or not the driver is making a patient face based on the acquired captured image. The face determination device 2 outputs information (hereinafter referred to as “face determination information”) indicating whether or not the driver is making a patient face to the face determination information acquisition section 11 . A detailed configuration example of the face determination device 2 will be described later.

The face determination information acquisition unit 11 acquires face determination information from the face determination device 2 .
The face determination information acquisition unit 11 outputs the acquired face determination information to the estimation unit 13 .

The brightness detection unit 12 detects brightness in the captured image captured by the imaging device 3 . The luminance detection unit 12 may acquire the captured image via the face determination device 2 .
In Embodiment 1, the brightness in the captured image is, for example, the average value of brightness values of pixels in the captured image. Note that this is merely an example, and the luminance detection unit 12 may set, for example, the median value of the luminance values of the pixels in the captured image as the luminance in the captured image. The luminance detection unit 12 should detect the luminance of at least an area including the driver's face (hereinafter referred to as "face area") on the captured image. For example, the brightness detection unit 12 may identify the driver's face area from the captured image using a known image processing technique. Further, the brightness detection unit 12 may acquire information on feature points indicating the driver's face (hereinafter referred to as "feature point information") from a feature point detection unit 22 (described later) of the face determination device 2, for example. .
The luminance detection unit 12 outputs information about the detected luminance in the captured image (hereinafter referred to as “luminance information”) to the estimation unit 13 . The luminance detection unit 12 uses information that associates the captured image with the detected luminance as luminance information.

The estimating unit 13 estimates the arousal level of the driver based on the face determination information acquired by the face determination information acquiring unit 11 and the brightness of the captured image detected by the brightness detecting unit 12 . Note that the degree of arousal estimated by the estimation unit 13 may be, for example, a degree of arousal represented by whether the degree of arousal is high or low, or may be a degree of arousal represented by stages such as five stages.
Specifically, the estimation unit 13 first determines whether or not the face determination information is face determination information indicating that the driver is making a patient face. It is determined whether or not it is determined that the
When the face determination information is face determination information indicating that the driver is making a patient face, the estimating unit 13 determines that the brightness of the captured image is equal to a preset threshold value (hereinafter referred to as "brightness determination threshold value"). Determine if it is less than
If the luminance in the captured image is less than the threshold value for luminance determination, the estimating unit 13 estimates that the driver's wakefulness has decreased, in other words, the driver is drowsy.
The estimating unit 13 estimates that the driver's arousal level has not decreased if the luminance in the captured image is equal to or higher than the threshold value for luminance determination. The state in which the driver's arousal level has not decreased includes a state in which the driver's arousal level has increased and a state in which the driver's arousal level is maintained without change.

The situations in which the driver makes a patient face are assumed to be situations in which the level of arousal has decreased, and situations in which the surroundings are bright and the driver feels dazzled. For example, it is assumed that the driver feels dazzled when sunlight is shining around the driver, in this case, inside the vehicle. When the driver feels dazzling, he/she may put up a face even if he/she is not drowsy. That is, when the driver makes a patient face in a situation where it is assumed that the surroundings are bright, it can be estimated that the driver is not drowsy but that the driver made the patient face due to the glare. On the other hand, when the driver makes a patient face in a situation where the surroundings are not bright enough to feel dazzling, it can be inferred that the driver made a patient face because he is drowsy.

Therefore, if the face determination information is face determination information indicating that the driver is making a patient face and the luminance in the captured image is less than the threshold value for luminance determination, the estimating unit 13 determines that the patient face is: It is presumed that this was done because the driver's arousal level was lowered, that is, the driver's arousal level was lowered. If the brightness in the captured image is less than the threshold for brightness determination, it can be determined that the surroundings of the driver are not bright enough to make the driver feel dazzled. The brightness determination threshold value is set based on the brightness of the captured image captured by the administrator or the like under the brightness that is assumed to make the driver feel dazzled. The imaging range here is the interior of the vehicle including at least the range where the driver's face should be present.
On the other hand, if the face determination information is face determination information indicating that the driver is making a patient face and the luminance in the captured image is equal to or higher than the threshold value for luminance determination, the estimating unit 13 determines that the patient face is: It is presumed that this is not caused by a decrease in the driver's arousal level, that is, the driver's arousal level has not decreased. In other words, it is presumed that the persevering face was made due to the feeling of being dazzled. If the brightness in the captured image is equal to or higher than the brightness determination threshold, it can be determined that the surroundings of the driver are bright enough to make the driver feel dazzled.
In this way, when it is determined that the driver is making a patient face, the estimating unit 13 determines whether the patient face is due to a decrease in the driver's arousal level by determining the brightness around the driver. It is discriminated whether it is due to the driver's feeling of being dazzled rather than the decrease in the arousal level of the driver.

However, even if the area around the driver is bright enough to make the driver feel dazzled, if the area around the driver's face is locally dark such as a shadow is formed on the driver's face, the driver will be dazzled. Sometimes I don't feel it. In this case, the driver does not put up with the glare. In other words, if the driver makes a patient face in a situation where the area around the driver's face is locally dark, it can be estimated that this is not due to the feeling of being dazzled, but due to a decrease in the level of arousal. .
Therefore, even if the face determination information is face determination information indicating that the driver is making a patient face and the luminance in the captured image is equal to or higher than the luminance determination threshold value, the estimation unit 13 If it is determined that there is a shadow around the driver's face based on the luminance distribution, it may be estimated that the driver's arousal level has decreased. Note that the luminance information output from the luminance detection unit 12 to the estimation unit 13 includes the captured image. The estimation unit 13 may determine the luminance distribution on the captured image from the luminance value of each pixel on the captured image. Also, the estimation unit 13 may identify the driver's face area on the captured image using a known image processing technique. The face area is, for example, a minimum rectangular area that includes all feature points that indicate the outline of the driver. Then, for example, if the average value of the brightness values of the pixels included in the face region is equal to or less than a preset threshold value (hereinafter referred to as “shadow determination threshold value”), the estimation unit 13 is determined to have a shadow.
For example, the estimation unit 13 determines whether or not there is a shadow around the face of the driver by comparing the average luminance value of pixels included in the area where the driver's eyes exist (hereinafter referred to as "eye area") and the shadow determination value. You may judge by comparison with a threshold value. The eye area is, for example, a minimum rectangular area that includes all characteristic points indicating the driver's eyes, such as the inner and outer corners of the eye.

Conversely, even if the surroundings of the driver are not bright enough to make the driver feel dazzled, the driver's face may not be shadowed or the driver's face may be illuminated only. The area around the face is locally bright, and the driver may feel dazzled. In this case, the driver makes a patient face due to the glare. In other words, if the driver makes a patient face in a situation where the area around the driver's face is locally bright, it can be inferred that this is not due to a decrease in arousal level but due to the feeling of being dazzled. .
Therefore, even if the face determination information is face determination information indicating that the driver is making a patient face and the luminance in the captured image is less than the luminance determination threshold value, the estimation unit 13 If it is determined that the area around the driver's face is bright based on the luminance distribution, it may be estimated that the driver's arousal level has not decreased. For example, if the average value of the brightness values of the pixels included in the face region of the driver on the captured image is equal to or greater than a preset threshold value (hereinafter referred to as "brightness determination threshold value"), the estimation unit 13 , it is determined that the area around the driver's face is bright.
For example, the estimating unit 13 may determine whether or not the area around the driver's face is bright by comparing the average brightness value of pixels included in the driver's eye region with a brightness determination threshold.

The estimating unit 13 determines whether or not the face determination information indicates that the driver is making a patient face. As a result, the face determination information indicates that the driver is not making a patient face. In other words, when the face determination device 2 determines that the driver is not making a patient face, using a known image-based arousal level estimation technique, Estimate the degree of arousal of the driver. For example, the estimating unit 13 estimates that the driver's wakefulness has decreased when the driver's eye opening degree detected based on the captured image is equal to or less than a preset threshold value. The estimating unit 13 estimates that the driver's arousal level has not decreased when the eye opening degree of the driver is larger than a preset threshold value.

When the estimation unit 13 estimates that the driver's arousal level has decreased, it outputs information (hereinafter referred to as "warning information") to alert the driver. For example, the estimation unit 13 outputs warning information for outputting a warning sound to an output device (not shown). The output device is assumed to be, for example, a speaker mounted on a vehicle. When the estimation unit 13 outputs warning information, the output device outputs a warning sound.

The estimation unit 13 may store the estimation result of the degree of arousal of the driver in the storage unit 26 in association with the estimation date and time. In this case, the estimation unit 13 can determine the tendency of the degree of arousal of the driver, such as how much the degree of arousal of the driver has changed, based on the past estimation result.

A configuration example of the face determination device 2 will be described.
As shown in FIG. 1 , the face determination device 2 includes an image acquisition section 21 , a feature point detection section 22 , a distance calculation section 23 , a reference distance calculation section 24 , a face determination section 25 and a storage section 26 . The face determination section 25 includes a recalculation instruction section 251 .

The image acquisition unit 21 acquires a captured image from the imaging device 3 .
The image acquisition unit 21 outputs the acquired captured image to the feature point detection unit 22 and the brightness detection unit 12 .

The feature point detection unit 22 detects a plurality of feature points of the driver's face on the captured image based on the captured image acquired by the image acquisition unit 21 .
The plurality of feature points detected by the feature point detection unit 22 include at least feature points indicating the inner corners of the driver's eyebrows. In Embodiment 1, as an example, the plurality of feature points detected by the feature point detection unit 22 are feature points indicating the inner corners of the eyebrows of the driver.
The feature point detection unit 22 outputs feature point information regarding the detected feature points, specifically, feature points indicating the inner corners of the eyebrows of the driver, to the distance calculation unit 23 .
Note that, for example, the feature point detection unit 22 may detect feature points indicating the upper and lower eyelids of both eyes of the driver and output feature point information indicating the feature points to the estimation unit 13 . In this case, the estimating unit 13 can, for example, calculate the degree of eye opening of the driver based on the feature point information output from the feature point detecting unit 22, and estimate the degree of arousal of the driver from the calculated degree of eye opening.

The distance calculation unit 23 converts the coordinates of the plurality of feature points detected by the feature point detection unit 22 on the captured image into three-dimensional coordinates. Then, the distance calculation unit 23 calculates distances between the plurality of feature points (hereinafter referred to as "distances between feature points") from the converted three-dimensional coordinates.
Specifically, here, the distance calculation unit 23 converts the coordinates of the feature points indicating the inner corners of the driver's eyebrows on the captured image into three-dimensional coordinates, and calculates the distance between the inner corners of the driver's eyebrows from the converted three-dimensional coordinates. The distance (hereinafter referred to as "eyebrow distance") is calculated as the distance between feature points.
The distance calculation unit 23 may convert the coordinates of the feature points on the captured image into three-dimensional coordinates using a known technique for converting two-dimensional coordinates into three-dimensional coordinates, such as AAM (Active Appearance Model).
The distance calculation unit 23 outputs information about the calculated distance between feature points (hereinafter referred to as “distance information”) to the face determination unit 25 .
Further, the distance calculation unit 23 causes the storage unit 26 to store the distance information. Note that the distance calculation unit 23 stores the distance information in chronological order in association with the calculation time of the inter-feature point distance.
The distance information stored in the storage unit 26 is deleted when the power of the vehicle is turned off.

The reference distance calculation unit 24 refers to the distance information stored in the storage unit 26, and determines the distance between feature points calculated by the distance calculation unit 23 for a preset period (hereinafter referred to as "reference calculation target period"). ), the distance between reference feature points is calculated. The reference inter-feature point distance is the inter-feature point distance that is used as a reference when determining whether or not the driver is making a patient face in the face determination device 2, and is the feature point when the driver is not making a patient face. This is the distance that is regarded as the inter-distance. The determination of the patient face is performed by the face determination unit 25 . Details of the face determination unit 25 will be described later.
For example, the reference distance calculator 24 sets the mode of the inter-feature point distances for the reference calculation target period as the reference inter-feature point distance. Note that this is merely an example, and the reference distance calculation unit 24 may use, for example, the average value or the median value of the distances between feature points for the reference calculation target period as the reference distance between feature points.

The reference distance calculator 24 calculates the distance between reference feature points at a predetermined timing (hereinafter referred to as "reference calculation timing"). Specifically, the reference distance calculator 24 calculates the distance between the reference feature points from the distance information for the reference calculation target period, which is stored in the storage unit 26 after it is determined that the reference calculation timing has come.
In the first embodiment, the reference calculation timing is when the power of the vehicle is turned on and when recalculation instruction unit 251 outputs an instruction to recalculate the distance between reference feature points. Details of the recalculation instruction unit 251 will be described later.
If the storage unit 26 does not store the distance information for the reference calculation period after the reference calculation timing, the reference distance calculation unit 24 waits until the distance information for the reference calculation period is stored. do.
The reference distance calculation unit 24 causes the storage unit 26 to store information about the calculated distance between reference feature points (hereinafter referred to as “reference distance information”).

The face determination unit 25 determines whether or not the driver is making a patient face by comparing the distance between feature points calculated by the distance calculation unit 23 and the reference distance between feature points. The face determination unit 25 can identify the distance between reference feature points from the reference distance information stored in the storage unit 26 .
Specifically, when the distance between feature points, here, the distance between the driver's eyebrows, is smaller than the reference distance between feature points by a preset threshold (hereinafter referred to as "first threshold") or more, the face determination unit 25 In other words, if "the distance between the reference feature points - the distance between the feature points ≥ the first threshold", it is determined that the driver is making a patient face.
In general, when a person has a face with wrinkled brows, the distance between the brows is shorter than when a person has a face without wrinkled brows. Therefore, when the distance between the driver's eyebrows is smaller than the distance between the reference feature points by a first threshold value or more, the face determination unit 25 determines that the driver has wrinkled eyebrows, that is, a patient face.
If the distance between feature points is not smaller than the distance between reference feature points by a first threshold or more, in other words, if "distance between reference feature points - distance between feature points < first threshold", the face determination unit 25 determines that the driver is judged to be not showing patience.

Here, when it is determined that the driver is not making a patient face, the face determination unit 25 further determines that the distance between feature points is greater than the reference distance between feature points by a preset threshold (hereinafter referred to as "second threshold"). ), in other words, it is determined whether or not "the distance between the feature points - the distance between the reference feature points ≥ the second threshold".
When the inter-feature point distance is greater than the reference inter-feature point distance by the second threshold or more, the face determination unit 25 notifies the recalculation instruction unit 251 to that effect. When the recalculation instruction unit 251 is notified by the face determination unit 25 that the inter-feature point distance is larger than the reference inter-feature point distance by the second threshold or more, the re-calculation instruction unit 251 instructs the reference distance calculation unit 24 to calculate the distance between the reference feature points. Instructs recalculation of the distance.
If the inter-feature point distance is larger than the reference inter-feature point distance by the second threshold or more, it is assumed that the reference inter-feature point distance is calculated based on the inter-feature point distance when the driver is making a patient face. be. As described above, the reference inter-feature point distance is the distance that is regarded as the inter-feature distance (eyebrow distance) when the driver is not making a patient face. The distance between reference feature points calculated based on is not suitable as the distance between reference feature points. Therefore, when the face determination unit 25 determines that the inter-feature point distance is greater than the reference inter-feature point distance by the second threshold or more, the recalculation instruction unit 251 instructs the reference distance calculation unit 24 to recalculate the inter-reference feature point distance. let them do the calculations.

The face determination unit 25 outputs to the face determination information acquisition unit 11 face determination information indicating whether or not it is determined that the driver is making a patient face. Specifically, when determining that the driver is making a patient face, the face determination unit 25 outputs face determination information indicating that the driver is making a patient face to the face determination information acquisition unit 11 . When the face determination unit 25 determines that the driver is not making a patient face, the face determination unit 25 outputs face determination information indicating that the driver is not making a patient face to the face determination information acquisition unit 11 .

However, when it is necessary to issue an instruction to recalculate the distance between the reference feature points, that is, when the face determination unit 25 determines that the distance between the reference feature points is greater than the distance between the reference feature points by the second threshold or more, the face determination unit 25 Judgment information is not output. This is because there is a possibility that the determination that the driver is not making a patient face is an erroneous determination.

The storage unit 26 stores distance information and reference distance information.
Although the storage unit 26 is provided in the face determination device 2 in Embodiment 1, this is merely an example. The storage unit 26 may be provided outside the face determination device 2 at a location that the face determination device 2 can refer to.

The operation of the awakening level estimation device 1 according to Embodiment 1 will be described.
FIG. 2 is a flowchart for explaining the operation of the wakefulness estimation device 1 according to the first embodiment. In FIG. 2, the processing of steps ST1 to ST5 is processing performed by the face determination device 2 included in the arousal level estimation device 1. FIG.

The image acquisition unit 21 acquires a captured image from the imaging device 3 (step ST1).
The image acquisition unit 21 outputs the acquired captured image to the feature point detection unit 22 and the brightness detection unit 12 .

The feature point detection unit 22 detects a plurality of feature points of the driver's face on the captured image based on the captured image acquired by the image acquisition unit 21 in step ST1 (step ST2). In Embodiment 1, the feature point detection unit 22 detects feature points indicating the inner corners of the eyebrows of the driver.
The feature point detection section 22 outputs the feature point information to the distance calculation section 23 .

The distance calculator 23 converts the coordinates of the plurality of feature points detected by the feature point detector 22 in step ST2 into three-dimensional coordinates (step ST3). Then, the distance calculation unit 23 calculates the inter-feature point distances between the plurality of feature points from the converted three-dimensional coordinates (step ST4). In Embodiment 1, the distance calculation unit 23 converts the coordinates of the characteristic points indicating the inner corners of the driver's eyebrows on the captured image into three-dimensional coordinates, and calculates the distance between the driver's eyebrows from the converted three-dimensional coordinates. Calculate as a distance.
The distance calculation section 23 outputs the distance information to the face determination section 25 .

The face determination unit 25 compares the distance between feature points calculated by the distance calculation unit 23 in step ST4 with the reference distance between feature points, thereby determining whether or not the driver is making a patient face. Processing is performed (step ST5).

The face determination information acquisition unit 11 acquires face determination information from the face determination device 2 (step ST6). Specifically, the face determination information acquisition unit 11 acquires the face determination information output by the face determination unit 25 of the face determination device 2 performing the patient face determination process in step ST5.
The face determination information acquisition unit 11 outputs the acquired face determination information to the estimation unit 13 .

The brightness detection unit 12 detects brightness in the captured image captured by the imaging device 3 (step ST7). Specifically, the brightness detection unit 12 acquires the captured image acquired by the image acquisition unit 21 of the face determination device 2 in step ST1, and detects the brightness in the captured image.
The luminance detection unit 12 outputs luminance information to the estimation unit 13 .

The estimating unit 13 estimates the degree of arousal of the driver based on the face determination information acquired by the face determination information acquiring unit 11 in step ST6 and the brightness in the captured image detected by the brightness detection unit 12 in step ST7. Degree estimation processing is performed (step ST8).

In addition, in the flowchart of FIG. 2, the processing is performed in the order of step ST6 and step ST7, but the processing order of step ST6 and step ST7 is not limited to this. The processing of step ST7 and step ST6 may be performed in order, or the processing of step ST6 and the processing of step ST7 may be performed in parallel. The process of step ST7 may be performed at any timing after step ST1 until the process of step ST8 is performed.

FIG. 3 is a flowchart for explaining the details of the patient face determination process by the face determination unit 25 in step ST5 of FIG.

The face determination unit 25 determines whether or not the distance between feature points, here, the distance between the driver's eyebrows, is smaller than the reference distance between feature points by a first threshold or more. First threshold" (step ST21).
If the inter-feature point distance is smaller than the reference inter-feature point distance by the first threshold or more ("YES" in step ST21), the face determination unit 25 determines that the driver is making a patient face (step ST22). .
If the inter-feature point distance is not smaller than the reference feature point distance by a first threshold or more, in other words, if "the reference feature point distance - the inter-feature point distance < the first threshold value"("NO" in step ST21) ), the face determination unit 25 determines that the driver is not making a patient face (step ST24).

If it is determined in step ST24 that the driver is not making a patient face, the face determining unit 25 determines whether the distance between feature points is greater than the reference distance between feature points by a second threshold or more. It is determined whether or not "distance between points - distance between reference feature points≧second threshold" (step ST25).
If the inter-feature point distance is greater than the reference inter-feature point distance by the second threshold or more (“YES” in step ST25), the face determination section 25 notifies the recalculation instruction section 251 of that effect. Then, the recalculation instruction section 251 instructs the reference distance calculation section 24 to recalculate the distance between reference feature points (step ST26).

If the inter-feature point distance is not larger than the reference inter-feature point distance by a second threshold or more, in other words, if "inter-feature point distance - reference inter-feature point distance < second threshold" (“NO” in step ST25 case), and when it is determined in step ST22 that the driver is making a patient face, the face determination section 25 outputs face determination information to the face determination information acquisition section 11 (step ST23).
If it is necessary to issue an instruction to recalculate the distance between the reference feature points, the face determination unit 25 determines in step ST25 that the distance between the feature points is larger than the distance between the reference feature points by the second threshold or more. In this case, face determination information is not output. This is because the determination that the driver is not making a patient face (see step ST24) may be an erroneous determination.

FIG. 4 is a flowchart for explaining the details of the arousal level estimation process by the estimation unit 13 in step ST8 of FIG.

The estimation unit 13 determines whether the face determination information acquired by the face determination information acquisition unit 11 in step ST6 in FIG. 2 is face determination information indicating that the driver is making a patient face. It is determined whether or not the determination device 2 has determined that the driver is making a patient face (step ST31).

When the face determination information indicates that the driver is making a patient face (“YES” in step ST31), the estimating unit 13 determines that the luminance detecting unit 12 in step ST7 of FIG. It is determined whether or not the brightness in the detected captured image is less than the brightness determination threshold (step ST32).

When it is determined in step ST32 that the brightness in the captured image is less than the threshold value for brightness determination (“YES” in step ST32), the estimating unit 13 determines that the driver's face periphery is based on the brightness distribution on the captured image. It is determined whether or not it is bright (step ST33).

When it is determined in step ST33 that the area around the driver's face is bright ("YES" in step ST33), the estimation unit 13 estimates that the driver's arousal level has not decreased (step ST35). The estimating unit 13 may store the result of estimating the degree of wakefulness of the driver in the storage unit 26 .
When it is determined in step ST33 that the area around the driver's face is not bright ("NO" in step ST33), the process of the estimating section 13 proceeds to step ST34.

When it is determined in step ST32 that the brightness of the captured image is not less than the brightness determination threshold value, in other words, when it is determined that the brightness of the captured image is equal to or greater than the brightness determination threshold value ("NO" in step ST32), The estimation unit 13 determines whether or not there is a shadow around the driver's face based on the luminance distribution on the captured image (step ST36).

When it is determined in step ST36 that there is a shadow around the driver's face ("YES" in step ST36), and when it is determined in step ST33 that the driver's face is not bright (step ST33). If "NO"), the estimation unit 13 estimates that the driver's wakefulness has decreased, in other words, the driver is drowsy (step ST34). Then, the estimation unit 13 outputs warning information. The estimating unit 13 may store the result of estimating the degree of wakefulness of the driver in the storage unit 26 .

When it is determined in step ST36 that there is no shadow around the driver's face ("NO" in step ST36), the estimation unit 13 estimates that the driver's arousal level has not decreased (step ST37). The estimating unit 13 may store the result of estimating the degree of wakefulness of the driver in the storage unit 26 .

On the other hand, when the face determination information is face determination information indicating that the driver is not making a patient face (“NO” in step ST31), the estimating unit 13 performs a known image-based wakefulness estimation. Technology is used to estimate the degree of alertness of the driver (step ST38). As a result of estimating the degree of alertness of the driver, the estimation unit 13 outputs warning information when it is estimated that the degree of alertness of the driver has decreased. The estimating unit 13 may store the result of estimating the degree of wakefulness of the driver in the storage unit 26 .

In addition, in the flowchart shown in FIG. 4, the processing of steps ST33, ST35, and ST36 can be omitted. When the process of step ST36 is omitted, if it is determined in step ST32 that the luminance in the captured image is not less than the threshold for luminance determination ("NO" in step ST32), the estimation unit 13 performs the process of step ST37. proceed to

FIG. 5 is a flowchart for explaining the calculation operation of the distance between reference feature points by the reference distance calculator 24 in the first embodiment.
The operation shown in the flowchart of FIG. 5 is performed in parallel with the operation of the face determination device 2 described using the flowchart of FIG.

The reference distance calculator 24 determines whether or not the reference calculation timing has come (step ST41). Specifically, the reference distance calculation unit 24 determines whether the power of the vehicle is turned on, or whether an instruction to recalculate the distance between reference feature points is output from the recalculation instruction unit 251 (see step ST26 in FIG. 3). , to determine.
If it is determined that the reference calculation timing has not come (“NO” in step ST41), the reference distance calculator 24 waits until the reference calculation timing comes.

If it is determined that the reference calculation timing has come (“YES” in step ST41), the reference distance calculation unit 24 refers to the distance information stored in the storage unit 26, and calculates the feature points for the reference calculation target period. The inter-characteristic distance is obtained (step ST42), and the reference inter-characteristic point distance is calculated based on the obtained inter-characteristic point distance for the reference calculation target period (step ST43). Note that the reference distance calculation unit 24 acquires the distance between feature points from the distance information for the reference calculation target period, which is stored in the storage unit 26 from the point in time when it is determined that the reference calculation timing has come. If the distance information for the reference calculation target period is not stored in the storage unit 26 after it is determined that the reference calculation timing has come, the reference distance calculation unit 24 stores the distance information for the reference calculation target period. After waiting, the distance between reference feature points is calculated.

The reference distance calculation unit 24 causes the storage unit 26 to store the reference distance information (step ST44).

Based on the reference distance information calculated by the reference distance calculation unit 24 and stored in the storage unit 26 in this way, the face determination unit 25 performs the patient face determination process (see step ST5 in FIG. 2).
Here, after the reference calculation timing, the reference distance calculation unit 24 completes the calculation of the reference feature point distance, and until the reference distance information is stored in the storage unit 26, the face determination unit 25 keeps the patient face. Do not perform judgment processing. For example, when the reference distance calculation unit 24 determines that it is the reference calculation timing (“YES” in step ST41 of FIG. 5), it sets the reference calculation in progress flag “1”. The reference-calculating flag is set at a location that can be referred to by the reference distance calculation unit 24 and the face determination unit 25. When the reference distance calculation unit 24 stores the reference distance information in the storage unit 26 (step ST44 in FIG. 5), the reference The distance calculator 24 sets the initial value to “0”.
For example, when the reference calculation in-progress flag is set to "1", the face determination unit 25 notifies the face determination information acquisition unit 11 of waiting information (hereinafter referred to as "waiting notification"). Output. Upon acquiring the standby notification, the face determination information acquisition unit 11 outputs the standby notification to the estimation unit 13 . When the face determination information acquisition unit 11 outputs the standby notification, the estimation unit 13 performs the same method as when the face determination information acquisition unit 11 outputs the face determination information indicating that the driver is not making a patient face. Then, the degree of wakefulness of the driver is estimated. That is, the estimation unit 13 estimates the degree of arousal of the driver using a known image-based arousal degree estimation technique.
Further, for example, when the reference calculation in-progress flag is set to “1”, the face determination unit 25 determines the pre-stored general distance between the inner corners of the eyebrows when the person is not making a patient face. It may be determined whether or not the driver is making a patient face, using an assumed and set distance (hereinafter referred to as “alternative reference distance”) as the reference distance between feature points. The alternative reference distance is set in advance by an administrator or the like and stored in the face determination section 25 .

Thus, the wakefulness estimation device 1 acquires face determination information indicating whether or not the driver is making a patient face from the face determination device 2, and detects the luminance in the captured image. Then, the awakening level estimation device 1 estimates the driver's awakening level based on the acquired face determination information and the detected brightness.
The arousal level estimating device 1 estimates the driver's arousal level by considering whether the surrounding situation is a situation where the driver is assumed to feel dazzling when the driver is making a patient face. It is possible to prevent erroneous estimation of the decrease in arousal level of

Specifically, the arousal level estimation device 1 detects that the acquired face determination information is face determination information indicating that the driver is making a patient face, and the luminance in the detected captured image is less than the luminance determination threshold value. If there is, it is estimated that the driver's arousal level has decreased. When the driver makes a patient face based on the face determination information and the brightness in the captured image, the arousal level estimation device 1 determines whether the patient face is a face made due to a decrease in the driver's arousal level or whether the driver is dazzling. I can distinguish whether it is a face made because I felt it. As a result, the awakening level estimation device 1 can prevent erroneous estimation that the driver's awakening level has decreased.

Further, when the acquired face determination information is face determination information indicating that the driver is making a patient face, and the luminance in the detected imaged image is equal to or higher than the luminance determination threshold value, the arousal level estimation device 1 , it is estimated that the arousal level of the driver has not decreased. In other words, it is presumed that the driver made a patient face because he felt that the light was bright. When the driver makes a patient face based on the face determination information and the brightness in the captured image, the arousal level estimation device 1 determines whether the patient face is a face made due to a decrease in the driver's arousal level or whether the driver is dazzling. I can distinguish whether it is a face made because I felt it. As a result, the awakening level estimation device 1 can prevent erroneous estimation that the driver's awakening level has decreased.

Further, the arousal level estimation device 1 determines whether the acquired face determination information is face determination information indicating that the driver is making a patient face, and the luminance in the detected captured image is less than the luminance determination threshold value. Alternatively, if it is determined that the area around the driver's face is bright based on the luminance distribution on the captured image, it may be estimated that the driver's arousal level has not decreased. As a result, the arousal level estimation device 1 can create a situation in which the area around the driver's face becomes locally bright and the driver feels dazzled, even if there is a shadow or the like in the surroundings and the brightness is not high enough for the driver to feel dazzled. Considering the case, it is possible to distinguish whether the driver's patient face is a face made because the driver's arousal level has decreased or a face made because the driver feels dazzling. The arousal level estimation device 1 can more accurately prevent erroneous estimation that the driver's arousal level has decreased.

Further, the arousal level estimation device 1 determines that the acquired face determination information is face determination information indicating that the driver is making a patient face, and the luminance in the detected imaged image is equal to or higher than the luminance determination threshold value. Alternatively, if it is determined that there is a shadow around the driver's face based on the luminance distribution on the captured image, it may be estimated that the driver's arousal level has decreased. As a result, even if the brightness of the surroundings is such that the driver feels dazzled, the wakefulness estimation device 1 is in a situation where the area around the driver's face is locally shadowed and the driver does not feel dazzled. Considering the situation, it is possible to distinguish whether the driver's patient face is a face made because the driver's arousal level has decreased or a face made because the driver feels dazzling. The wakefulness estimation device 1 can more accurately estimate the decrease in the driver's wakefulness.

In addition, the arousal level estimation device 1, more specifically, the face determination device 2 of the arousal level estimation device 1, detects a plurality of feature points (both eyebrow tips) of the driver's face detected from the captured image, and calculates the coordinates on the captured image as 3 Dimensional coordinates are converted, and distances between feature points (eyebrow distance) between a plurality of feature points are calculated from the three-dimensional coordinates after conversion. Then, the arousal level estimation device 1 determines whether or not the driver is making a patient face by comparing the distance between the feature points with the distance between the reference feature points, and the distance between the feature points is greater than the distance between the reference feature points. is smaller than the first threshold value, it is determined that the driver is making a patient face.
The arousal level estimation device 1 converts the coordinates of a plurality of feature points on the captured image into three-dimensional coordinates and calculates the distance between the feature points. Alternatively, it can be made unaffected by tilting to the left or right. That is, the wakefulness estimation apparatus 1 can calculate the distance between feature points with higher accuracy than when calculating the distance between feature points without converting the coordinates of the feature points on the captured image into three-dimensional coordinates. As a result, the awakening level estimation device 1 can accurately determine whether or not the driver is making a patient face.

Further, the awakening level estimation device 1 calculates the distance between the reference feature points based on the history of the calculated distance between the feature points for the reference calculation target period, and compares the distance between the feature points with the distance between the reference feature points. . The arousal level estimation device 1 can set the distance between reference feature points according to the driver whose arousal level is to be estimated.
Further, when the arousal level estimation device 1 determines that the distance between feature points is greater than the distance between reference feature points by the second threshold or more as a result of comparing the distance between feature points and the distance between reference feature points, the wakefulness estimation apparatus 1 Recalculate the distance between points. As a result, the arousal level estimation device 1 determines whether the driver is making a patient face based on the reference feature point distance that is not suitable as the reference feature point distance that is regarded as the feature point distance when the driver is not making a patient face. It is possible not to judge whether or not there is.

6A and 6B are diagrams showing an example of the hardware configuration of the wakefulness estimation device 1 according to Embodiment 1. FIG.
In Embodiment 1, the face determination information acquisition unit 11, the luminance detection unit 12, the estimation unit 13, the image acquisition unit 21, the feature point detection unit 22, the distance calculation unit 23, and the reference distance calculation unit 24 , the functions of the face determination unit 25 are realized by the processing circuit 101 . That is, the arousal level estimating device 1 performs control for estimating the driver's arousal level based on the result of the face determination device 2 determining whether or not the driver is making a patient face based on the captured image and the luminance in the captured image. A processing circuit 101 is provided for.
The processing circuitry 101 may be dedicated hardware, as shown in FIG. 6A, or a processor 104 executing a program stored in memory, as shown in FIG. 6B.

If the processing circuit 101 is dedicated hardware, the processing circuit 101 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), or an FPGA (Field-Programmable). Gate Array), or a combination thereof.

When the processing circuit is the processor 104, the face determination information acquisition unit 11, the luminance detection unit 12, the estimation unit 13, the image acquisition unit 21, the feature point detection unit 22, the distance calculation unit 23, and the reference distance calculation unit. 24 and the functions of the face determination unit 25 are implemented by software, firmware, or a combination of software and firmware. Software or firmware is written as a program and stored in memory 105 . Processor 104 reads out and executes programs stored in memory 105 to obtain face determination information acquiring unit 11, luminance detecting unit 12, estimating unit 13, image acquiring unit 21, and feature point detecting unit 22. , the distance calculator 23 , the reference distance calculator 24 , and the face determination unit 25 . That is, wakefulness estimation apparatus 1 includes memory 105 for storing a program that, when executed by processor 104, results in execution of steps ST1 to ST8 in FIG. 2 described above. The programs stored in the memory 105 include a face determination information acquisition unit 11, a brightness detection unit 12, an estimation unit 13, an image acquisition unit 21, a feature point detection unit 22, a distance calculation unit 23, a reference It can also be said that the distance calculation unit 24 and the processing procedure or method of the face determination unit 25 are executed by a computer. Here, the memory 105 is a non-volatile memory such as RAM, ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable Read Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), etc. or volatile A semiconductor memory, a magnetic disk, a flexible disk, an optical disk, a compact disk, a mini disk, a DVD (Digital Versatile Disc), or the like is applicable.

Note that the face determination information acquisition unit 11, the luminance detection unit 12, the estimation unit 13, the image acquisition unit 21, the feature point detection unit 22, the distance calculation unit 23, the reference distance calculation unit 24, and the face determination unit 25 may be partly implemented by dedicated hardware and partly implemented by software or firmware. For example, the function of the image acquisition unit 21 is realized by a processing circuit 101 as dedicated hardware. The functions of the distance calculation unit 23, the reference distance calculation unit 24, and the face determination unit 25 can be realized by the processor 104 reading and executing programs stored in the memory 105. FIG.
The storage unit 26 is configured by, for example, a memory.
The wakefulness estimation apparatus 1 also includes a device such as the imaging device 3 or an output device (not shown), and an input interface device 102 and an output interface device 103 that perform wired or wireless communication.

In the first embodiment described above, the plurality of feature points detected by the feature point detection unit 22 are the feature points indicating the inner ends of the driver's eyebrows, and the distance calculation unit 23 detects the feature points indicating the inner ends of the driver's eyebrows on the captured image. The distance between the eyebrows was calculated as the distance between feature points based on the coordinates of . Then, the face determination unit 25 compares the distance between the eyebrows calculated by the distance calculation unit 23 and the distance between the reference feature points to determine whether or not the driver is making a patient face. However, this is only an example.
For example, in Embodiment 1 described above, the plurality of feature points detected by the feature point detection unit 22 are feature points indicating the inner corners of the driver's eyebrows and the inner corners of the driver's eyes. The distance between the eyebrows and the distance between the inner corners of the eyebrows may be calculated as the distances between the feature points, respectively, based on the coordinates of the feature points indicating the inner corners of the eyebrows and the inner corners of the eyes. The face determination unit 25 compares the distance between the eyebrows and the distance between the inner corners of the eyes with the distance between the reference feature points to determine whether or not the driver is making a patient face. In this case, the reference distance calculator 24 calculates the distance between the reference feature points corresponding to the distance between the eyebrows and the distance between the reference feature points corresponding to the distance between the inner corners of the eyes.
Further, for example, in Embodiment 1 described above, the plurality of feature points detected by the feature point detection unit 22 are feature points indicating the inner corners of the driver's eyes, and the distance calculation unit 23 detects the inner corners of the driver's eyes on the captured image. The distance between the inner corners of the eyes may be calculated as the inter-feature point distance based on the coordinates of the feature points indicating . The face determination unit 25 compares the distance between the inner corners of the eyes with the distance between the reference feature points to determine whether or not the driver is making a patient face. In this case, the reference distance calculator 24 calculates the distance between the reference feature points corresponding to the distance between the inner corners of the eyes.
However, the feature points related to the eyes are more likely not to be detected than the feature points related to the eyebrows. For example, when the driver wears sunglasses, the feature point detection unit 22 cannot detect feature points related to the driver's eyes. Therefore, in the wakefulness estimation devices 1 and 1a, the distance between the eyebrows calculated based on the feature points indicating the inner corners of the eyebrows is compared with the distance between the reference feature points to determine whether or not the driver is making a patient face. is preferred. The arousal level estimation devices 1 and 1a compare the distance between the eyebrows calculated based on the feature points indicating the inner corners of the eyebrows and the distance between the reference feature points to determine whether or not the driver is making a patient face. It is possible to reduce the possibility of falling into a situation where judgment cannot be made.

Further, in Embodiment 1 described above, the arousal level estimation device can also have a personal authentication function. With the personal authentication function, the arousal level estimating device can use the reference feature point distance calculated during the previous driving without waiting for the calculation of the reference feature point distance when the driver starts driving. It can be determined whether or not a certain driver is making a patient face. In this case, the wakefulness estimation device does not have to delete the reference distance information stored in the storage unit when the power of the vehicle is turned off. A detailed description will be given below.

FIG. 7 is a diagram showing a configuration example of the awakening level estimation device 1a provided with a personal authentication function in Embodiment 1. As shown in FIG.
In FIG. 7, the same components as those of the awakening level estimation device 1 described with reference to FIG. Here, it is assumed that the awakening level estimation device 1a has a personal authentication function in the face determination device 2a.
A face determination device 2a shown in FIG. 7 differs from the face determination device 2 described with reference to FIG. Further, the specific operations of the reference distance calculation unit 24 and the face determination unit 25 in the face determination device 2a shown in FIG. behavior is different.

The personal authentication unit 27 performs personal authentication of the driver.
For example, the personal authentication unit 27 stores a face image of the driver in advance, and performs personal authentication of the driver by pattern matching with the captured image obtained from the image obtaining unit 21 . It should be noted that this is merely an example, and the individual authentication unit 27 may perform individual authentication of the driver using a known individual authentication technique.
Personal authentication unit 27 outputs the result of personal authentication (hereinafter referred to as “personal authentication result”) to reference distance calculation unit 24 and face determination unit 25 .

The reference distance calculator 24 calculates the distance between reference feature points for each individual based on the result of personal authentication by the personal authentication unit 27 . Specifically, the reference distance calculator 24 calculates the reference feature point distance as the reference feature point distance of the driver authenticated by the personal authentication section 27 .
The reference distance calculation unit 24 causes the storage unit 26 to store, as reference distance information, information in which the calculated distance between reference feature points is associated with information that can identify the individual driver. It should be noted that the reference distance calculation unit 24 includes information that can identify the individual driver in the personal authentication result.
Moreover, the face determination unit 25 determines whether or not the driver is making a patient face based on the result of personal authentication by the personal authentication unit 27 . Specifically, the face determination unit 25 compares the personal identification result of the personal identification unit 27 with the reference distance information stored in the storage unit 26 to specify the distance between the reference feature points of the driver. The face determination unit 25 then compares the distance between feature points calculated by the distance calculation unit 23 with the reference distance between feature points of the specified driver, thereby determining whether or not the driver is making a patient face. . The face determination unit 25 determines that the inter-feature point distance is greater than the reference inter-feature point distance by a second threshold or more, and when notifying the recalculation instruction unit 251 to that effect, the face determination unit 25 uses information that can identify the individual driver. shall be notified together. When the recalculation instruction unit 251 instructs the reference distance calculation unit 24 to recalculate the distance between reference feature points, the recalculation instruction unit 251 also outputs information that can identify the individual driver. After recalculating the distance between the reference feature points of the driver, the reference distance calculation unit 24 updates the reference distance information corresponding to the driver stored in the storage unit 26 .

In this way, the reference distance calculation unit 24 stores in the storage unit 26 the reference distance information in which the information that can identify an individual and the distance between reference feature points are associated, so that the face determination unit 25 can When a certain driver starts driving, if there is a reference feature point distance that has been calculated during the previous drive without waiting for the calculation of the reference feature point distance, the driver can use the reference feature point distance to It can be determined whether or not the person is making a patient face.

FIG. 8 is a flow chart for explaining the operation of the awakening level estimation device 1a provided with the personal authentication function in the first embodiment.
Specific operations of steps ST51, ST53 to ST55, and steps ST57 to ST59 in FIG. 8 are similar to the specific operations of steps ST1 to ST4 and steps ST6 to ST8 in FIG. We omit the explanation.

The personal authentication unit 27 performs personal authentication of the driver (step ST52).
Personal authentication section 27 outputs the personal authentication result to reference distance calculation section 24 and face determination section 25 .

The face determination section 25 performs a patient face determination process to determine whether or not the driver is making a patient face based on the result of personal authentication by the personal authentication section 27 in step ST52 (step ST56).
In the patient face determination process, when reference distance information corresponding to the driver is stored in the storage unit 26, the face determination unit 25 acquires the distance between the reference feature points of the driver based on the reference distance information, and determines the feature. Compare with distance between points.
In the patient face determination process, when the recalculation instruction section 251 instructs the reference distance calculation section 24 to recalculate the distance between reference feature points, the recalculation instruction section 251 also outputs information that can identify the individual driver.

When the wakefulness estimation device 1a has a configuration as shown in FIG. 7, the reference distance calculation unit 24 calculates FIG. Then, the distance between the reference feature points for each individual is calculated.
Then, the reference distance calculation unit 24 causes the storage unit 26 to store, as reference distance information, information in which the calculated distance between reference feature points is associated with information that can identify the individual driver.

6A and 6B show an example of the hardware configuration of the wakefulness estimation device 1a when it has the configuration shown in FIG. 7. FIG.
A face determination information acquisition unit 11, a luminance detection unit 12, an estimation unit 13, an image acquisition unit 21, a feature point detection unit 22, a distance calculation unit 23, a reference distance calculation unit 24, and a face determination unit 25. , the functions of the individual authentication unit 27 are realized by the processing circuit 101 . That is, the arousal level estimation device 1a estimates the driver's arousal level based on the brightness in the captured image and the result of the face determination device 2a determining whether or not the driver is making a patient face based on the captured image. A processing circuit 101 is provided for performing authentication and performing control for storing the personal authentication result and the reference distance information in association with each other.
The processing circuit 101 reads out and executes programs stored in the memory 105 to obtain a face determination information acquisition unit 11, a luminance detection unit 12, an estimation unit 13, an image acquisition unit 21, and a feature point detection unit 22. , the functions of the distance calculation unit 23, the reference distance calculation unit 24, the face determination unit 25, and the personal authentication unit 27 are executed. That is, wakefulness estimation apparatus 1a includes memory 105 for storing a program that, when executed by processing circuit 101, results in execution of steps ST51 to ST59 in FIG. The programs stored in the memory 105 include a face determination information acquisition unit 11, a brightness detection unit 12, an estimation unit 13, an image acquisition unit 21, a feature point detection unit 22, a distance calculation unit 23, a reference It can also be said that the distance calculation unit 24, the face determination unit 25, and the personal identification unit 27 are executed by a computer.
The wakefulness estimation device 1a includes a device such as the imaging device 3 or an output device (not shown), and an input interface device 102 and an output interface device 103 that perform wired or wireless communication.

Further, in the first embodiment described above, it is assumed that the person whose arousal level is estimated by the arousal level estimation device 1 is the driver of the vehicle, and that the person whose arousal level is estimated is one person. But this is just one example. There may be a plurality of people whose arousal level is estimated by the arousal level estimation device. For example, the arousal level estimation device may estimate the arousal levels of a plurality of vehicle occupants, including the driver, a front passenger seat passenger, or a rear seat passenger.

When the wakefulness estimation device 1 is configured as shown in FIG. 1 and does not have a personal authentication function, for example, the feature point detection unit 22 determines the seat position of each passenger based on the captured image, and determines the seat position. A plurality of feature points are detected for each Then, the feature point detection unit 22 outputs to the distance calculation unit 23, as feature point information, information associated with the plurality of detected feature points and the information that can identify each occupant. Here, the information that can identify each occupant is information about the seat position.
The distance calculation unit 23 calculates the distance between feature points for each occupant corresponding to the seat position determined by the feature point detection unit 22 . The distance calculation unit 23 causes the storage unit 26 to store information that associates the information that can identify each passenger with the distance between feature points as distance information, and outputs the information to the face determination unit 25 .
The reference distance calculator 24 calculates the distance between reference feature points for each occupant corresponding to the seat position determined by the feature point detector 22 . The reference distance calculation unit 24 causes the storage unit 26 to store, as reference distance information, information that associates information that can identify each occupant with the distance between reference feature points.
Based on the distance information and the reference distance information, the face determination unit 25 determines whether or not each passenger is making a patient face.
The face determination unit 25 outputs to the face determination information acquisition unit 11 as face determination information information in which information that can identify each passenger is associated with information indicating whether or not the passenger is making a patient face.
When the recalculation instruction section 251 instructs the reference distance calculation section 24 to recalculate the distance between reference feature points, it also outputs information that can identify each passenger. After recalculating the distance between reference feature points, the reference distance calculation unit 24 updates the reference distance information of the corresponding occupant stored in the storage unit 26 .
The estimation unit 13 estimates the degree of wakefulness for each passenger. Note that the estimation unit 13 can determine each passenger from, for example, feature point information output from the feature point detection unit 22 and face determination information output from the face determination information acquisition unit 11 .

For example, when the wakefulness estimation device 1a has the configuration shown in FIG. It suffices to determine whether or not there is. Also, the estimation unit 13 may estimate the degree of wakefulness for each passenger based on the personal authentication result.
Specifically, the feature point detection unit 22 detects a plurality of feature points for each occupant based on the personal authentication result by the personal authentication unit 27 . Note that an arrow from the personal authentication unit 27 to the feature point detection unit 22 is omitted in FIG. The feature point detection unit 22 outputs information in which the plurality of detected feature points and the personal authentication result are associated with each other to the distance calculation unit 23 as feature point information.
The distance calculation unit 23 calculates the distance between feature points for each passenger based on the result of personal authentication by the personal authentication unit 27 . The distance calculation unit 23 causes the storage unit 26 to store information in which the calculated inter-feature point distance and the personal authentication result are associated with each other as distance information, and outputs the distance information to the face determination unit 25 .
The reference distance calculator calculates the distance between reference feature points for each occupant based on the result of personal identification by the personal identification unit 27 . The reference distance calculation unit 24 causes the storage unit 26 to store, as reference distance information, information in which the distance between reference feature points and the personal authentication result are associated with each other.
The face determination unit 25 determines whether or not each passenger is making a patient face based on the result of personal identification by the personal identification unit 27 . The face determination unit 25 outputs to the face determination information acquisition unit 11 as face determination information information in which the information indicating whether or not the person is making a patient face is associated with the personal authentication result.
The estimating unit 13 estimates the degree of wakefulness for each occupant based on the result of personal identification by the personal identification unit 27 . Note that the estimation unit 13 can determine each passenger from, for example, feature point information output from the feature point detection unit 22 and face determination information output from the face determination information acquisition unit 11 .

Further, in Embodiment 1 described above, the wakefulness level estimation devices 1 and 1a are in-vehicle devices mounted in vehicles, and the face determination information acquisition unit 11, the brightness detection unit 12, the estimation unit 13, and the image acquisition unit 21, a feature point detection unit 22, a distance calculation unit 23, a reference distance calculation unit 24, a face determination unit 25, and a personal authentication unit when the wakefulness level estimation device 1a is configured as shown in FIG. 27 is provided in the in-vehicle device.
Not limited to this, the face determination information acquisition unit 11, the brightness detection unit 12, the estimation unit 13, the image acquisition unit 21, the feature point detection unit 22, the distance calculation unit 23, the reference distance calculation unit 24, It is assumed that part of the face determination unit 25 and the personal authentication unit 27 is installed in an in-vehicle device of the vehicle, and the others are installed in a server connected to the in-vehicle device via a network. A wakefulness estimation system may be configured.
Also, a face determination information acquisition unit 11, a luminance detection unit 12, an estimation unit 13, an image acquisition unit 21, a feature point detection unit 22, a distance calculation unit 23, a reference distance calculation unit 24, and a face determination unit 25 and the personal authentication unit 27 may all be provided in the server.

Further, in Embodiment 1 described above, the face determination device 2 is provided in the wakefulness estimation device 1 and the face determination device 2a is provided in the wakefulness estimation device 1a, but this is merely an example. The face determination device 2 may be connected to the wakefulness estimation device 1 outside the wakefulness estimation device 1 . Further, the face determination device 2a may be connected to the wakefulness estimation device 1a outside the wakefulness estimation device 1a.
Further, some of the components included in the face determination device 2 may be included in the wakefulness estimation device 1, and some of the components included in the face determination device 2a may be included in the wakefulness estimation device. It may be provided in the device 1a. For example, the image acquisition unit 21, the feature point detection unit 22, the distance calculation unit 23, or the reference distance calculation unit 24 may be included in the wakefulness estimation device 1, and the image acquisition unit 21 and the feature point detection unit 22 , the distance calculator 23, the reference distance calculator 24, or the personal authentication unit 27 may be provided in the wakefulness estimation device 1a.

Further, in the first embodiment described above, the person whose awakening level is estimated by the awakening level estimation devices 1 and 1a is the driver of the vehicle or the passenger of the vehicle including the driver, but this is only an example. do not have. The awakening degree estimation devices 1 and 1a can estimate the awakening degree of not only the vehicle occupants but also all persons. For example, the wakefulness estimation devices 1 and 1a can estimate the wakefulness of a moving object other than a vehicle or a person in a living room. In this case, the imaging device 3 is also not limited to one mounted on the vehicle. The imaging device 3 may be installed at a position capable of imaging the range in which the face of a person whose arousal level is to be estimated by the arousal level estimating devices 1 and 1a should exist.

As described above, according to the first embodiment, the arousal level estimation apparatuses 1 and 1a acquire face determination information indicating whether or not a person has a face with wrinkles between the eyebrows (restrained face). A determination information acquisition unit 11; a brightness detection unit 12 that detects brightness in a captured image in which a range in which a human face should exist is captured; face determination information acquired by the face determination information acquisition unit 11; and an estimating unit 13 for estimating the degree of arousal of a person based on the detected luminance. Therefore, the arousal level estimation device 1 estimates the arousal level of a person in consideration of whether the surrounding situation is a situation in which the person is assumed to feel dazzling when the person is in a state where the eyebrows are wrinkled. By doing so, it is possible to prevent erroneous estimation of a decrease in a person's arousal level.

It should be noted that any component of the embodiment can be modified, or any component of the embodiment can be omitted.

The arousal level estimation device of the present disclosure estimates a person's arousal level by considering whether the surrounding situation is a situation where the person is assumed to feel dazzling when the person is in a state where the eyebrows are wrinkled. Therefore, it is possible to prevent erroneous estimation of a decrease in a person's arousal level.

Reference distance Calculation unit 25 Face determination unit 251 Recalculation instruction unit 26 Storage unit 27 Personal authentication unit 3 Imaging device 101 Processing circuit 102 Input interface device 103 Output interface device 104 Processor 105 Memory.

Claims (13)

1. a face determination information acquiring unit that acquires face determination information indicating whether or not a person has a face with wrinkled eyebrows;
   a brightness detection unit that detects brightness in a captured image in which a range in which a human face should exist is captured;
   an estimating unit for estimating the degree of arousal of the person based on the face determination information acquired by the face determination information acquiring unit and the brightness detected by the brightness detection unit.
2. The estimation unit
   The face determination information acquired by the face determination information acquisition unit is the face determination information indicating that the person has a face with wrinkled eyebrows, and the brightness detected by the brightness detection unit. is less than the luminance determination threshold, it is estimated that the person's arousal level has decreased.
3. The estimation unit
   The face determination information acquired by the face determination information acquisition unit is the face determination information indicating that the person has a face with wrinkled eyebrows, and the brightness detected by the brightness detection unit. is equal to or greater than the luminance determination threshold, it is estimated that the person's arousal level has not decreased.
4. The estimation unit
   The face determination information acquired by the face determination information acquisition unit is the face determination information indicating that the person has a face with wrinkled eyebrows, and the brightness detected by the brightness detection unit. is less than the brightness determination threshold, if it is determined that the area around the person's face is bright based on the brightness distribution on the captured image, it is estimated that the person's arousal level has not decreased. 3. The wakefulness estimation device according to claim 2.
5. The estimation unit
   The face determination information acquired by the face determination information acquisition unit is the face determination information indicating that the person has a face with wrinkled eyebrows, and the brightness detected by the brightness detection unit. is greater than or equal to the threshold for luminance determination, when it is determined that there is a shadow around the face of the person based on the luminance distribution on the captured image, it is estimated that the arousal level of the person has decreased. 4. The awakening level estimation device according to claim 3.
6. The estimation unit
   When the face determination information acquired by the face determination information acquisition unit is the face determination information indicating that the person does not have a face with wrinkles between the eyebrows, based on the captured image, 2. The arousal level estimation device according to claim 1, wherein the arousal level is estimated.
7. 2. The wakefulness level estimation device according to claim 1, wherein the luminance detection unit uses an average value of luminance values of pixels on the captured image as the luminance of the captured image.
8. an image acquisition unit that acquires the captured image;
   a feature point detection unit that detects a plurality of feature points of the person's face on the captured image acquired by the image acquisition unit;
   converting the coordinates of the plurality of feature points detected by the feature point detection unit on the captured image into three-dimensional coordinates, and using the three-dimensional coordinates after conversion as an inter-feature point distance between the plurality of feature points; a distance calculation unit that calculates
   By comparing the distance between the feature points calculated by the distance calculation unit and the distance between the reference feature points, it is determined whether or not the person has a face with wrinkled eyebrows, and the distance between the feature points is determined by comparing the distance between the feature points. 2. The wakefulness estimation device according to claim 1, further comprising: a face determination unit that determines that the person has a face with wrinkled eyebrows when the distance between the reference feature points is smaller than the reference feature point distance by a first threshold or more.
9. a reference distance calculation unit that calculates the reference distance between the feature points based on a history of the distance between the feature points calculated by the distance calculation unit for a reference calculation target period;
   9. Arousal level estimation according to claim 8, wherein the face determination unit compares the distance between feature points calculated by the distance calculation unit and the reference distance between feature points calculated by the reference distance calculation unit. Device.
10. When the face determination unit compares the inter-feature point distance and the reference inter-feature point distance and determines that the inter-feature point distance is larger than the reference inter-feature point distance by a second threshold or more, the reference distance a recalculation instruction unit that instructs the calculation unit to recalculate the distance between the reference feature points;
    10. The awakening level according to claim 9, wherein the reference distance calculation unit recalculates the reference inter-feature point distance when instructed by the recalculation instruction unit to recalculate the reference inter-feature point distance. estimation device.
11. a personal authentication unit that performs personal authentication of the person;
    The reference distance calculation unit calculates the distance between the reference feature points for each individual based on the result of the personal authentication by the personal authentication unit,
    The face determination unit compares the distance between the feature points with the distance between the feature points based on the result of the personal authentication by the person authentication unit and the distance between the reference feature points for each individual calculated by the reference distance calculation unit. 10. The arousal level estimation device according to claim 9, wherein distances between points are specified.
12. 9. The awakening degree estimation device according to claim 8, wherein the plurality of feature points are the inner corners of the eyebrows of the person, and the distance between the feature points is the distance between the inner corners of the eyebrows of the person.
13. a step in which a face determination information acquisition unit acquires face determination information indicating whether or not a person has a face with wrinkled eyebrows;
    a step in which the brightness detection unit detects brightness in a captured image in which a range in which a human face should exist is captured;
    an estimation unit estimating the degree of arousal of the person based on the face determination information acquired by the face determination information acquisition unit and the brightness detected by the brightness detection unit.

Images