JP7403729B2 - Awakening level estimation device and arousal level estimation method - Google Patents

Description

The present disclosure relates to an alertness level estimation device and an alertness level estimation method.

When people become drowsy and their level of alertness decreases, their eyebrows may become wrinkled. Conventionally, people develop wrinkles between their eyebrows because the distance between feature points representing the inner corners of both eyes and eyebrows of a person detected based on an image obtained by capturing a person's face is less than a predetermined threshold. There is a known technique for estimating a person's sleepiness state based on the frequency with which the state is detected (for example, Patent Document 1).

JP2008-212298A

People's eyebrows furrowed not only when they become sleepy and their alertness decreases. People may also wrinkle their eyebrows when they feel bright light. Therefore, it is difficult to distinguish whether the person's alertness level has decreased or whether the person feels dazzling just by looking at the facial expression of the person's eyebrows furrowed.
With conventional technology, typified by the technology disclosed in Patent Document 1, it is difficult to distinguish whether a person's alertness level has decreased or whether the person feels dazzling just by looking at the facial expression of the person's eyebrows furrowed. Since this could not be taken into account, there was a possibility of incorrectly estimating that a person's alertness level had decreased.

The present disclosure has been made in order to solve the above-mentioned problems, and it is necessary to investigate whether the surrounding situation is such that the person would feel dazzling when the person has wrinkled eyebrows. It is an object of the present invention to provide an alertness level estimation device that prevents erroneous estimation of a decrease in a person's alertness level by estimating the alertness level of the person.

The arousal level estimation device according to the present disclosure includes a face determination information acquisition unit that acquires face determination information indicating whether or not a person has a face with wrinkles between the eyebrows, and a range in which the human face should be present. a brightness detection unit that detects the brightness in the captured image, and an estimation unit that estimates the degree of arousal of the person based on the face determination information acquired by the face determination information acquisition unit and the brightness detected by the brightness detection unit. It is something that

According to the present disclosure, when a person has wrinkled eyebrows, the degree of arousal of the person is estimated by considering whether the surrounding situation is such that the person would feel dazzling. , it is possible to prevent incorrect estimation of a person's alertness level decline.

1 is a diagram illustrating a configuration example of an alertness level estimation device according to Embodiment 1. FIG. 3 is a flowchart for explaining the operation of the alertness level estimating device according to the first embodiment. 3 is a flowchart for explaining details of the patient face determination process by the face determination section in step ST5 of FIG. 2. FIG. 3 is a flowchart for explaining details of the arousal level estimation process by the estimator in step ST8 of FIG. 2. FIG. 7 is a flowchart for explaining an operation of calculating a reference inter-feature point distance by a reference distance calculation unit in the first embodiment. 6A and 6B are diagrams illustrating an example of the hardware configuration of the alertness level estimation device according to the first embodiment. 1 is a diagram illustrating a configuration example of an alertness level estimation device including a personal authentication function in Embodiment 1. FIG. 2 is a flowchart for explaining the operation of the alertness level estimation device 1 equipped with a personal authentication function in the first embodiment.

Embodiments of the present disclosure will be described in detail below with reference to the drawings.
Embodiment 1.
The alertness level estimation device according to the first embodiment estimates a person's alertness level based on a captured image in which at least the person's face is captured. In particular, when a person makes a face with wrinkles between their eyebrows, the alertness estimation device can determine whether the face was made due to a decrease in the person's alertness level or because the person felt dazzling. It distinguishes whether it is a face or not and estimates the person's level of arousal.
In the first embodiment, a face with wrinkles between the eyebrows is referred to as a "patience face." The ``patience face'' is a so-called frowning face with wrinkles between the eyebrows and narrowed eyes.
Further, in the first embodiment, as an example, the person whose arousal degree is estimated by the arousal degree estimating device is a driver of a vehicle.
The estimation result of whether the driver's alertness level has decreased, in other words, whether the driver is drowsy, which is estimated by the alertness level estimation device, is used, for example, to detect whether the driver is dozing off.

FIG. 1 is a diagram showing a configuration example of an alertness level estimation device 1 according to the first embodiment.
The arousal level estimation device 1 includes a face determination device 2, a face determination information acquisition section 11, a brightness detection section 12, and an estimation section 13.
The alertness estimation device 1 is connected to the imaging device 3. The imaging device 3 is mounted on a vehicle (not shown) and is installed so as to be able to image at least an area where the driver's face should be present. For example, the imaging device 3 may be shared with a so-called DMS (Driver Monitoring System) installed for the purpose of monitoring the interior of a vehicle. The imaging device 3 is a visible light camera or an infrared camera. The imaging device 3 outputs the captured image to the face determination device 2 included in the arousal level estimation device 1 .

The face determination device 2 acquires a captured image and determines whether or not the driver is making a restrained face based on the acquired captured image. The face determination device 2 outputs information indicating whether the driver is making a restrained face (hereinafter referred to as "face determination information") to the face determination information acquisition unit 11. A detailed configuration example of the face determination device 2 will be described later.

The face determination information acquisition unit 11 acquires face determination information from the face determination device 2 .
The face determination information acquisition unit 11 outputs the acquired face determination information to the estimation unit 13.

The brightness detection unit 12 detects the brightness in the captured image captured by the imaging device 3. The brightness detection unit 12 may acquire the captured image via the face determination device 2.
In the first embodiment, the brightness in a captured image is, for example, the average value of the brightness values of each pixel in the captured image. Note that this is just an example, and the brightness detection unit 12 may, for example, set the median value of the brightness values of each pixel in the captured image as the brightness in the captured image. The brightness detection unit 12 may detect the brightness of at least an area on the captured image that includes an area where the driver's face exists (hereinafter referred to as a "face area"). The brightness detection unit 12 may, for example, identify the driver's face area from the captured image using a known image processing technique. Further, the brightness detection unit 12 may obtain information regarding feature points indicating the driver's face (hereinafter referred to as “feature point information”) from a feature point detection unit 22 (described later) of the face determination device 2, for example. .
The brightness detection unit 12 outputs information regarding the brightness in the detected captured image (hereinafter referred to as “brightness information”) to the estimation unit 13. The brightness detection unit 12 uses information in which the captured image and the detected brightness are associated with each other as brightness information.

The estimation unit 13 estimates the driver's degree of wakefulness based on the face determination information acquired by the face determination information acquisition unit 11 and the brightness in the captured image detected by the brightness detection unit 12. Note that the degree of arousal estimated by the estimation unit 13 may be, for example, a degree of arousal expressed as whether the degree of arousal is high or low, or may be a degree of arousal expressed in stages such as five levels.
Specifically, the estimation unit 13 first determines whether the face determination information is face determination information indicating that the driver is making a patient face. It is determined whether or not it is determined that the
When the face determination information is face determination information indicating that the driver is making a restrained face, the estimation unit 13 determines that the brightness in the captured image is a preset threshold (hereinafter referred to as "brightness determination threshold"). Determine whether it is less than
If the brightness in the captured image is less than the brightness determination threshold, the estimation unit 13 estimates that the driver's alertness level has decreased, in other words, that the driver is drowsy.
If the brightness in the captured image is equal to or greater than the brightness determination threshold, the estimation unit 13 estimates that the driver's alertness level has not decreased. Note that the state in which the driver's arousal level has not decreased includes a state in which the driver's arousal level has increased, and a state in which the driver's arousal level remains unchanged.

Situations in which the driver makes a patient face include a situation in which the level of alertness has decreased, and a situation in which the surroundings are bright and the driver feels dazzling. For example, if sunlight is shining around the driver, here, inside the car, it is assumed that the driver feels dazzled. When a driver feels dazzled, he or she may look resigned even if he or she is not drowsy. That is, when a driver makes a patient face in a situation where it is assumed that the surroundings are bright, it can be estimated that the driver made a patient face not because he was drowsy but because of the glare. On the other hand, if the driver makes a patient face in a situation where the surroundings are not so bright that the driver feels dazzling, it can be assumed that the driver made the patient face because he/she is feeling sleepy.

Therefore, if the face determination information is face determination information indicating that the driver is making a patient face, and the brightness in the captured image is less than the brightness determination threshold, the estimation unit 13 determines that the patient face is It is estimated that this is caused by a decrease in the driver's alertness level, that is, the driver's alertness level has decreased. If the brightness in the captured image is less than the brightness determination threshold, it can be determined that the surroundings of the driver are not so bright that the driver feels dazzled. Note that the brightness determination threshold is set based on the brightness of a captured image captured by an administrator or the like under a brightness in which the imaging range is assumed to cause the driver to feel dazzling. Here, the imaging range is the interior of the vehicle, including at least the range where the driver's face should be present.
On the other hand, if the face determination information is face determination information indicating that the driver is making a patient face, and the brightness in the captured image is greater than or equal to the brightness determination threshold, the estimation unit 13 determines that the patient face is It is assumed that this is not caused by a decrease in the driver's alertness level, that is, the driver's alertness level has not decreased. In other words, it is presumed that the patient made the patient's face because he/she felt that it was bright. If the brightness in the captured image is equal to or greater than the brightness determination threshold, it can be determined that the surroundings of the driver are bright enough to cause the driver to feel dazzled.
In this way, when it is determined that the driver is making a patient face, the estimation unit 13 determines whether the patient's patient face is due to a decrease in the driver's arousal level by determining the brightness around the driver. We will distinguish whether this is due to the driver feeling dazzling rather than a decrease in alertness.

However, even if the driver's surroundings are bright enough to cause the driver to feel dazzled, if the area around the driver's face is locally dark, such as when there is a shadow on the driver's face, the driver may be dazzled. Sometimes I don't feel that way. In this case, the driver does not put up with the glare. In other words, if the driver makes a resigned face in a situation where the area around the driver's face is locally darkened, it can be assumed that this is not due to the feeling of being dazzled, but due to a decrease in alertness. .
Therefore, even if the face determination information is face determination information indicating that the driver is making a patient face, and the brightness in the captured image is equal to or higher than the brightness determination threshold, the estimation unit 13 calculates that If it is determined that there is a shadow around the driver's face based on the brightness distribution, it may be estimated that the driver's alertness level has decreased. Note that the brightness information output from the brightness detection unit 12 to the estimation unit 13 includes a captured image. The estimation unit 13 may determine the brightness distribution on the captured image from the brightness value of each pixel on the captured image. Furthermore, the estimation unit 13 may specify the driver's face area on the captured image using a known image processing technique. The face area is, for example, a minimum rectangular area that includes all feature points showing the outline of the driver. Then, for example, if the average value of the luminance values of pixels included in the face area is equal to or less than a preset threshold (hereinafter referred to as "shadow determination threshold"), the estimation unit 13 calculates It is determined that there is a shadow.
For example, the estimation unit 13 determines whether there is a shadow around the driver's face using the average luminance value of the pixels included in the area where the driver's eyes exist (hereinafter referred to as the "eye area") and the shadow determination. The determination may be made by comparison with a threshold value. The eye area is a minimum rectangular area that includes all feature points indicating the driver's eyes, such as the inner and outer corners of the eyes.

Conversely, even if the driver's surroundings are not bright enough to cause the driver to feel dazzled, there may be other problems such as there being no shadows on the driver's face or light shining only on the driver's face. The area around the face is locally bright, and the driver may feel dazzled. In this case, the driver looks resigned to the glare. In other words, if the driver makes a resigned face in a situation where the area around the driver's face is locally bright, it can be assumed that this is not due to a decrease in arousal level but because the driver feels dazzled. .
Therefore, even if the face determination information is face determination information indicating that the driver is making a patient face, and the brightness in the captured image is less than the brightness determination threshold, the estimation unit 13 calculates the If it is determined that the area around the driver's face is bright based on the brightness distribution, it may be estimated that the driver's alertness level has not decreased. For example, if the average value of the brightness values of pixels included in the face area of the driver on the captured image is greater than or equal to a preset threshold (hereinafter referred to as "brightness determination threshold"), the estimation unit 13 determines that , it is determined that the area around the driver's face is bright.
For example, the estimation unit 13 may determine whether or not the area around the driver's face is bright by comparing the average brightness value of pixels included in the driver's eye area with a brightness determination threshold.

As a result of determining whether the face determination information is face determination information indicating that the driver is making a patient face, the estimation unit 13 determines that the face determination information indicates that the driver is not making a patient face. In other words, if the face determination device 2 determines that the driver is not making a restrained face, using a known technique for estimating the degree of arousal based on the image, Estimate the driver's alertness level. For example, the estimating unit 13 estimates that the driver's wakefulness level has decreased when the driver's eye open level detected based on the captured image is less than or equal to a preset threshold. If the degree of eye opening of the driver is greater than a preset threshold, the estimation unit 13 estimates that the degree of wakefulness of the driver has not decreased.

If the estimation unit 13 estimates that the driver's alertness level has decreased, it outputs information for alerting the driver (hereinafter referred to as "warning information"). For example, the estimation unit 13 outputs warning information that causes an output device (not shown) to output a warning sound. The output device is assumed to be, for example, a speaker mounted on a vehicle. The output device outputs a warning sound when the warning information is output from the estimation unit 13.

The estimation unit 13 may store the estimation result of the driver's wakefulness level in the storage unit 26 in association with the date and time when the estimation was performed. In this case, the estimation unit 13 can determine the tendency of the driver's arousal degree, such as how much the driver's arousal degree has changed, based on past estimation results.

A configuration example of the face determination device 2 will be explained.
As shown in FIG. 1, the face determination device 2 includes an image acquisition section 21, a feature point detection section 22, a distance calculation section 23, a reference distance calculation section 24, a face determination section 25, and a storage section 26. The face determination section 25 includes a recalculation instruction section 251.

The image acquisition unit 21 acquires a captured image from the imaging device 3.
The image acquisition unit 21 outputs the acquired captured image to the feature point detection unit 22 and the brightness detection unit 12.

The feature point detection unit 22 detects a plurality of feature points of the driver's face on the captured image based on the captured image acquired by the image acquisition unit 21.
The plurality of feature points detected by the feature point detection unit 22 include at least a feature point indicating the tip of both eyebrows of the driver. In Embodiment 1, as an example, the plurality of feature points detected by the feature point detection unit 22 are feature points indicating the ends of both eyebrows of the driver.
The feature point detection unit 22 outputs the detected feature point, specifically, feature point information regarding the feature point indicating the tip of both eyebrows of the driver, to the distance calculation unit 23.
Note that, for example, the feature point detection unit 22 may detect feature points indicating the upper and lower eyelids of both eyes of the driver, and output feature point information indicating the feature points to the estimation unit 13. In this case, the estimation unit 13 can calculate the degree of eye opening of the driver based on the feature point information output from the feature point detection unit 22, and estimate the degree of wakefulness of the driver from the calculated degree of eye opening.

The distance calculation unit 23 converts the coordinates of the plurality of feature points detected by the feature point detection unit 22 on the captured image into three-dimensional coordinates. Then, the distance calculation unit 23 calculates the distance between the plurality of feature points (hereinafter referred to as "distance between feature points") from the converted three-dimensional coordinates.
Specifically, here, the distance calculation unit 23 converts the coordinates of the feature points indicating both eyebrows of the driver on the captured image into three-dimensional coordinates, and calculates the distance between the driver's eyebrows from the converted three-dimensional coordinates. The distance (hereinafter referred to as "glabellar distance") is calculated as the distance between feature points.
The distance calculation unit 23 may convert the coordinates of the feature points on the captured image into three-dimensional coordinates using a known technique such as AAM (Active Appearance Model) that converts two-dimensional coordinates into three-dimensional coordinates.
The distance calculation unit 23 outputs information regarding the calculated distance between feature points (hereinafter referred to as “distance information”) to the face determination unit 25.
Further, the distance calculation unit 23 causes the storage unit 26 to store the distance information. Note that the distance calculation unit 23 stores the distance information in chronological order in association with the calculation time of the distance between feature points.
It is assumed that the distance information stored in the storage unit 26 is deleted when the power of the vehicle is turned off.

The reference distance calculation unit 24 refers to the distance information stored in the storage unit 26 and determines a preset period (hereinafter referred to as “reference calculation target period”) of the distance between feature points calculated by the distance calculation unit 23. ) is used to calculate the distance between reference feature points. The reference distance between feature points is the distance between feature points that serves as a reference when determining whether or not the driver is making a patient face in the face determination device 2, and is the distance between feature points when the driver is not making a patient face. This is the distance considered as the distance between the two. Note that the determination of a patient face is performed by the face determination unit 25. Details of the face determination section 25 will be described later.
For example, the reference distance calculation unit 24 sets the mode of the distance between feature points for the reference calculation target period as the reference distance between feature points. Note that this is just an example, and the reference distance calculation unit 24 may use, for example, the average value or median value of the distances between feature points for the reference calculation target period as the reference distance between feature points.

The reference distance calculation unit 24 calculates the distance between reference feature points at a predetermined timing (hereinafter referred to as "reference calculation timing"). Specifically, the reference distance calculation unit 24 calculates the reference inter-feature point distance from the distance information for the reference calculation target period, which is stored in the storage unit 26 from the time when it is determined that the reference calculation timing has arrived.
In the first embodiment, the reference calculation timing is when the vehicle is powered on and when the recalculation instruction unit 251 outputs an instruction to recalculate the distance between reference feature points. Details of the recalculation instruction unit 251 will be described later.
Note that, if the distance information for the reference calculation target period is not stored in the storage unit 26 after the reference calculation timing, the reference distance calculation unit 24 waits until the distance information for the reference calculation target period is stored. do.
The reference distance calculation unit 24 causes the storage unit 26 to store information regarding the calculated reference distance between feature points (hereinafter referred to as “reference distance information”).

The face determination unit 25 determines whether the driver is making a restrained face by comparing the distance between feature points calculated by the distance calculation unit 23 and the reference distance between feature points. The face determination unit 25 can specify the reference distance between feature points from the reference distance information stored in the storage unit 26.
Specifically, the face determination unit 25 detects when the distance between feature points, in this case, the distance between the driver's eyebrows, is smaller than the reference distance between feature points by a preset threshold (hereinafter referred to as "first threshold") or more. In other words, if "reference inter-feature point distance - inter-feature point distance≧first threshold", it is determined that the driver is making a patient face.
Generally, when a person has a face with wrinkles between the eyebrows, the distance between the eyebrows becomes shorter than when the person has a face without wrinkles. Therefore, if the distance between the driver's eyebrows is smaller than the reference distance between feature points by a first threshold value or more, the face determination unit 25 determines that the driver has a face with wrinkles between the eyebrows, that is, a patient face.
If the distance between feature points is not smaller than the reference distance between feature points by a first threshold value or more, in other words, if "reference distance between feature points - distance between feature points < first threshold", the face determination unit 25 determines whether the driver It is judged that he does not have a patient look on his face.

Here, if it is determined that the driver is not making a patient face, the face determination unit 25 further determines that the distance between feature points is smaller than the reference distance between feature points by a preset threshold (hereinafter referred to as "second threshold"). ), in other words, it is determined whether "distance between feature points - reference distance between feature points ≧second threshold".
If the distance between feature points is larger than the reference distance between feature points by the second threshold or more, the face determination unit 25 notifies the recalculation instruction unit 251 to that effect. When the face determination unit 25 notifies the recalculation instruction unit 251 that the distance between feature points is larger than the reference distance between feature points by the second threshold or more, the recalculation instruction unit 251 instructs the reference distance calculation unit 24 to calculate the distance between the reference feature points. Instructs to recalculate distance.
If the distance between feature points is larger than the reference distance between feature points by the second threshold or more, it is assumed that the reference distance between feature points is calculated based on the distance between feature points when the driver is making a patient face. Ru. As mentioned above, the standard distance between feature points is the distance that is considered to be the distance between feature points (distance between eyebrows) when the driver is not making a patient face, so it is the distance between feature points when the driver is making a patient face. The reference inter-feature point distance calculated based on is not suitable as the reference inter-feature point distance. Therefore, when the face determination unit 25 determines that the distance between feature points is larger than the reference distance between feature points by the second threshold or more, the recalculation instruction unit 251 instructs the reference distance calculation unit 24 to recalculate the distance between reference feature points. Have them do the calculations.

The face determination unit 25 outputs face determination information indicating whether or not it has been determined that the driver is wearing a restrained face to the face determination information acquisition unit 11 . Specifically, when the face determination unit 25 determines that the driver is wearing a restrained expression, it outputs face determination information indicating that the driver is wearing a restrained expression to the face determination information acquisition unit 11 . If the face determination unit 25 determines that the driver is not making a patient face, it outputs face determination information indicating that the driver is not making a patient face to the face determination information acquisition unit 11 .

However, if it is necessary to instruct recalculation of the distance between reference feature points, that is, if it is determined that the distance between feature points is larger than the reference distance between feature points by the second threshold or more, the face determination unit 25 Judgment information is not output. This is because there is a possibility that the determination that the driver is not showing patience may be an erroneous determination.

The storage unit 26 stores distance information and reference distance information.
Note that in the first embodiment, the storage unit 26 is provided in the face determination device 2, but this is only an example. The storage unit 26 may be provided at a location outside the face determination device 2 that the face determination device 2 can refer to.

The operation of the alertness level estimation device 1 according to the first embodiment will be explained.
FIG. 2 is a flowchart for explaining the operation of the alertness level estimation device 1 according to the first embodiment. Note that in FIG. 2, the processing from step ST1 to step ST5 is performed by the face determination device 2 included in the arousal level estimation device 1.

The image acquisition unit 21 acquires a captured image from the imaging device 3 (step ST1).
The image acquisition unit 21 outputs the acquired captured image to the feature point detection unit 22 and the brightness detection unit 12.

The feature point detection unit 22 detects a plurality of feature points of the driver's face on the captured image based on the captured image acquired by the image acquisition unit 21 in step ST1 (step ST2). In the first embodiment, the feature point detection unit 22 detects feature points indicating the ends of both eyebrows of the driver.
The feature point detection unit 22 outputs feature point information to the distance calculation unit 23.

The distance calculation unit 23 converts the coordinates of the plurality of feature points detected by the feature point detection unit 22 in step ST2 on the captured image into three-dimensional coordinates (step ST3). Then, the distance calculation unit 23 calculates the inter-feature point distance between the plurality of feature points from the converted three-dimensional coordinates (step ST4). In the first embodiment, the distance calculation unit 23 converts the coordinates of feature points indicating both eyebrows of the driver on the captured image into three-dimensional coordinates, and calculates the distance between the driver's eyebrows from the converted three-dimensional coordinates between the feature points. Calculate as distance.
The distance calculation unit 23 outputs distance information to the face determination unit 25.

The face determination unit 25 performs a patience face determination to determine whether the driver is making a patience face by comparing the distance between feature points calculated by the distance calculation unit 23 in step ST4 and the reference distance between feature points. Processing is performed (step ST5).

The face determination information acquisition unit 11 acquires face determination information from the face determination device 2 (step ST6). Specifically, the face determination information acquisition unit 11 acquires the face determination information output by the face determination unit 25 of the face determination device 2 after performing the patient face determination process in step ST5.
The face determination information acquisition unit 11 outputs the acquired face determination information to the estimation unit 13.

The brightness detection unit 12 detects the brightness in the captured image captured by the imaging device 3 (step ST7). Specifically, the brightness detection unit 12 acquires the captured image acquired by the image acquisition unit 21 of the face determination device 2 in step ST1, and detects the brightness in the captured image.
The brightness detection unit 12 outputs brightness information to the estimation unit 13.

The estimation unit 13 estimates the driver's degree of arousal based on the face determination information acquired by the face determination information acquisition unit 11 in step ST6 and the brightness in the captured image detected by the brightness detection unit 12 in step ST7. degree estimation processing is performed (step ST8).

Note that in the flowchart of FIG. 2, the processing is performed in the order of step ST6 and step ST7, but the order of the processing of step ST6 and step ST7 is not limited to this. The processing may be performed in the order of step ST7 and step ST6, or the processing of step ST6 and the processing of step ST7 may be performed in parallel. The process in step ST7 may be performed at any timing after step ST1 until the process in step ST8 is performed.

FIG. 3 is a flowchart for explaining details of the patient face determination process by the face determination unit 25 in step ST5 of FIG.

The face determination unit 25 determines whether the distance between feature points, here the distance between the driver's eyebrows, is smaller than the reference distance between feature points by a first threshold value or more, in other words, "reference distance between feature points - distance between feature points ≧ It is determined whether the threshold value is the first threshold (step ST21).
If the distance between feature points is smaller than the reference distance between feature points by the first threshold value or more (“YES” in step ST21), the face determination unit 25 determines that the driver is making a restrained face (step ST22). .
If the distance between feature points is not smaller than the reference feature point distance by more than the first threshold, in other words, if "reference distance between feature points - distance between feature points < first threshold" (if "NO" in step ST21) ), the face determining unit 25 determines that the driver is not making a restrained face (step ST24).

In step ST24, when it is determined that the driver is not making a patient face, the face determining unit 25 determines whether the distance between feature points is larger than the reference distance between feature points by the second threshold or more, in other words, whether the distance between feature points is larger than the reference distance between feature points, It is determined whether "distance between points - distance between reference feature points≧second threshold" (step ST25).
If the inter-feature point distance is larger than the reference inter-feature point distance by the second threshold or more (“YES” in step ST25), the face determination unit 25 notifies the recalculation instruction unit 251 to that effect. Then, the recalculation instruction unit 251 instructs the reference distance calculation unit 24 to recalculate the reference inter-feature point distance (step ST26).

When the distance between feature points is not larger than the reference distance between feature points by the second threshold value or more, in other words, when "distance between feature points - reference distance between feature points < second threshold"("NO" in step ST25) ), and if it is determined in step ST22 that the driver is making a restrained face, the face determination section 25 outputs face determination information to the face determination information acquisition section 11 (step ST23).
Note that when it is necessary to instruct recalculation of the reference inter-feature point distance, the face determination unit 25 determines in step ST25 that the inter-feature point distance is greater than the reference inter-feature point distance by a second threshold value or more. In this case, face recognition information will not be output. This is because the determination that the driver is not holding back (see step ST24) may be an erroneous determination.

FIG. 4 is a flowchart for explaining details of the alertness level estimation process by the estimator 13 in step ST8 of FIG.

The estimation unit 13 determines whether the face determination information acquired by the face determination information acquisition unit 11 in step ST6 in FIG. The determination device 2 determines whether it is determined that the driver is wearing a restrained expression (step ST31).

If the face determination information is face determination information indicating that the driver is making a patient face (in the case of "YES" in step ST31), the estimation unit 13 determines that the brightness detection unit 12 It is determined whether the brightness in the detected captured image is less than a brightness determination threshold (step ST32).

If it is determined in step ST32 that the brightness in the captured image is less than the brightness determination threshold (“YES” in step ST32), the estimation unit 13 determines that the area around the driver's face is It is determined whether or not it is bright (step ST33).

If it is determined in step ST33 that the area around the driver's face is bright (“YES” in step ST33), the estimation unit 13 estimates that the driver's alertness level has not decreased (step ST35). The estimation unit 13 may store the estimation result of the driver's wakefulness level in the storage unit 26.
If it is determined in step ST33 that the area around the driver's face is not bright (“NO” in step ST33), the process of the estimation unit 13 proceeds to step ST34.

If it is determined in step ST32 that the brightness in the captured image is not less than the brightness determination threshold, in other words, if it is determined that the brightness in the captured image is greater than or equal to the brightness determination threshold (“NO” in step ST32), The estimation unit 13 determines whether there is a shadow around the driver's face based on the brightness distribution on the captured image (step ST36).

If it is determined in step ST36 that there is a shadow around the driver's face (“YES” in step ST36), and if it is determined in step ST33 that the area around the driver's face is not bright (“YES” in step ST33), If "NO"), the estimating unit 13 estimates that the driver's alertness level has decreased, in other words, that the driver is drowsy (step ST34). Then, the estimation unit 13 outputs warning information. The estimation unit 13 may store the estimation result of the driver's wakefulness level in the storage unit 26.

If it is determined in step ST36 that there is no shadow around the driver's face (“NO” in step ST36), the estimation unit 13 estimates that the driver's alertness level has not decreased (step ST37). The estimation unit 13 may store the estimation result of the driver's wakefulness level in the storage unit 26.

On the other hand, if the face determination information is face determination information indicating that the driver is not making a patient face (“NO” in step ST31), the estimation unit 13 uses a known image-based alertness level estimation method. Using the technique, the driver's degree of alertness is estimated (step ST38). As a result of estimating the driver's arousal level, the estimation unit 13 outputs warning information when it is estimated that the driver's arousal level has decreased. The estimation unit 13 may store the estimation result of the driver's wakefulness level in the storage unit 26.

Note that in the flowchart shown in FIG. 4, the processes of step ST33, step ST35, and step ST36 can be omitted. If the process in step ST36 is omitted, if it is determined in step ST32 that the brightness in the captured image is not less than the brightness determination threshold (in the case of "NO" in step ST32), the estimation unit 13 performs the process in step ST37. Proceed to.

FIG. 5 is a flowchart for explaining the operation of calculating the distance between reference feature points by the reference distance calculation unit 24 in the first embodiment.
The operation shown in the flowchart of FIG. 5 is performed in parallel with the operation of the face determination device 2 described using the flowchart of FIG.

The reference distance calculation unit 24 determines whether or not the reference calculation timing has arrived (step ST41). Specifically, the reference distance calculation unit 24 determines whether the power of the vehicle has been turned on or whether an instruction to recalculate the distance between reference feature points has been output from the recalculation instruction unit 251 (see step ST26 in FIG. 3). , is determined.
If it is determined that the reference calculation timing has not arrived (“NO” in step ST41), the reference distance calculation unit 24 waits until the reference calculation timing has arrived.

When it is determined that the reference calculation timing has come (“YES” in step ST41), the reference distance calculation unit 24 refers to the distance information stored in the storage unit 26 and calculates the feature points for the reference calculation target period. The distance between the reference points is obtained (step ST42), and the reference distance between the feature points is calculated based on the obtained distance between the feature points for the reference calculation target period (step ST43). Note that the reference distance calculation unit 24 acquires the distance between feature points from the distance information for the reference calculation target period, which is stored in the storage unit 26 from the time when it is determined that the reference calculation timing has arrived. If the storage unit 26 does not store distance information for the standard calculation target period from the time when it is determined that the standard calculation timing has come, the standard distance calculation unit 24 stores the distance information for the standard calculation target period until the distance information for the standard calculation target period is stored. After waiting, the distance between reference feature points is calculated.

The reference distance calculation unit 24 stores the reference distance information in the storage unit 26 (step ST44).

In this manner, based on the reference distance information calculated by the reference distance calculation unit 24 and stored in the storage unit 26, the face determination unit 25 performs the patient face determination process (see step ST5 in FIG. 2).
Here, after the reference calculation timing is reached, the face determination unit 25 performs the patient face determination process until the reference distance calculation unit 24 completes the calculation of the reference feature point distance and the reference distance information is stored in the storage unit 26. No judgment processing is performed. For example, when determining that it is the reference calculation timing (“YES” in step ST41 in FIG. 5), the reference distance calculation unit 24 sets the reference calculation in progress flag to “1”. The reference calculation in progress flag is set at a location that can be referenced by the reference distance calculation unit 24 and the face determination unit 25, and when the reference distance calculation unit 24 stores the reference distance information in the storage unit 26 (step ST44 in FIG. 5), the reference distance calculation unit 24 stores the reference distance information in the storage unit 26 (step ST44 in FIG. 5). The distance calculation unit 24 sets the initial value to "0".
For example, when the standard calculation flag is set to "1", the face determination section 25 sends information to the face determination information acquisition section 11 that it is on standby (hereinafter referred to as "standby notification"). Output. Upon acquiring the standby notification, the face determination information acquisition unit 11 outputs the standby notification to the estimation unit 13 . The estimation unit 13 uses the same method as when the face determination information acquisition unit 11 outputs face determination information indicating that the driver is not holding back when the standby notification is output from the face determination information acquisition unit 11. Then, the driver's level of alertness is estimated. That is, the estimation unit 13 estimates the driver's level of alertness using a known technique for estimating the level of alertness based on images.
Further, for example, when the standard calculation flag is set to "1", the face determination unit 25 calculates the distance between the eyebrows and the tips of the eyebrows when the person is not making a restrained face, which is stored in advance. It may be determined whether or not the driver is making a patient face by using a distance that has been assumed and set (hereinafter referred to as an "alternative reference distance") as a reference distance between feature points. The alternative reference distance is set in advance by an administrator or the like and stored in the face determination unit 25.

In this way, the alertness level estimation device 1 acquires face determination information indicating whether the driver is making a restrained face from the face determination device 2, and detects the brightness in the captured image. Then, the arousal degree estimating device 1 estimates the driver's arousal degree based on the acquired face determination information and the detected brightness.
The alertness level estimating device 1 estimates the driver's alertness level by considering whether the surrounding situation is such that the driver is expected to feel dazzling when the driver is holding back. It is possible to prevent erroneous estimation of the decrease in arousal level.

Specifically, the arousal level estimation device 1 determines that the acquired face determination information is face determination information indicating that the driver is making a patient face, and that the brightness in the detected captured image is less than the brightness determination threshold. If so, it is assumed that the driver's alertness level has decreased. The alertness level estimating device 1 determines, based on the face determination information and the brightness in the captured image, when the driver makes a patient face, whether the patient's patient face is a face made due to a decrease in the driver's arousal level or whether the driver is dazzling. You can tell whether it's a face made because you felt that way or not. Thereby, the alertness level estimation device 1 can prevent erroneous estimation that the driver's alertness level has decreased.

In addition, when the acquired face determination information is face determination information indicating that the driver is making a patient face, and the brightness in the detected captured image is equal to or higher than the brightness determination threshold, the alertness level estimation device 1 , it is estimated that the driver's alertness level has not decreased. In other words, it is presumed that the driver put up his patience because he felt dazzled. The alertness level estimating device 1 determines, based on the face determination information and the brightness in the captured image, when the driver makes a patient face, whether the patient's patient face is a face made due to a decrease in the driver's arousal level or whether the driver is dazzling. You can tell whether it's a face made because you felt that way or not. Thereby, the alertness level estimation device 1 can prevent erroneous estimation that the driver's alertness level has decreased.

Further, the arousal level estimation device 1 determines that the acquired face determination information is face determination information indicating that the driver is making a patient face, and that the brightness in the detected captured image is less than the brightness determination threshold. Also, if it is determined that the area around the driver's face is bright based on the brightness distribution on the captured image, it may be estimated that the driver's alertness level has not decreased. As a result, the alertness estimation device 1 can detect a situation in which the area around the driver's face becomes locally bright and the driver feels dazzled, even if there are shadows etc. in the surrounding area and the brightness is not so bright that the driver feels dazzled. It is possible to distinguish whether the driver's patience face is a face made due to a decrease in the driver's alertness level or a face made because the driver feels dazzling. The alertness level estimation device 1 can more accurately prevent erroneous estimation of a decrease in the driver's alertness level.

Further, the arousal level estimation device 1 determines that the acquired face determination information is face determination information indicating that the driver is making a patient face, and that the brightness in the detected captured image is equal to or higher than a brightness determination threshold. Also, if it is determined that there is a shadow around the driver's face based on the brightness distribution on the captured image, it may be estimated that the driver's alertness level has decreased. As a result, the alertness level estimation device 1 is able to create a situation in which even if the surroundings are bright enough to cause the driver to feel dazzled, the area around the driver's face is locally shaded and the driver does not feel dazzled. Considering the situation, it is possible to distinguish whether the driver's patience face is a face made because the driver's alertness level has decreased or a face made because the driver feels dazzling. The alertness level estimation device 1 can estimate a decrease in the driver's alertness level with higher accuracy.

In addition, the wakefulness estimation device 1, in detail, the face determination device 2 of the wakefulness estimation device 1 calculates the coordinates on the captured image of a plurality of feature points (the tips of both eyebrows) of the driver's face detected from the captured image. The feature points are converted into dimensional coordinates, and the inter-feature distance (distance between the eyebrows) between the plurality of feature points is calculated from the converted three-dimensional coordinates. Then, the alertness estimation device 1 determines whether the driver is making a patient face by comparing the distance between feature points and the reference distance between feature points, and the distance between feature points is larger than the reference distance between feature points. is smaller than the first threshold value, it is determined that the driver is making a patient face.
The alertness estimation device 1 calculates the distance between the feature points by converting the coordinates of the plurality of feature points on the captured image into three-dimensional coordinates, so that the distance between the feature points calculated is the distance between the front and back of the driver's face. Or it can be made unaffected by tilting to the left or right. In other words, the arousal level estimation device 1 can calculate the distance between feature points with higher accuracy than when calculating the distance between feature points without converting the coordinates of the feature points on the captured image into three-dimensional coordinates. As a result, the alertness level estimation device 1 can accurately determine whether the driver is wearing a patient expression.

In addition, the arousal level estimation device 1 calculates a reference distance between feature points based on the history of the calculated distance between feature points for the reference calculation target period, and compares the distance between feature points and the reference distance between feature points. . The alertness level estimation device 1 can set the distance between reference feature points according to the driver whose alertness level is to be estimated.
Further, as a result of comparing the inter-feature point distance and the reference inter-feature point distance, if the arousal level estimation device 1 determines that the inter-feature point distance is greater than the reference feature point distance by the second threshold or more, the reference Recalculate the distance between points. As a result, the alertness estimation device 1 determines whether the driver is making a patient face based on the reference feature point distance that is not suitable as the reference feature point distance, which is considered as the distance when the driver is not making a patient face. It is possible to avoid determining whether or not the

6A and 6B are diagrams illustrating an example of the hardware configuration of the alertness level estimation device 1 according to the first embodiment.
In the first embodiment, the face determination information acquisition section 11, the brightness detection section 12, the estimation section 13, the image acquisition section 21, the feature point detection section 22, the distance calculation section 23, and the reference distance calculation section 24. , the functions of the face determination section 25 are realized by the processing circuit 101. That is, the arousal degree estimation device 1 performs control to estimate the driver's arousal degree based on the result of the face determination device 2 determining whether the driver is making a restrained face based on the captured image and the brightness in the captured image. A processing circuit 101 is provided.
The processing circuit 101 may be dedicated hardware as shown in FIG. 6A, or may be a processor 104 that executes a program stored in memory as shown in FIG. 6B.

When the processing circuit 101 is dedicated hardware, the processing circuit 101 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), or an FPGA (Field-Programmable Circuit). Gate Array), or a combination of these.

When the processing circuit is the processor 104, the face determination information acquisition section 11, the brightness detection section 12, the estimation section 13, the image acquisition section 21, the feature point detection section 22, the distance calculation section 23, and the reference distance calculation section The functions of 24 and the face determination unit 25 are realized by software, firmware, or a combination of software and firmware. Software or firmware is written as a program and stored in memory 105. The processor 104 reads out and executes the program stored in the memory 105 to operate the face determination information acquisition section 11, the brightness detection section 12, the estimation section 13, the image acquisition section 21, and the feature point detection section 22. , executes the functions of the distance calculation section 23, the reference distance calculation section 24, and the face determination section 25. That is, the alertness estimation device 1 includes a memory 105 for storing a program that, when executed by the processor 104, results in the execution of steps ST1 to ST8 in FIG. Further, the program stored in the memory 105 includes the face determination information acquisition unit 11, the brightness detection unit 12, the estimation unit 13, the image acquisition unit 21, the feature point detection unit 22, the distance calculation unit 23, and the reference It can also be said that the computer is caused to execute the processing procedure or method of the distance calculation section 24 and the face determination section 25. Here, the memory 105 includes, for example, RAM, ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable Read Only Memory), and EEPROM (Electrically Erasable Programmable Memory). non-volatile or volatile This includes semiconductor memory, magnetic disks, flexible disks, optical disks, compact disks, mini disks, DVDs (Digital Versatile Discs), and the like.

Note that the face determination information acquisition section 11, the brightness detection section 12, the estimation section 13, the image acquisition section 21, the feature point detection section 22, the distance calculation section 23, the reference distance calculation section 24, and the face determination section Regarding the 25 functions, some may be realized by dedicated hardware, and some may be realized by software or firmware. For example, the function of the image acquisition unit 21 is realized by the processing circuit 101 as dedicated hardware, and includes the face determination information acquisition unit 11, the brightness detection unit 12, the estimation unit 13, the feature point detection unit 22, The functions of the distance calculation section 23, reference distance calculation section 24, and face determination section 25 can be realized by the processor 104 reading out and executing programs stored in the memory 105.
The storage unit 26 is composed of, for example, a memory.
The alertness estimation device 1 also includes an input interface device 102 and an output interface device 103 that perform wired or wireless communication with an imaging device 3 or a device such as an output device (not shown).

In the above-described first embodiment, the plurality of feature points detected by the feature point detection unit 22 are feature points indicating both eyebrows of the driver, and the distance calculation unit 23 detects feature points indicating both eyebrows of the driver on the captured image. Based on the coordinates of , the distance between the eyebrows was calculated as the distance between feature points. Then, the face determination unit 25 compared the distance between the eyebrows calculated by the distance calculation unit 23 with the reference inter-feature point distance, and determined whether the driver was making a restrained face. However, this is just one example.
For example, in the first embodiment described above, the plurality of feature points detected by the feature point detection section 22 are feature points indicating the inner corners of both eyebrows of the driver and the inner corners of both eyes of the driver, and the distance calculation section 23 detects the features of the driver on the captured image. The distance between the eyebrows and the distance between the inner corners of the eyes may be calculated as distances between the feature points, respectively, based on the coordinates of the feature points indicating the inner corners of both eyebrows and the inner corners of the eyes. The face determination unit 25 compares the distance between the eyebrows and the distance between the inner corners of the eyes with the reference feature point distance, respectively, and determines whether the driver is making a restrained face. In this case, the reference distance calculation unit 24 calculates a reference inter-feature point distance corresponding to the distance between the eyebrows and a reference inter-feature point distance corresponding to the distance between the inner corners of the eyes.
Further, for example, in the first embodiment described above, the plurality of feature points detected by the feature point detection unit 22 are feature points indicating the inner corners of both eyes of the driver, and the distance calculation unit 23 detects the inner corners of both eyes of the driver on the captured image. The distance between the inner corners of both eyes may be calculated as the inter-feature point distance based on the coordinates of the feature points that indicate . The face determination unit 25 compares the distance between the inner corners of both eyes with the reference distance between feature points, and determines whether the driver is making a restrained face. Note that in this case, the reference distance calculation unit 24 calculates the reference inter-feature point distance corresponding to the distance between the inner corners of both eyes.
However, feature points related to eyes are more likely not to be detected than feature points related to eyebrows. For example, if the driver is wearing sunglasses, the feature point detection unit 22 cannot detect feature points related to the driver's eyes. Therefore, in the arousal level estimation devices 1 and 1a, the distance between the eyebrows calculated based on the feature points indicating the tips of both eyebrows is compared with the distance between the reference feature points to determine whether or not the driver is making a patient face. is preferred. The arousal level estimation devices 1 and 1a compare the distance between the eyebrows calculated based on the feature points indicating the tips of both eyebrows and the distance between the reference feature points to determine whether or not the driver is making a patient face. The possibility of falling into a situation where judgment cannot be made can be reduced.

Furthermore, in the first embodiment described above, the alertness level estimating device may also include a personal authentication function. By being equipped with a personal authentication function, when a driver starts driving, the alertness estimation device can use the distance between standard feature points that has already been calculated during the previous driving, without waiting for the calculation of the distance between standard feature points. It is possible to determine whether a certain driver is holding back. In this case, the alertness level estimation device does not need to delete the reference distance information stored in the storage unit when the vehicle is powered off. This will be explained in detail below.

FIG. 7 is a diagram illustrating a configuration example of an alertness level estimation device 1a equipped with a personal authentication function in the first embodiment.
In FIG. 7, the same components as the alertness level estimating device 1 explained using FIG. Here, it is assumed that the arousal level estimation device 1a includes a personal authentication function in the face determination device 2a.
The face determination device 2a shown in FIG. 7 differs from the face determination device 2 described using FIG. 1 in that it includes a personal authentication section 27. Further, the specific operations of the reference distance calculation unit 24 and the face determination unit 25 in the face determination device 2a shown in FIG. This is different from normal operation.

The personal authentication unit 27 performs personal authentication of the driver.
For example, the personal authentication unit 27 stores the driver's face image in advance, and performs personal authentication of the driver by pattern matching with the captured image acquired from the image acquisition unit 21. Note that this is just an example, and the personal authentication section 27 may perform personal authentication of the driver using a known personal authentication technique.
The personal authentication unit 27 outputs the result of personal authentication (hereinafter referred to as “personal authentication result”) to the reference distance calculation unit 24 and the face determination unit 25.

The reference distance calculation unit 24 calculates the distance between reference feature points for each individual based on the individual authentication result by the individual authentication unit 27. Specifically, the reference distance calculation unit 24 calculates the distance between reference feature points as the distance between reference feature points of the driver authenticated by the personal authentication unit 27 .
The reference distance calculation unit 24 causes the storage unit 26 to store, as reference distance information, information in which the calculated reference distance between feature points is associated with information that can identify the individual driver. Note that the reference distance calculation unit 24 includes information that can identify the individual driver in the personal authentication result.
Further, the face determination unit 25 determines whether the driver is making a restrained face based on the personal authentication result by the personal authentication unit 27. Specifically, the face determination section 25 matches the personal authentication result by the personal authentication section 27 with the reference distance information stored in the storage section 26, and specifies the distance between reference feature points of the driver. Then, the face determination unit 25 determines whether the driver is making a patient face by comparing the distance between feature points calculated by the distance calculation unit 23 and the reference distance between feature points of the specified driver. . When the face determination unit 25 determines that the distance between feature points is larger than the reference distance between feature points by a second threshold value or more and notifies the recalculation instruction unit 251 to that effect, the face determination unit 25 uses information that can identify the individual driver. We will also notify you. When instructing the reference distance calculation unit 24 to recalculate the distance between reference feature points, the recalculation instruction unit 251 also outputs information that can identify the individual driver. When the reference distance calculation unit 24 recalculates the distance between reference feature points of the driver, the reference distance calculation unit 24 updates the reference distance information corresponding to the driver stored in the storage unit 26.

In this way, the reference distance calculation unit 24 stores in the storage unit 26 the reference distance information in which personally identifiable information and the reference inter-feature point distance are associated with each other, so that the face determination unit 25 can When a certain driver starts driving, without having to wait for the calculation of the distance between reference feature points, if there is a distance between reference feature points that has already been calculated during the previous driving, the driver can start driving using the distance between reference feature points. It is possible to determine whether or not the person is showing patience.

FIG. 8 is a flowchart for explaining the operation of the alertness level estimation device 1a equipped with a personal authentication function in the first embodiment.
The specific operations of step ST51, steps ST53 to ST55, and steps ST57 to ST59 in FIG. 8 are the same as the specific operations of steps ST1 to ST4 and steps ST6 to ST8 in FIG. 2, respectively, so there is no overlap. The explanation given will be omitted.

The personal authentication unit 27 performs personal authentication of the driver (step ST52).
The personal authentication section 27 outputs the personal authentication result to the reference distance calculation section 24 and the face determination section 25.

The face determination unit 25 performs a patience face determination process to determine whether or not the driver is making a patience face based on the personal authentication result by the personal authentication unit 27 in step ST52 (step ST56).
In the patient face determination process, if the storage unit 26 stores reference distance information corresponding to the driver, the face determination unit 25 acquires the distance between the driver's reference feature points based on the reference distance information, and determines the distance between the driver's feature points. Compare with the distance between points.
In addition, in the patience face determination process, when instructing the reference distance calculation unit 24 to recalculate the distance between reference feature points, the recalculation instruction unit 251 also outputs information that can identify the individual driver.

Note that when the alertness level estimating device 1a has a configuration as shown in FIG. The distance between reference feature points for each individual is calculated by performing the process described using the above method.
Then, the reference distance calculation unit 24 causes the storage unit 26 to store, as reference distance information, information in which the calculated reference distance between feature points is associated with information that can identify the individual driver.

An example of the hardware configuration when the alertness level estimation device 1a has the configuration shown in FIG. 7 is as shown in FIGS. 6A and 6B.
A face determination information acquisition section 11, a brightness detection section 12, an estimation section 13, an image acquisition section 21, a feature point detection section 22, a distance calculation section 23, a reference distance calculation section 24, and a face determination section 25. , the functions of the personal authentication section 27 are realized by the processing circuit 101. That is, the arousal degree estimation device 1a estimates the driver's arousal degree based on the brightness in the captured image and the result of the face determination device 2a determining whether the driver is making a patient face based on the captured image, and also estimates the driver's arousal level based on the brightness in the captured image. A processing circuit 101 is provided for controlling the authentication and storing the personal authentication result and reference distance information in association with each other.
The processing circuit 101 reads out and executes a program stored in the memory 105, thereby controlling the face determination information acquisition section 11, the brightness detection section 12, the estimation section 13, the image acquisition section 21, and the feature point detection section 22. Then, the functions of the distance calculation section 23, reference distance calculation section 24, face determination section 25, and personal authentication section 27 are executed. That is, the alertness estimation device 1a includes a memory 105 for storing a program that, when executed by the processing circuit 101, results in the execution of steps ST51 to ST59 in FIG. 8 described above. Further, the program stored in the memory 105 includes the face determination information acquisition unit 11, the brightness detection unit 12, the estimation unit 13, the image acquisition unit 21, the feature point detection unit 22, the distance calculation unit 23, and the reference It can also be said that the computer is made to execute the processing procedures or methods of the distance calculation section 24, face determination section 25, and personal authentication section 27.
The alertness estimation device 1a includes an input interface device 102 and an output interface device 103 that perform wired or wireless communication with a device such as the imaging device 3 or an output device (not shown).

Furthermore, in the first embodiment described above, it is assumed that the person whose arousal degree is estimated by the arousal degree estimating device 1 is the driver of the vehicle, and that there is only one person whose arousal degree is estimated. However, this is just one example. The number of people whose arousal level is estimated by the arousal level estimating device may be plural. For example, the alertness level estimating device may estimate the alertness levels of multiple occupants of the vehicle, including the driver, front passenger seat occupant, or rear seat occupant.

If the alertness level estimation device 1 has the configuration shown in FIG. 1 and does not include a personal authentication function, for example, the feature point detection unit 22 determines the seat position of each passenger based on the captured image, and determines the seat position. Detect multiple feature points at each time. Then, the feature point detection unit 22 outputs information that allows identification of each occupant and information associated with the detected plurality of feature points to the distance calculation unit 23 as feature point information. Here, the information that can identify each occupant is information regarding the seat position.
The distance calculation unit 23 calculates the distance between feature points for each passenger corresponding to the seat position determined by the feature point detection unit 22. The distance calculation unit 23 stores information in which information that allows identification of each occupant and distances between feature points to be associated with each other as distance information in the storage unit 26 and outputs the information to the face determination unit 25.
The reference distance calculation unit 24 calculates the distance between reference feature points for each passenger corresponding to the seat position determined by the feature point detection unit 22. The reference distance calculation unit 24 causes the storage unit 26 to store, as reference distance information, information in which information that allows identification of each occupant and reference inter-feature point distances are associated with each other.
The face determination unit 25 determines whether or not each occupant is wearing a restrained expression based on the distance information and the reference distance information.
The face determination unit 25 outputs, as face determination information, information in which information that allows identification of each occupant to be associated with information indicating whether or not the passenger is making a restrained face to the face determination information acquisition unit 11 .
Note that when the recalculation instruction unit 251 instructs the reference distance calculation unit 24 to recalculate the distance between reference feature points, it also outputs information that can identify each occupant. When the reference distance calculation unit 24 recalculates the distance between reference feature points, the reference distance calculation unit 24 updates the reference distance information of the corresponding occupant stored in the storage unit 26.
The estimation unit 13 estimates the degree of wakefulness for each occupant. Note that the estimation unit 13 can determine each occupant, for example, based on the feature point information output from the feature point detection unit 22 and the face determination information output from the face determination information acquisition unit 11.

For example, if the alertness estimation device 1a has a configuration as shown in FIG. 7 and is equipped with a personal authentication function, the face determination device 2a can make a patient face for each passenger based on the personal authentication results. All you have to do is determine whether or not there is one. Furthermore, the estimation unit 13 may estimate the degree of wakefulness for each occupant based on the personal authentication result.
Specifically, the feature point detection unit 22 detects a plurality of feature points for each occupant based on the personal authentication result by the personal authentication unit 27. Note that in FIG. 7, the arrow from the personal authentication section 27 to the feature point detection section 22 is omitted. The feature point detection unit 22 outputs information in which the detected feature points are associated with the personal authentication result to the distance calculation unit 23 as feature point information.
The distance calculation unit 23 calculates the distance between feature points for each occupant based on the personal authentication result by the personal authentication unit 27. The distance calculation unit 23 stores information in which the calculated inter-feature point distance and the personal authentication result are associated with each other as distance information in the storage unit 26 and outputs it to the face determination unit 25.
The reference distance calculating section calculates the distance between reference feature points for each occupant based on the personal authentication result by the personal authentication section 27. The reference distance calculation unit 24 causes the storage unit 26 to store information in which the reference inter-feature point distance and the personal authentication result are associated with each other as reference distance information.
The face determination section 25 determines whether or not each occupant is wearing a restrained expression based on the personal authentication result obtained by the personal authentication section 27 . The face determination unit 25 outputs, as face determination information, information in which the information indicating whether the person is making a restrained face and the personal authentication result are associated with each other to the face determination information acquisition unit 11 .
The estimation unit 13 estimates the degree of wakefulness for each occupant based on the personal authentication result by the personal authentication unit 27. Note that the estimation unit 13 can determine each occupant, for example, based on the feature point information output from the feature point detection unit 22 and the face determination information output from the face determination information acquisition unit 11.

In the first embodiment described above, the alertness level estimation devices 1 and 1a are in-vehicle devices installed in a vehicle, and include a face determination information acquisition section 11, a brightness detection section 12, an estimation section 13, and an image acquisition section. 21, the feature point detection unit 22, the distance calculation unit 23, the reference distance calculation unit 24, the face determination unit 25, and the personal authentication unit when the arousal level estimation device 1a is configured as shown in FIG. 27 is assumed to be provided in the in-vehicle device.
However, the present invention is not limited to this, but the face determination information acquisition section 11, the brightness detection section 12, the estimation section 13, the image acquisition section 21, the feature point detection section 22, the distance calculation section 23, the reference distance calculation section 24, A part of the face determination unit 25 and the personal authentication unit 27 is installed in the in-vehicle device of the vehicle, and the other part is provided in a server connected to the in-vehicle device via a network, so that the in-vehicle device and the server are provided. An alertness estimation system may also be configured.
Further, the face determination information acquisition unit 11, the brightness detection unit 12, the estimation unit 13, the image acquisition unit 21, the feature point detection unit 22, the distance calculation unit 23, the reference distance calculation unit 24, and the face determination unit 25 and the personal authentication section 27 may all be provided in the server.

Further, in the first embodiment described above, the face determination device 2 is included in the arousal level estimation device 1, and the face determination device 2a is included in the arousal level estimation device 1a, but this is only an example. The face determination device 2 may be connected to the arousal level estimation device 1 outside of the arousal level estimation device 1 . Further, the face determination device 2a may be connected to the arousal level estimation device 1a outside of the arousal level estimation device 1a.
Further, some of the components included in the face determination device 2 may be included in the arousal level estimation device 1, and some of the components included in the face determination device 2a may be included in the arousal level estimation device 1. It may be provided in the device 1a. For example, the image acquisition unit 21, the feature point detection unit 22, the distance calculation unit 23, or the reference distance calculation unit 24 may be included in the arousal level estimation device 1, or the image acquisition unit 21, the feature point detection unit 22 , the distance calculation section 23, the reference distance calculation section 24, or the personal authentication section 27 may be included in the alertness level estimation device 1a.

Further, in the first embodiment described above, the person whose alertness level is estimated by the alertness level estimation devices 1 and 1a is the driver of the vehicle or the occupant of the vehicle including the driver, but this is only an example. do not have. The alertness level estimation devices 1 and 1a are capable of estimating the alertness level of not only the occupants of a vehicle but also all people. For example, the alertness level estimation devices 1 and 1a can estimate the alertness level of a moving object other than a vehicle or a person present in a room. Note that in this case, the imaging device 3 is not limited to one that is mounted on a vehicle. The imaging device 3 may be installed at a position where it can image the range where the face of the person whose arousal degree is to be estimated by the arousal level estimation devices 1 and 1a is present.

As described above, according to Embodiment 1, the arousal level estimating device 1, 1a uses a face for acquiring face determination information indicating whether or not a person has a face with wrinkles between the eyebrows (patience face). A determination information acquisition unit 11 , a brightness detection unit 12 that detects the brightness in a captured image in which a range where a human face should exist, face determination information acquired by the face determination information acquisition unit 11 , and a brightness detection unit 12 The apparatus is configured to include an estimator 13 that estimates the degree of arousal of the person based on the detected brightness. Therefore, the alertness estimation device 1 estimates the alertness level of a person by considering whether the surrounding situation is such that the person would feel dazzling when the person has wrinkled eyebrows. By doing so, it is possible to prevent erroneous estimation of a decrease in a person's alertness level.

Note that it is possible to modify any component of the embodiment or omit any component of the embodiment.

The alertness estimation device of the present disclosure estimates a person's alertness level by considering whether the surrounding situation is such that the person is expected to feel dazzling when the person has furrowed eyebrows. Therefore, it is possible to prevent erroneous estimation of a decrease in a person's alertness level.

Reference Signs List 1, 1a alertness level estimation device, 11 face determination information acquisition unit, 12 brightness detection unit, 13 estimation unit, 2, 2a face determination device, 21 image acquisition unit, 22 feature point detection unit, 23 distance calculation unit, 24 reference distance calculation unit, 25 face determination unit, 251 recalculation instruction unit, 26 storage unit, 27 personal authentication unit, 3 imaging device, 101 processing circuit, 102 input interface device, 103 output interface device, 104 processor, 105 memory.

Claims (13)

a face determination information acquisition unit that acquires face determination information indicating whether a person has a face with wrinkles between the eyebrows;
a brightness detection unit that detects brightness in a captured image in which a range where a human face should exist;
An arousal level estimating device comprising: an estimating unit that estimates a degree of arousal of the person based on the face determination information acquired by the face determination information acquisition unit and the brightness detected by the brightness detection unit. The estimation unit is
The face determination information acquired by the face determination information acquisition unit is the face determination information indicating that the person has a face with wrinkles between the eyebrows, and the brightness detected by the brightness detection unit The alertness level estimating device according to claim 1, wherein if the brightness determination threshold is less than a brightness determination threshold, it is estimated that the person's alertness level has decreased. The estimation unit is
The face determination information acquired by the face determination information acquisition unit is the face determination information indicating that the person has a face with wrinkles between the eyebrows, and the brightness detected by the brightness detection unit 2. The alertness level estimating device according to claim 1, wherein the alertness level estimating device estimates that the person's alertness level has not decreased when the brightness determination threshold value is greater than or equal to a brightness determination threshold. The estimation unit is
The face determination information acquired by the face determination information acquisition unit is the face determination information indicating that the person has a face with wrinkles between the eyebrows, and the brightness detected by the brightness detection unit is less than the brightness determination threshold, if it is determined that the area around the person's face is bright based on the brightness distribution on the captured image, it is estimated that the person's alertness level has not decreased. The alertness level estimating device according to claim 2. The estimation unit is
The face determination information acquired by the face determination information acquisition unit is the face determination information indicating that the person has a face with wrinkles between the eyebrows, and the brightness detected by the brightness detection unit is greater than or equal to the brightness determination threshold, if it is determined that there is a shadow around the person's face based on the brightness distribution on the captured image, it is estimated that the person's alertness level has decreased. The alertness level estimating device according to claim 3. The estimation unit is
When the face determination information acquired by the face determination information acquisition unit is the face determination information indicating that the person does not have a face with wrinkled eyebrows, the face determination information of the person is determined based on the captured image. The alertness level estimating device according to claim 1, further comprising: estimating the alertness level. The alertness level estimation device according to claim 1, wherein the brightness detection unit sets an average value of brightness values of pixels on the captured image as the brightness of the captured image. an image acquisition unit that acquires the captured image;
a feature point detection unit that detects a plurality of feature points of the human face on the captured image acquired by the image acquisition unit;
The coordinates of the plurality of feature points detected by the feature point detection unit on the captured image are converted into three-dimensional coordinates, and the inter-feature distance between the plurality of feature points is determined from the converted three-dimensional coordinates. a distance calculation unit that calculates
By comparing the distance between the feature points calculated by the distance calculation unit and the reference distance between the feature points, it is determined whether the person has a face with wrinkles between the eyebrows, and the distance between the feature points is determined. The arousal level estimating device according to claim 1, further comprising: a face determining unit that determines that the person has a face with wrinkles between the eyebrows when the distance is smaller than the reference inter-feature point distance by a first threshold value or more. a reference distance calculation unit that calculates the reference distance between feature points based on a history of the distance between feature points calculated by the distance calculation unit for a reference calculation target period;
The arousal level estimation according to claim 8, wherein the face determination unit compares the inter-feature point distance calculated by the distance calculation unit and the reference inter-feature point distance calculated by the reference distance calculation unit. Device. If the face determination unit compares the inter-feature point distance with the reference inter-feature point distance and determines that the inter-feature point distance is greater than the reference inter-feature point distance by a second threshold value or more, the reference distance comprising a recalculation instruction unit that instructs the calculation unit to recalculate the distance between the reference feature points,
The alertness level according to claim 9, wherein the reference distance calculation unit recalculates the reference inter-feature point distance when instructed to recalculate the reference inter-feature point distance from the recalculation instruction unit. Estimation device. comprising a personal authentication unit that performs personal authentication of the person,
The reference distance calculation unit calculates the distance between the reference feature points for each individual based on the result of the individual authentication by the individual authentication unit,
The face determination unit compares the reference feature with the distance between feature points based on the result of the personal authentication by the individual authentication unit and the reference distance between feature points for each individual calculated by the reference distance calculation unit. The alertness level estimation device according to claim 9, characterized in that a distance between points is specified. The arousal level estimation device according to claim 8, wherein the plurality of feature points are the tips of both eyebrows of the person, and the distance between the feature points is a distance between the tips of the eyebrows of the person. a step in which the face determination information acquisition unit acquires face determination information indicating whether the person has a face with wrinkles between the eyebrows;
a step in which the brightness detection unit detects brightness in a captured image in which a range where a human face should exist;
An arousal degree estimation method, comprising: an estimating section estimating the arousal degree of the person based on the face determination information acquired by the face determination information acquisition section and the brightness detected by the brightness detection section.

Images