JP6689470B1 - Information processing apparatus, program, and information processing method - Google Patents

Abstract

A posture holding function evaluation unit (102) that evaluates the state of the posture holding function that is the function of the driver to hold the posture from an image including the driver's head, and the driver's arousal level using the evaluation. And an arousal level estimation unit (106) for estimating the level.

Description

  The present invention relates to an information processing device, a program, and an information processing method.

  Driver monitoring system has been put to practical use. The driver monitoring system is not limited to commercial vehicles such as trucks and buses, but is widely used in general vehicles.

  The main functions installed in the driver monitoring system are a safe driving support function and a detection function for a dozing driving and a side-looking driving. Further, in the automatic driving Lv (Level) 3, since the authority is transferred from the automatic driving system to humans, these functions are required as a technique for determining whether the driver is in a drivable state.

  In the related art, a driver's face is photographed by a camera mounted in the vehicle, and the driver's condition is estimated from the facial expression and the like. For example, as a technique for estimating drowsiness, a technique for detecting the degree of eye opening is often used, as described in Patent Document 1.

JP 61-175129 A

  It is difficult for the conventional technology for detecting drowsiness to constantly detect the drowsiness of the driver while traveling. For example, in the related art, an infrared camera is often used, but when the driver wears sunglasses that shield infrared rays, the state of the eyes cannot be seen, and thus the degree of eyelid opening can be detected. Can not.

  Therefore, it is an object of the present invention to be able to detect the level of the driver's arousal level even when the driver's eyes are hidden.

An information processing apparatus according to an aspect of the present invention includes a head coordinate detection unit that detects head coordinates that are coordinates of the driver's head from an image including a driver's head; A head tilt angle detection unit that detects a head tilt angle that is a tilt angle of the driver's head, a dispersion of the head coordinates in a predetermined period, and the head tilt in the predetermined period. An evaluation unit that evaluates the state of the posture maintaining function, which is a function of the driver to hold the posture, and the arousal level estimation that estimates the level of the driver's arousal level using the evaluation based on the variance of the angles. And a section.

A program according to an aspect of the present invention causes a computer to detect, from an image including a driver's head, a head coordinate detection unit that detects head coordinates that are coordinates of the driver's head, from the image, A head tilt angle detection unit that detects a head tilt angle that is a tilt angle of a driver's head, a variance of the head coordinates in a predetermined period, and the head tilt angle in the predetermined period. And an evaluation unit that evaluates the state of the posture holding function that is a function of the driver to hold the posture, and awakening degree estimation that estimates the level of the awakening degree of the driver using the evaluation. It is characterized in that it functions as a section.

In the information processing method according to an aspect of the present invention, the head coordinate detection unit detects head coordinates that are the coordinates of the driver's head from an image including the driver's head, and calculates the head inclination angle. A detection unit detects a head inclination angle that is an inclination angle of the driver's head from the image, and an evaluation unit disperses the head coordinates in a predetermined period, and the predetermined From the variance of the head inclination angle during the period, the state of the posture holding function, which is a function of the driver to hold the posture, is evaluated, and the awakening degree estimation unit uses the evaluation to awaken the driver. It is characterized by estimating the level of degrees.

  According to one or more aspects of the present invention, it is possible to detect the level of the driver's arousal level even when the driver's eyes are hidden.

FIG. 3 is a block diagram schematically showing the configuration of the information processing device according to the first embodiment. FIG. 3 is a schematic diagram for explaining a coordinate system in the first embodiment. FIG. 3 is a schematic diagram showing an arrangement example of the information processing device according to the first embodiment in a vehicle. FIG. 3 is a block diagram showing a hardware configuration example of the information processing device according to the first embodiment. 5 is a flowchart showing the operation of the information processing device according to the first embodiment. 6 is a block diagram schematically showing the configuration of an information processing device according to a second embodiment. FIG. FIG. 6 is a schematic diagram showing an arrangement example of an information processing device according to a second embodiment in a vehicle. 7 is a flowchart showing the operation of the information processing apparatus according to the second embodiment.

Embodiment 1.
FIG. 1 is a block diagram schematically showing the configuration of an information processing device 100 which is the awakening level estimation device according to the first embodiment.
As shown in FIG. 1, the information processing apparatus 100 is connected to the imaging device 130 and the awakening degree lowering notification device 140.

The information processing apparatus 100 receives the image data indicating the image captured by the image capturing apparatus 130, estimates the level of the driver's arousal level, and determines that the driver's arousal level is reduced by the estimated level. When the determination is made, the alertness lowering notification device 140 is notified.
The imaging device 130 may use, for example, a CCD (Charge Coupled Device) or the like, and the alertness lowering notification device 140 may use a display, a speaker, or the like.

The information processing apparatus 100 includes an input interface unit (hereinafter referred to as an input I / F unit) 101, a posture holding function evaluation unit 102, an awakening degree estimation unit 106, and an output interface unit (hereinafter referred to as an output I / F unit). And 107.
The posture holding function evaluation unit 102 includes a head coordinate detection unit 103, a head inclination angle detection unit 104, and an evaluation unit 105.

  The input I / F unit 101 is an input unit that receives input of image data from the imaging device 130. The input I / F unit 101 gives the input image data to the head coordinate detection unit 103 and the head inclination angle detection unit 104.

  Here, the image represented by the image data may include one driver or another occupant. The image may be an RGB (Red Green Blue) image, an IR (infrared) image, or a distance image as long as the head coordinates and the head tilt angle can be detected.

  The posture holding function evaluation unit 102 evaluates the state of the posture holding function, which is the function of the driver holding the posture, from the image represented by the image data.

  The head coordinate detection unit 103 detects the head coordinates that are the coordinates of the driver's head from the image represented by the image data given from the input I / F unit 101. The head coordinate detection unit 103 gives the detected head coordinates to the evaluation unit 105.

For example, the head coordinate detection unit 103 detects the driver's face from the image represented by the image data provided from the input I / F unit 101.
The face detection method for detecting the driver's face is not particularly limited.
Specifically, the head coordinate detection unit 103 may detect the driver's face using an AdaBoost-based detector that uses a commonly used Haar-like feature amount.

  It should be noted that the “Haar-like feature amount” is a feature amount that captures the feature based on the difference in brightness of the image, and is less susceptible to fluctuations in illumination conditions or noise, as compared with the case where the pixel value is used as the feature amount. Further, “AdaBoost” is an abbreviation of Adaptive Boosting, which is a machine learning algorithm with improved performance by using a plurality of weak classifiers in combination.

Next, the head coordinate detection unit 103 detects the head coordinates that are the coordinates of the driver's head from the face detection frame that is the area including the detected face.
The calculation method of the head coordinates may be the center of the face detection frame, a representative point of the head may be set from the feature points of the face, and the representative point may be used as the head coordinates. For example, the midpoint between the left and right eyes may be the representative point. However, there is no characteristic edge between the eyes, or because the edge is created by wearing glasses, the midpoint of the left and right eyes is not directly detected, and the midpoint of the left and right inner canthus, or The representative point may be estimated by setting the midpoint of the outer corner of the eye.

  The “face detection frame” is a frame indicating a region where a face is present in the image, and includes feature points of the face. In most cases, the frame has a square shape. The “face feature point” is a coordinate that constitutes face-likeness, and indicates a coordinate such as a contour, a nostril, an outer corner of the eye, an inner corner of the eye, a corner of the mouth, a lip or an eyebrow.

  The head coordinate detecting unit 103 gives face area information indicating the detected face detection frame to the head inclination angle detecting unit 104.

  The head tilt angle detection unit 104 detects the head tilt angle, which is the tilt angle of the driver's head, from the image represented by the image data provided from the input I / F unit 101. The head tilt angle detection unit 104 gives the detected head tilt angle to the evaluation unit 105.

  For example, the head inclination angle detection unit 104, in the image shown by the image data given from the input I / F unit 101, the image in the face detection frame shown by the face area information given by the head coordinate detection unit 103. The tilt angle of the driver's head is detected from the face area image.

Specifically, as a method of detecting the tilt angle of the head, the head tilt angle detection unit 104 creates a detector from an image taken every 10 degrees of the head tilt angle, The head inclination angle may be detected by using the detector.
Further, the head tilt angle detection unit 104 may specify a plurality of feature points from the face area image and detect the head tilt angle from the distance between the feature points. For example, the head inclination angle detection unit 104 may specify two feature points in the vertical direction of the face area image and detect the head inclination angle based on the distance between the two feature points. For example, as the head tilt angle increases, the distance between the two feature points decreases. In this case, if the image data captured by a stereo camera or a TOF (Time of Flight) camera whose distance is required is used, the distance can be accurately and easily obtained.

The evaluation unit 105 calculates the variation of the head coordinates detected by the head coordinate detection unit 103 during a predetermined period and the head tilt angle detected by the head tilt angle detection unit 104 during the predetermined period. The variation is used to evaluate the state of the posture holding function of the driver.
Here, the evaluation unit 105 uses the variance of the head coordinates detected by the head coordinate detection unit 103 in a predetermined period and the head detected by the head inclination angle detection unit 104 in the predetermined period. The state of the posture holding function of the driver is evaluated by using the inclination angle variance.

For example, the evaluation unit 105 calculates the variance of the head coordinates in a predetermined period. Specifically, the evaluation unit 105 calculates the variance of head coordinates for the last 20 seconds. In this case, assuming that the image capturing apparatus 130 inputs image data at 30 fps (frame per second), the evaluation unit 105 records 600 frames of the head coordinates in the gravity direction of the earth and calculates the variance of the coordinates. . As shown in FIG. 2, the gravity direction of the earth is the Y axis here. The frame rate of the image capturing apparatus 130 and the time for calculating the variance are not limited to this example.
In the example shown in FIG. 2, the direction around the X axis is the Pitch direction, the direction around the Y axis is the Yaw direction, and the direction around the Z axis is the Roll direction.

  Here, the evaluation unit 105 calculates the variance of the coordinates in the gravity direction of the earth, but may similarly calculate the variance of the coordinates in the horizontal direction with respect to the ground. Note that, as shown in FIG. 2, the horizontal direction to the ground is the X axis here. When estimating the state of the posture holding function described later from the distribution of the coordinates in the horizontal direction, the corresponding head tilt angle is the Yaw direction or the Roll direction.

  Further, the evaluation unit 105 calculates the variance of the head inclination angle in a predetermined period. Specifically, the evaluation unit 105 calculates the variance of the head inclination angle for the latest 20 seconds. In this case, assuming that the imaging device 130 inputs the image data at 30 fps, the evaluation unit 105 records the angle in the Pitch direction, which is the inclination angle of the earth in the gravity direction of 600 frames, and calculates the variance.

  Here, the evaluation unit 105 records the angle in the Pitch direction and calculates the variance, but as described above, the angle may be recorded in the Yaw direction or the Roll direction to calculate the variance.

  Then, the evaluation unit 105 calculates an evaluation value indicating the state of the posture holding function from the variance of the head coordinates and the variance of the head inclination angle. For example, the evaluation unit 105 divides the variance of the head coordinates in the predetermined period by the variance of the head inclination angle in the predetermined period, or calculates the head inclination angle in the predetermined period. A value obtained by dividing the variance by the variance of the head coordinates in a predetermined period is calculated as an evaluation value.

Specifically, the evaluation unit 105 calculates the evaluation value by the following equation (1).
Evaluation value = (dispersion of Pitch angle) ÷ (dispersion of Y coordinate) (1)
In addition, the evaluation unit 105 may calculate the evaluation value by the following equation (2) or equation (3).
Evaluation value = (dispersion of X coordinate) ÷ (dispersion of Yaw angle) (2)
Evaluation value = (dispersion of Roll angle) ÷ (dispersion of X coordinate) (3)
The evaluation unit 105 provides the arousal level estimation unit 106 with the evaluation value calculated as described above.

For example, in the expression (1), when the evaluation value is large, the head position is not changed and the head is swinging in the up-and-down direction. It can be judged that it is not being exerted.
Even in the expression (2), when the evaluation value becomes large, it is judged that the driver's arousal level is lowered and the posture holding function is not exerted because the head is swaying left and right without rotating. You can
Also in the expression (3), when the evaluation value becomes large, the head is swinging left and right without changing the position of the head, so that the driver's arousal level is lowered and the posture maintaining function is exerted. It can be judged that there is no state.

The arousal level estimation unit 106 estimates the level of the driver's arousal level using the evaluation performed by the posture holding function evaluation unit 102.
For example, the arousal level estimation unit 106 estimates the level of the driver's arousal level based on the evaluation value provided by the evaluation unit 105.
The level of arousal level may be divided into 6 levels that can be estimated from facial expressions or body movements, as in a known evaluation method defined by NEDO (New Energy and Industrial Technology Development Organization), or arbitrary arousal level. You may divide into degrees.

  When the arousal level is divided into levels by the NEDO evaluation method, the arousal level estimation unit 106 estimates the level of the arousal level by providing a threshold value of the evaluation value for each level. In other words, it is assumed that the evaluation value has a corresponding level specified in advance. Then, when the estimated level of the arousal level is lower than a predetermined threshold value, the awakening level estimation unit 106 issues a notification instruction from the output I / F unit 107 to the awakening level decrease notification device 140.

The output I / F unit 107 is an output unit for outputting an instruction or the like to the alertness lowering notification device 140.
The alertness lowering notification device 140 notifies the driver and fellow passengers of the lowered alertness in response to the notification instruction from the alertness estimation unit 106. For example, the wakefulness lowering notification device 140 notifies the driver and the like of a decrease in wakefulness using a speaker and a display, and indicates that the driver is in a state of being incapable of taking a break or taking a break.

FIG. 3 is a schematic diagram showing an arrangement example of the information processing apparatus 100 according to the first embodiment in the vehicle 150.
Inside the vehicle 150, an in-vehicle camera 151 functioning as the imaging device 130, an information processing device 100, a storage device 152, a speaker 153 functioning as the alertness lowering notification device 140, and a display 154 are arranged.

  Here, the vehicle-mounted camera 151 captures an image of the upper body including the driver's head to be detected. The vehicle-mounted camera 151 is connected to the information processing device 100. The vehicle-mounted camera 151 shown in FIG. 3 may be connected via a wiring such as a wire harness or may be connected wirelessly.

Further, the vehicle-mounted camera 151 may be a general RGB camera or an IR camera. Further, the vehicle-mounted camera 151 may be a depth sensor.
An "RGB camera" is a camera that communicates signals of three colors of red, green, and blue by using three different cables and the like, and generally uses three independent CCD sensors.
An "IR camera" is an infrared camera, which is sensitive to wavelengths in the infrared region.
The “Depth sensor” is also called a 3D (three dimensions) sensor, a distance measurement sensor, or a depth camera, and is a sensor that can generate a depth image in addition to a two-dimensional image. The distance measuring method includes a stereo camera method, a pattern irradiation method, a TOF method, etc., but any type of sensor may be used.

FIG. 4 is a block diagram showing a hardware configuration example of the information processing device 100 according to the first embodiment.
The information processing apparatus 100 includes a CPU 160, a program memory 161, a data memory 162, an interface 163, and a bus 164 connecting these.
An in-vehicle camera 151, a speaker 153, and a display 154 are connected to the interface 163.

Here, the head coordinate detection unit 103, the head inclination angle detection unit 104, the evaluation unit 105, and the awakening degree estimation unit 106 can be configured by the CPU 160 executing a program stored in the program memory 161. .
The input I / F unit 101 and the output I / F unit 107 can be configured by the interface 163.
As described above, the information processing device 100 can be realized by a computer that stores a specific program.

Note that "CPU" is an abbreviation for Central Processing Unit, and is a device that performs calculation and control.
The CPU 160 operates according to the program stored in the program memory 161. The CPU 160 stores various data in the data memory 162 in the process of operation. As a result of recognition of the image input from the vehicle-mounted camera 151 through the interface 163, the speaker 153 and the display 154 are controlled through the interface 163.
The operation performed by the information processing device 100 will be described below with reference to FIG.

FIG. 5 is a flowchart showing the operation of the information processing device 100 according to the first embodiment.
The flowchart shown in FIG. 5 includes an image data acquisition step (S10), a face detection step (S11), a head coordinate detection step (S12), a head coordinate variance calculation step (S13), and a head. It has an inclination angle detection step (S14), a head inclination angle dispersion calculation step (S15), an evaluation value calculation step (S16), an awakening degree estimation step (S17), and an awakening degree reduction notification step (S18). .

  In the image data acquisition step S10, the input I / F unit 101 acquires image data of an image of the upper body including the driver's head.

  In the face detection step S11, the head coordinate detection unit 103 detects the driver's face from the image represented by the acquired image data. It should be noted that the head coordinate detection unit 103 gives face area information indicating a face detection frame that is an area including the detected face to the head inclination angle detection unit 104.

In the head coordinate detection step S12, the head coordinate detection unit 103 detects the head coordinates from the face detection frame that is the area including the face detected in the face detection step S11.
In the head coordinate variance calculation step S13, the evaluation unit 105 calculates the variance of the head coordinates.

In the head inclination angle detection step S14, the head inclination angle detection unit 104 detects the head inclination angle from the face detection frame that is the area including the face detected in the face detection step S11.
In the head tilt angle variance calculation step S15, the evaluation unit 105 calculates the variance of the head tilt angle detected in the head tilt angle detection step S14.

In the evaluation value calculation step S16, the evaluation unit 105 calculates an evaluation value indicating the state of the posture holding function from the variance of the head coordinates and the variance of the head inclination angle.
In arousal level estimation step S17, the arousal level estimation unit 106 estimates the level of arousal level from the evaluation value calculated in the evaluation value calculation step S16.

  In awakening degree decrease notification step S18, the awakening degree estimating unit 106 outputs the output I / F unit 107 via the output I / F unit 107 when the level of the awakening degree estimated in the awakening degree estimating step S17 is below a predetermined threshold value. The awakening degree lowering notification device 140 is instructed to notify the driver or the like of the lowering of the awakening degree. Receiving such an instruction, the alertness lowering notification device 140 notifies the driver, the passenger, and the like of the lowered alertness by using at least one of the speaker 153 and the display 154, and prompts a break. Indicates that the operation cannot be transferred.

  As described above, according to the first embodiment, the level of the driver's arousal level can be estimated from the state of the posture maintaining function. In addition, it is possible to notify the driver and fellow passengers according to the estimation result.

In the first embodiment, since the level of the arousal level is estimated by evaluating the state of the posture holding function of the head from the coordinates of the head and the inclination angle of the head, the arousal level is awake even when the eyes are hidden. The level of degree can be estimated.
Further, since a sensor other than the camera is not necessary for estimating the arousal level, cost reduction and downsizing are possible.
Further, since the inclination angle of the head is used for estimation, it is possible to separate whether the movement of the head is caused by the turning direction of the face or the vibration of the vehicle. As a result, it is possible to prevent erroneous detection due to the movement of the head caused by the turning direction or the like at the time of safety confirmation.

Embodiment 2.
FIG. 6 is a block diagram schematically showing the configuration of the information processing device 200 which is the awakening level estimation device according to the second embodiment.
As shown in FIG. 6, the information processing device 200 is connected to the imaging device 130 and the alertness lowering notification device 140. The imaging device 130 and the alertness lowering notification device 140 according to the second embodiment are the same as the imaging device 130 and the alertness lowering notification device 140 according to the first embodiment.

  The information processing device 200 according to the second embodiment is also connected to the CAN 270. "CAN" is an abbreviation for Controller Area Network, and is a network corresponding to a standard for connecting electronic circuits and respective devices developed as a communication technology between vehicle-mounted devices.

The information processing device 200 includes an input I / F unit 201, a posture holding function evaluation unit 102, an awakening degree estimation unit 206, an output I / F unit 107, a PERCLOS calculation unit 208, and a yawn detection unit 209. .
The posture holding function evaluation unit 102 includes a head coordinate detection unit 103, a head inclination angle detection unit 104, and an evaluation unit 105.

The posture holding function evaluation unit 102 and the output I / F unit 107 in the second embodiment are the same as the posture holding function evaluation unit 102 and the output I / F unit 107 in the first embodiment.
However, the head coordinate detection unit 103 of the posture holding function evaluation unit 102 applies the face area information indicating the detected face detection frame to the PERCLOS calculation unit 208 and the yawn detection unit 209 as well as the head inclination angle detection unit 104. give.

The input I / F unit 201 receives input of image data from the imaging device 130. The input I / F unit 201 gives the input image data to the head coordinate detection unit 103, the head inclination angle detection unit 104, the PERCLOS calculation unit 208, and the yawn detection unit 209.
The input I / F unit 201 also receives, from the CAN 270, vehicle information such as the steering angle of the steering wheel, the position of the gear, or the vehicle speed, which indicates the state of the vehicle driven by the driver. The input I / F unit 201 gives the input vehicle information to the awakening degree estimation unit 206.

The PERCLOS calculation unit 208 detects the degree of eye opening, which is the rate at which the driver's eyes are open, from the image represented by the image data provided from the input I / F unit 201, and according to the degree of eye opening, the driver in a certain period of time. Is a closed-eye ratio calculation unit that calculates a closed-eye ratio, which is the ratio of the time the eyes are closed.
Here, the PERCLOS calculation unit 208 calculates the PERCLOS of the driver. PERCLOS is an abbreviation for “percentage of eye close”, and is the ratio of eye-closing time within a certain period of time. PERCLOS is known to have a high correlation with the subjective drowsiness of the subject (driver in this case), and can be used as a drowsiness index.

For example, the PERCLOS calculation unit 208, among the images represented by the image data provided by the input I / F unit 201, the face area image corresponding to the face detection frame represented by the face area information provided by the head coordinate detection unit 103. In, the degree of eye opening is calculated from the distance between the upper and lower eyelids. Specifically, the PERCLOS calculation unit 208 obtains the midpoint between the inner and outer canthus of the upper and lower eyelids on the arc of the upper and lower eyelids, respectively, and calculates the ratio of the distance between the upper and lower eyelids to the distance between the upper and lower eyelids when the eye is maximally opened. The degree of eye opening.
It should be noted that the distance between the upper and lower eyelids with respect to the distance between the upper and lower eyelids at the time of maximum eye opening is set in advance. Therefore, the definition of the degree of eye opening is calculated for each frame, with the degree of eye opening when the eyes are closed being 0% and the degree of eye opening when awakening being 100%.

  Then, the PERCLOS calculation unit 208 calculates PERCLOS from the calculated eye opening degree. The PERCLOS calculator 208 determines whether the calculated degree of eye opening is eye-opening or eye-closing with a predetermined threshold for each frame, and when it is determined that the eye-closing degree is closed, the PERCLOS calculating unit 208 integrates the eye-opening frame. For example, when the PERCLOS calculation unit 208 is provided with a buffer for 1 minute and calculates PERCLOS from an image captured by the imaging device 130 at 10 fps, the PERCLOS calculation unit 208 divides the number of closed eye frames accumulated in 1 minute by 6000 frames. , PERCLOS can be calculated.

  The yawning detection unit 209 detects the degree of opening, which is the ratio of the driver's mouth being open, from the image represented by the image data provided from the input I / F unit 201, and the driver yawns according to the degree of opening. It is detected whether or not.

  For example, the yawning detection unit 209 selects the face area image corresponding to the face detection frame indicated by the face area information given by the head coordinate detection unit 103 among the images shown by the image data given by the input I / F unit 201. In, the degree of mouth opening is detected from the feature points of the face from the distance between the upper and lower lips. Specifically, the yawning detecting unit 209 obtains the midpoint between the left and right mouth angles on the arc of the upper lip and the lower lip, and sets the distance between the upper and lower midpoints at the maximum opening as the opening degree. calculate.

  Then, the yawn detecting unit 209 detects the yawn of the driver based on the calculated opening degree. As a characteristic of yawning, the mouth opens continuously for a certain period of time, the nostrils widen, tears occur, or inhalation is performed before a yawn occurs, but here the mouth opens continuously for a certain period of time. The case where the features are used will be described.

  Specifically, the yawn detection unit 209 determines that the corresponding frame is a yawn frame when the calculated opening degree is equal to or more than a predetermined threshold value. Then, the yawn detection unit 209 determines that the yawn frame is a yawn when the yawn frame continues for a predetermined time or more, which is a threshold value. Further, since yawning does not occur continuously in a short time, the yawning detecting unit 209 does not detect yawning for a certain period of time after detecting yawning once. This can prevent erroneous detection.

  The awakening degree estimation unit 206 includes vehicle information provided from the input I / F unit 201, an evaluation value provided from the evaluation unit 105, a PERCLOS provided from the PERCLOS calculation unit 208, and a yawn presence / absence provided from the yawn detection unit 209. Then, the driver's arousal level is estimated.

For example, the awakening degree estimation unit 206 refers to the vehicle state indicated by the vehicle state information and estimates the driver's awakening level when the vehicle state does not correspond to a predetermined condition.
Here, the predetermined condition is that when the back or the parking is selected as the gear, the vehicle speed is lower than the predetermined threshold value, or the steering angle is higher than the predetermined threshold value. Is. When none of these conditions are met, the arousal level estimation unit 206 estimates the level of the arousal level.

  By considering the above predetermined conditions, in the second embodiment, the awakening degree estimation unit 206 causes the vehicle state indicated by the vehicle state information to travel straight ahead when the vehicle speed is equal to or higher than the predetermined vehicle speed. If so, the driver's arousal level is estimated.

  Then, when the awakening degree estimation unit 206 determines to estimate the level of the driver's awakening degree, the vehicle information provided from the input I / F unit 201, the evaluation value provided from the evaluation unit 105, and the PERCLOS calculation unit. The level of the driver's arousal level is estimated from the PERCLOS provided by 208 and the presence or absence of a yawn provided by the yawn detector 209.

  The awakening degree estimation unit 206 receives the vehicle information provided from the input I / F unit 201, the evaluation value provided from the evaluation unit 105, the PERCLOS provided from the PERCLOS calculation unit 208, and the yawning provided from the yawn detection unit 209. The level of arousal level corresponding to each of the presence and absence is estimated. Then, the alertness estimation unit 206 adopts the lowest alertness level at the same timing as the alertness level at that timing.

Here, the change over time of the steering angle of the steering wheel is used as the vehicle information. It is known that the steering angle of the steering changes finely when the level of arousal level decreases, and this index is used. That is, the level of the awakening level may be associated in advance according to the magnitude of the change over time of the steering angle.
Further, with respect to PERCLOS, it is sufficient that the values of the arousal level are associated with the respective values in advance, and the presence or absence of a yawn may be associated with the levels of the arousal level in advance.

Here, the level of the awakening degree corresponding to the evaluation value is the first level, the level of the awakening degree corresponding to the state of the vehicle indicated by the vehicle information is the second level, and the level of the awakening degree corresponding to PERCLOS is the third level. And the level of the awakening degree corresponding to whether or not yawning is the fourth level, the awakening degree estimation unit 206 determines the first level, the second level, the third level, and the third level. The final level of arousal level is specified from the level of 4. However, when the third level cannot be acquired due to wearing sunglasses, etc., the final level of arousal is specified from the first level, the second level, and the fourth level. . Here, of these levels, the one with the lowest alertness is specified as the final alertness level.

  As for the level of arousal level, the NEDO evaluation method may be used as in the first embodiment, but any level of arousal level may be used. It is desirable to calculate the level of alertness for each index together with the reliability. The reliability is an index indicating whether the output value is a likely value. The value output when the reliability is high is a likely value, and the value output when the reliability is low is adjusted to be erroneous in many cases.

  As the method of estimating the arousal level, the lowest level among the indices whose reliability is higher than a predetermined threshold may be adopted. As a result, even a driver whose eyes do not have open / closed eyes can correctly determine the level of the arousal level based on the yawn or the variation in the steering angle.

  That is, it is known that the method of estimating the level of the driver's arousal level from the degree of eyelid opening is effective, and that the PERCLOS increases when the level of the driver's arousal level decreases. A general method is to detect an open / closed eye using an image pickup device 130 that is sensitive to infrared rays, but the state of the eyes cannot be detected when wearing sunglasses with low infrared ray transmittance. However, according to the method described above, the level of the arousal level due to PERCLOS is not estimated when wearing sunglasses, but the level estimated by another method can be applied.

FIG. 7 is a schematic diagram showing an arrangement example of information processing apparatus 200 according to the second embodiment in vehicle 150.
Inside the vehicle 150, an in-vehicle camera 151 functioning as the image capturing device 130, an information processing device 200, a storage device 152, a speaker 153 and a display 154 functioning as the alertness lowering notification device 140, a vehicle speed sensor 255, A gear box 256 and a steering wheel 257 are arranged.

The vehicle-mounted camera 151, the storage device 152, the speaker 153, and the display 154 in the second embodiment are the same as the vehicle-mounted camera 151, the storage device 152, the speaker 153, and the display 154 in the first embodiment.
In the second embodiment, information that can be acquired from the vehicle speed sensor 255, the gear box 256, and the steering wheel 257 is sent to the information processing device 200 via the CAN 270 as vehicle information.

As shown in FIG. 4, the information processing device 200 according to the second embodiment also includes a CPU 160, a program memory 161, a data memory 162, an interface 163, and a bus 164 connecting these.
An in-vehicle camera 151, a speaker 153, and a display 154 are connected to the interface 163.
It should be noted that in the second embodiment, the interface 163 is also connected to the CAN 270.

FIG. 8 is a flowchart showing the operation of the information processing device 200 according to the second embodiment.
The flowchart shown in FIG. 8 includes an image data acquisition step (S20), a face detection step (S21), a head coordinate detection step (S22), a head coordinate variance calculation step (S23), and a head. Inclination angle detection step (S24), head inclination angle variance calculation step (S25), eye opening degree detection step (S26), PERCLOS calculation step (S27), aperture degree detection step (S28), and yawn detection step. (S29), evaluation value calculation step (S30), awakening degree estimation step (S31), and awakening degree decrease notification step (S32).

  In the image data acquisition step S20, the input I / F unit 201 acquires image data of an image of the upper body including the driver's head. The image represented by the image data is an RGB image, an IR image, or an IR image if the calculation of the head coordinates, the calculation of the head inclination angle, the calculation of PERCLOS, and the detection of the yawn that are performed in the subsequent processing are possible. It may be any image such as a range image.

  In the face detection step S21, the head coordinate detection unit 103 detects the face of the driver from the image represented by the acquired image data. The head coordinate detection unit 103 gives face area information indicating a detection frame that is an area including the detected face to the head inclination angle detection unit 104, the PERCLOS calculation unit 208, and the yawn detection unit 209.

In the head coordinate detection step S22, the head coordinate detection unit 103 detects the head coordinates from the face detection frame that is the area including the face detected in the face detection step S21.
In the head coordinate variance calculation step S23, the evaluation unit 105 calculates the variance of the head coordinates.

In the head inclination angle detection step S24, the head inclination angle detection unit 104 detects the head inclination angle from the face detection frame that is the area including the face detected in the face detection step S21.
In the head tilt angle variance calculation step S25, the evaluation unit 105 calculates the variance of the head tilt angle detected in the head tilt angle detection step S24.

In the eye opening degree detecting step S26, the PERCLOS calculating unit 208 uses the feature point of the driver's face to detect the eye opening degree, which is the degree of eye opening, from the face detection frame that is the area including the face detected in the face detecting step S21. To detect.
In the PERCLOS calculating step S27, the PERCLOS calculating unit 208 calculates PERCLOS from the eye opening degree detected in the eye opening degree detecting step S26.

In the opening degree detection step S28, the yawning detection unit 209 uses the feature points of the driver's face from the face detection frame, which is the area including the face detected in the face detection step S21, to determine the degree of opening of the mouth. Detect the degree.
In the yawn detection step S29, the yawn detection unit 209 detects the presence or absence of a yawn, using the opening degree detected in the opening degree detection step S28.

  In the evaluation value calculation step S30, the evaluation unit 105 calculates an evaluation value indicating the state of the posture holding function from the variance of the head coordinates and the variance of the head inclination angle.

  In awakening degree estimation step S31, the awakening degree estimation unit 206 refers to the vehicle state indicated by the vehicle state information, and estimates the level of the driver's awakening degree when the vehicle state does not correspond to a predetermined condition. . For example, the awakening degree estimation unit 206 includes the vehicle information provided from the input I / F unit 201, the evaluation value provided from the evaluation unit 105, the PERCLOS provided from the PERCLOS calculation unit 208, and the yawn provided from the yawn detection unit 209. The level of the driver's arousal level is estimated from the presence or absence of.

  In awakening degree decrease notification step S32, the awakening degree estimating unit 206 outputs the output I / F unit 107 through the output I / F unit 107 when the level of the awakening degree estimated in the awakening degree estimating step S31 falls below a predetermined threshold value. The awakening degree lowering notification device 140 is instructed to notify the driver or the like of the lowering of the awakening degree. Receiving such an instruction, the alertness lowering notification device 140 notifies the driver, passenger, etc. of the lowered alertness by using at least one of the speaker 153 and the display 154, and prompts a break. Indicates that the operation cannot be transferred.

As described above, according to the second embodiment, the evaluation can be performed using an index suitable for evaluating the level of the driver's arousal level.
Further, by controlling the speaker 153 and the display 154 according to the result of the evaluation, it is possible to notify the driver and the fellow passenger of the awakening degree.

  100, 200 Information processing device, 101, 201 Input I / F unit, 103 Head coordinate detection unit, 104 Head tilt angle detection unit, 105 Evaluation unit, 106, 206 Arousal level estimation unit, 107 Output I / F unit, 208 PERCLOS calculation unit, 209 yawn detection unit, 130 imaging device, 140 awakening degree deterioration notification device, 150 vehicle, 151 vehicle-mounted camera, 152 storage device, 153 speaker, 154 display, 160 CPU, 161, program memory, 162 data memory, 163 Interface, 164 bus.

Claims (8)

From an image including the driver's head, a head coordinate detection unit that detects head coordinates that are the coordinates of the driver's head,
From the image, a head tilt angle detection unit that detects a head tilt angle that is a tilt angle of the driver's head,
From the variance of the head coordinates in a predetermined period and the variance of the head inclination angle in the predetermined period , the state of the posture holding function, which is the function of the driver to hold the posture, is evaluated. The evaluation department to do,
An awakening degree estimating unit that estimates the level of the driver's awakening degree using the evaluation, the information processing apparatus. The evaluation unit is a value obtained by dividing the variance of the head coordinates in the predetermined period by the variance of the head inclination angle in the predetermined period, or the head in the predetermined period. The information processing apparatus according to claim 1, wherein a value obtained by dividing a variance of the inclination angle by a variance of the head coordinates in the predetermined period is calculated as an evaluation value. The information processing apparatus according to claim 2, wherein the level corresponding to the evaluation value is specified in advance. An input unit that receives an input of vehicle state information indicating a vehicle state, which is a state of a vehicle driven by the driver,
From the image, the degree of eye opening, which is the rate at which the driver's eyes are open, is detected, and the degree of eye opening is calculated according to the degree of eye opening, which is the rate of the time the driver closes his eyes in a certain period of time. A ratio calculation unit,
From the image, to detect the opening degree that is the ratio of the driver's mouth is open, according to the opening degree, a yawn detecting unit for detecting whether the driver is yawning, further comprising:
A first level of the awakening degree corresponding to the evaluation, a second level of the vehicle state, a third level of the awakening degree of the eye-closing rate, and whether or not the yawning is performed. The fourth level of the arousal level corresponding to is predetermined,
The awakening level estimation unit, the first level, the second level, the third level, and from the fourth level, in claim 1 or 2, characterized in that identifying the level The information processing device described. The arousal level estimation unit identifies, as the level, the lowest level of the arousal level among the first level, the second level, the third level, and the fourth level. The information processing device according to claim 4. The awakening degree estimation unit specifies the level when the vehicle state indicates that the vehicle state is straight ahead at a vehicle speed equal to or higher than a predetermined vehicle speed. 5. The information processing device according to item 5. Computer,
From an image including the driver's head, a head coordinate detection unit that detects head coordinates that are the coordinates of the driver's head,
From the image, a head inclination angle detection unit that detects a head inclination angle that is the inclination angle of the driver's head,
From the variance of the head coordinates in a predetermined period and the variance of the head inclination angle in the predetermined period , the state of the posture holding function, which is the function of the driver to hold the posture, is evaluated. Evaluation section to do, and
A program that causes the driver to function as a wakefulness estimation unit that estimates the level of wakefulness of the driver using the evaluation. The head coordinate detection unit detects head coordinates which are coordinates of the driver's head from an image including the driver's head,
The head tilt angle detection unit detects a head tilt angle that is a tilt angle of the driver's head from the image,
Based on the variance of the head coordinates in a predetermined period , and the variance of the head inclination angle in the predetermined period, the evaluation unit has a posture holding function that is a function of holding the posture of the driver. Evaluate the condition,
An information processing method, wherein the arousal level estimation unit estimates the level of the driver's arousal level using the evaluation.

Images