JPWO2020255238A1 - Information processing equipment, programs and information processing methods - Google Patents

Abstract

From the image including the driver's head, the posture holding function evaluation unit (102) that evaluates the state of the posture holding function, which is the function for the driver to hold the posture, and the evaluation are used to determine the driver's alertness. It is provided with an alertness estimation unit (106) for estimating the level.

Description

The present invention relates to an information processing device, a program, and an information processing method.

A driver monitoring system has been put into practical use. Driver monitoring systems are becoming more widespread not only in commercial vehicles such as trucks and buses, but also in general vehicles.

The main functions installed in the driver monitoring system are the detection functions for drowsy driving and sideways driving as safe driving support functions. Further, in the automatic driving Lv (Level) 3, since the authority is transferred from the automatic driving system to a human being, these functions are required as a technique for determining whether or not the driver is in a driving state.

In the conventional technology, the driver's face is photographed by a camera mounted in the vehicle, and the driver's condition is estimated from the facial expression and the like. For example, as a technique for estimating drowsiness, as described in Patent Document 1, a technique for detecting the degree of eye opening is often used.

Japanese Unexamined Patent Publication No. 61-175129

It is difficult for the conventional technique for detecting drowsiness to constantly detect the drowsiness of the driver while driving. For example, in the conventional technology, an infrared camera is often used, but when the driver wears sunglasses that block infrared rays, the state of the eyes cannot be seen, so the degree of eyelid opening can be detected. Can not.

Therefore, it is an object of the present invention to be able to detect the level of alertness of the driver even when the driver's eyes are hidden.

The information processing apparatus according to one aspect of the present invention includes a posture holding function evaluation unit that evaluates the state of the posture holding function, which is a function of the driver holding the posture, from an image including the driver's head, and the posture holding function evaluation unit. It is characterized by including an arousal level estimation unit that estimates the level of arousal level of the driver using evaluation.

The program according to one aspect of the present invention includes a posture holding function evaluation unit that evaluates the state of the posture holding function, which is a function of the driver holding the posture, from an image including the driver's head, and a computer. It is characterized in that it functions as an arousal level estimation unit that estimates the level of arousal level of the driver by using the evaluation.

In the information processing method according to one aspect of the present invention, the posture holding function evaluation unit evaluates the state of the posture holding function, which is a function of the driver holding the posture, from an image including the driver's head. The arousal level estimation unit estimates the level of arousal level of the driver by using the evaluation.

According to one or more aspects of the invention, the level of alertness of the driver can be detected even when the driver's eyes are hidden.

It is a block diagram which shows schematic structure of the information processing apparatus which concerns on Embodiment 1. FIG. It is the schematic for demonstrating the coordinate system in Embodiment 1. FIG. It is the schematic which shows the arrangement example in the vehicle of the information processing apparatus which concerns on Embodiment 1. FIG. It is a block diagram which shows the hardware configuration example of the information processing apparatus which concerns on Embodiment 1. FIG. It is a flowchart which shows the operation of the information processing apparatus which concerns on Embodiment 1. FIG. It is a block diagram which shows schematic structure of the information processing apparatus which concerns on Embodiment 2. FIG. It is the schematic which shows the arrangement example in the vehicle of the information processing apparatus which concerns on Embodiment 2. FIG. It is a flowchart which shows the operation of the information processing apparatus which concerns on Embodiment 2.

Embodiment 1.
FIG. 1 is a block diagram schematically showing the configuration of the information processing device 100, which is the alertness estimation device according to the first embodiment.
As shown in FIG. 1, the information processing device 100 is connected to the image pickup device 130 and the alertness decrease notification device 140.

The information processing device 100 estimates the level of the driver's arousal level in response to the input of image data indicating the image captured by the image pickup device 130, and the driver's arousal level is lowered by the estimated level. If it is determined, the alertness decrease notification device 140 is notified.
The image pickup device 130 may use, for example, a CCD (Charge Coupled Device) or the like, and the alertness decrease notification device 140 may use a display, a speaker, or the like.

The information processing device 100 includes an input interface unit (hereinafter referred to as an input I / F unit) 101, a posture holding function evaluation unit 102, an alertness estimation unit 106, and an output interface unit (hereinafter referred to as an output I / F unit). It is provided with 107.
The posture holding function evaluation unit 102 includes a head coordinate detection unit 103, a head inclination angle detection unit 104, and an evaluation unit 105.

The input I / F unit 101 is an input unit that receives input of image data from the image pickup apparatus 130. The input I / F unit 101 gives the input image data to the head coordinate detection unit 103 and the head inclination angle detection unit 104.

Here, the image shown by the image data may show one driver or another occupant. Further, the image may be an RGB (Red Green Blue) image, an IR (infrared) image, or a distance image as long as the head coordinates can be detected and the head tilt angle can be detected.

The posture holding function evaluation unit 102 evaluates the state of the posture holding function, which is a function for the driver to hold the posture, from the image shown by the image data.

The head coordinate detection unit 103 detects the head coordinates, which are the coordinates of the driver's head, from the image indicated by the image data given from the input I / F unit 101. The head coordinate detection unit 103 gives the detected head coordinates to the evaluation unit 105.

For example, the head coordinate detection unit 103 detects the driver's face from the image indicated by the image data given from the input I / F unit 101.
The face detection method, which is a method of detecting the driver's face, is not particularly limited.
Specifically, the head coordinate detection unit 103 may detect the driver's face using an AdaBoost-based detector that uses a commonly used Haar-like feature.

The "Haar-like feature amount" is a feature amount that captures a feature based on the difference in brightness of an image, and is less susceptible to fluctuations in lighting conditions or noise as compared with the case where a pixel value is used as a feature amount as it is. "Adaboost" is an abbreviation for Adaptive Boosting, and is a machine learning algorithm that improves performance by using a plurality of weak classifiers in combination.

Next, the head coordinate detection unit 103 detects the head coordinates, which are the coordinates of the driver's head, from the face detection frame, which is an area including the detected face.
The method of calculating the head coordinates may be the center of the face detection frame, or a representative point of the head may be set from the feature points of the face, and the representative point may be used as the head coordinates. For example, the midpoint between the left and right eyes may be used as the representative point. However, since there is no characteristic edge between the eyes or an edge is created by wearing glasses, the midpoint of the left and right eyes is not directly detected, but the midpoint of the left and right inner corners of the eye is used. The representative point may be estimated by setting it as the midpoint of the outer corner of the eye.

The "face detection frame" is a frame indicating an area where the face exists in the image, and includes the feature points of the face. In many cases, the shape of the frame is square. The "facial feature point" is a coordinate that constitutes the facial appearance, and refers to the coordinates of the contour, nostril, outer corner of the eye, inner corner of the eye, corner of the mouth, lips, eyebrows, and the like.

The head coordinate detection unit 103 provides the head tilt angle detection unit 104 with face area information indicating the detected face detection frame.

The head inclination angle detection unit 104 detects the head inclination angle, which is the inclination angle of the driver's head, from the image indicated by the image data given from the input I / F unit 101. The head tilt angle detection unit 104 gives the detected head tilt angle to the evaluation unit 105.

For example, the head tilt angle detection unit 104 is an image in the face detection frame indicated by the face area information given by the head coordinate detection unit 103 among the images indicated by the image data given by the input I / F unit 101. The tilt angle of the driver's head is detected from the face area image.

Specifically, as a method of detecting the inclination angle of the head, the head inclination angle detection unit 104 creates a detector from images taken every 10 degrees of the inclination angle of the head, and creates a detector as a face area image. , The detector may be used to detect the head inclination angle.
Further, the head tilt angle detection unit 104 may specify a plurality of feature points from the face region image and detect the head tilt angle from the distance between the feature points. For example, the head tilt angle detection unit 104 may specify two feature points in the vertical direction of the face region image and detect the head tilt angle by the distance between the two feature points. For example, as the head tilt angle increases, the distance between the two feature points decreases. In this case, if the image data captured by a stereo camera or a TOF (Time of Flight) camera or the like for which the distance can be obtained is used, the distance can be obtained accurately and easily.

The evaluation unit 105 determines the variation of the head coordinates detected by the head coordinate detection unit 103 in a predetermined period and the head inclination angle detected by the head inclination angle detection unit 104 in the predetermined period. The state of the driver's posture holding function is evaluated using the variation.
Here, the evaluation unit 105 disperses the head coordinates detected by the head coordinate detection unit 103 in a predetermined period, and the head detected by the head inclination angle detection unit 104 in a predetermined period. The state of the driver's posture holding function is evaluated by using the dispersion of the inclination angle.

For example, the evaluation unit 105 calculates the variance of the head coordinates in a predetermined period. Specifically, the evaluation unit 105 calculates the variance of the head coordinates for the last 20 seconds. In this case, assuming that the image pickup apparatus 130 inputs image data at 30 fps (frame per second), the evaluation unit 105 records 600 frames of the head coordinates in the gravity direction of the earth and calculates the dispersion of the coordinates. .. As shown in FIG. 2, the direction of gravity of the earth is taken as the Y axis here. The frame rate of the image pickup apparatus 130 and the time for calculating the variance are not limited to this example.
Further, in the example shown in FIG. 2, the direction around the X-axis is the Pitch direction, the direction around the Y-axis is the Yaw direction, and the direction around the Z-axis is the Roll direction.

Here, the evaluation unit 105 calculated the variance of the coordinates in the direction of gravity of the earth, but similarly, the variance of the coordinates in the horizontal direction with respect to the ground may be calculated. As shown in FIG. 2, the horizontal direction with respect to the ground is defined as the X-axis here. When estimating the state of the posture holding function described later from the dispersion of the coordinates in the horizontal direction, the corresponding head inclination angle is the Yaw direction or the Roll direction.

In addition, the evaluation unit 105 calculates the variance of the head inclination angle in a predetermined period. Specifically, the evaluation unit 105 calculates the variance of the head inclination angle for the last 20 seconds. In this case, assuming that the image pickup apparatus 130 inputs image data at 30 fps, the evaluation unit 105 records the angle in the Pitch direction, which is the inclination angle of the earth in the gravity direction for 600 frames, and calculates the dispersion.

Here, the evaluation unit 105 records the angle in the Pitch direction and calculates the variance, but as described above, the evaluation unit 105 may record the angle in the Yaw direction or the Roll direction and calculate the variance.

Then, the evaluation unit 105 calculates an evaluation value indicating the state of the posture holding function from the dispersion of the head coordinates and the dispersion of the head inclination angle. For example, the evaluation unit 105 divides the variance of the head coordinates in the predetermined period by the variance of the head inclination angle in the predetermined period, or the value of the head inclination angle in the predetermined period. The value obtained by dividing the variance by the variance of the head coordinates in a predetermined period is calculated as the evaluation value.

Specifically, the evaluation unit 105 calculates the evaluation value by the following equation (1).
Evaluation value = (dispersion of Pitch angle) ÷ (variance of Y coordinate) (1)
The evaluation unit 105 may calculate the evaluation value by the following equation (2) or (3).
Evaluation value = (variance of X coordinates) ÷ (variance of Yaw angle) (2)
Evaluation value = (variance of Roll angle) ÷ (variance of X coordinates) (3)
The evaluation unit 105 gives the evaluation value calculated as described above to the alertness estimation unit 106.

For example, in the equation (1), when the evaluation value becomes large, the position of the head does not change and the head swings in the vertical direction, so that the driver's alertness decreases and the posture holding function is performed. It can be judged that it is not exhibited.
Even in equation (2), when the evaluation value becomes large, it is judged that the driver's alertness is lowered and the posture holding function is not exhibited because the head is swinging from side to side without rotating. Can be done.
Even in equation (3), when the evaluation value becomes large, the position of the head does not change and the head swings left and right, so the driver's alertness decreases and the posture holding function is exhibited. It can be judged that there is no state.

The arousal level estimation unit 106 estimates the driver's arousal level using the evaluation performed by the posture holding function evaluation unit 102.
For example, the alertness estimation unit 106 estimates the driver's alertness level based on the evaluation value given by the evaluation unit 105.
The level of arousal level may be divided into 6 stages that can be estimated from facial expressions or body movements, as in the known evaluation method defined by NEDO (New Energy and Industrial Technology Development), for example. It may be divided into degree levels.

When the arousal level is divided into levels by the NEDO evaluation method, the arousal level estimation unit 106 sets a threshold value of the evaluation value for each level and estimates the level of the arousal level. In other words, it is assumed that the corresponding level is specified in advance in the evaluation value. Then, when the estimated level of arousal level is lower than a predetermined threshold value, the arousal level estimation unit 106 gives a notification instruction from the output I / F unit 107 to the arousal level decrease notification device 140.

The output I / F unit 107 is an output unit for outputting an instruction or the like to the alertness decrease notification device 140.
The alertness decrease notification device 140 notifies the driver and passengers of the decrease in alertness in response to the notification instruction from the alertness estimation unit 106. For example, the alertness decrease notification device 140 notifies the driver and the like of the decrease in alertness by using the speaker and the display, and indicates that a break is urged or the driving is in a non-transferable state.

FIG. 3 is a schematic view showing an example of arrangement of the information processing device 100 according to the first embodiment in the vehicle 150.
Inside the vehicle 150, an in-vehicle camera 151 that functions as an image pickup device 130, an information processing device 100, a storage device 152, a speaker 153 that functions as an alertness decrease notification device 140, and a display 154 are arranged.

Here, the in-vehicle camera 151 captures an image of the upper body including the driver's head to be detected. The in-vehicle camera 151 is connected to the information processing device 100. The in-vehicle camera 151 shown in FIG. 3 may be connected via wiring such as a wire harness, or may be connected wirelessly.

Further, the in-vehicle camera 151 may be a general RGB camera or an IR camera. Further, the vehicle-mounted camera 151 may be a Depth sensor.
An "RGB camera" is a camera that communicates signals of three colors of red, green, and blue using three different cables and the like, and generally uses three independent CCD sensors.
An "IR camera" is an infrared camera, which is sensitive to wavelengths in the infrared region.
The "Dept sensor" is also referred to as a 3D (three dimensions) sensor, a distance measuring sensor, or a Dept camera, and is a sensor capable of generating a depth image in addition to a two-dimensional image. The distance measuring method includes a stereo camera method, a pattern irradiation method, a TOF method, and the like, but any type of sensor may be used.

FIG. 4 is a block diagram showing a hardware configuration example of the information processing apparatus 100 according to the first embodiment.
The information processing device 100 includes a CPU 160, a program memory 161 and a data memory 162, an interface 163, and a bus 164 connecting them.
An in-vehicle camera 151, a speaker 153, and a display 154 are connected to the interface 163.

Here, the head coordinate detection unit 103, the head inclination angle detection unit 104, the evaluation unit 105, and the arousal degree estimation unit 106 can be configured by the CPU 160 executing a program stored in the program memory 161. ..
The input I / F unit 101 and the output I / F unit 107 can be configured by the interface 163.
As described above, the information processing apparatus 100 can be realized by a computer that stores a specific program.

In addition, "CPU" is an abbreviation for Central Processing Unit, and is a device that performs calculation and control.
The CPU 160 operates according to the program stored in the program memory 161. The CPU 160 stores various data in the data memory 162 in the process of operation. As a result of recognizing the image input from the vehicle-mounted camera 151 via the interface 163, the speaker 153 and the display 154 are controlled via the interface 163.
Hereinafter, the operation performed by the information processing apparatus 100 will be described with reference to FIG.

FIG. 5 is a flowchart showing the operation of the information processing apparatus 100 according to the first embodiment.
The flowchart shown in FIG. 5 includes an image data acquisition step (S10), a face detection step (S11), a head coordinate detection step (S12), a head coordinate distribution calculation step (S13), and a head. It has an inclination angle detection step (S14), a head inclination angle dispersion calculation step (S15), an evaluation value calculation step (S16), an arousal degree estimation step (S17), and an arousal degree decrease notification step (S18). ..

In the image data acquisition step S10, the input I / F unit 101 acquires image data of an image of the upper body including the driver's head.

In the face detection step S11, the head coordinate detection unit 103 detects the driver's face from the image indicated by the acquired image data. The head coordinate detection unit 103 provides the head inclination angle detection unit 104 with face area information indicating a face detection frame, which is a region including the detected face.

In the head coordinate detection step S12, the head coordinate detection unit 103 detects the head coordinates from the face detection frame, which is an area including the face detected in the face detection step S11.
In the head coordinate variance calculation step S13, the evaluation unit 105 calculates the variance of the head coordinates.

In the head tilt angle detection step S14, the head tilt angle detection unit 104 detects the head tilt angle from the face detection frame which is a region including the face detected in the face detection step S11.
In the head tilt angle dispersion calculation step S15, the evaluation unit 105 calculates the variance of the head tilt angle detected in the head tilt angle detection step S14.

In the evaluation value calculation step S16, the evaluation unit 105 calculates an evaluation value indicating the state of the posture holding function from the dispersion of the head coordinates and the dispersion of the head inclination angle.
In the arousal level estimation step S17, the arousal level estimation unit 106 estimates the level of the arousal level from the evaluation value calculated in the evaluation value calculation step S16.

In the arousal level reduction notification step S18, the arousal level estimation unit 106 uses the output I / F unit 107 via the output I / F unit 107 when the level of the arousal level estimated in the arousal level estimation step S17 falls below a predetermined threshold value. The arousal level decrease notification device 140 is instructed to notify the driver and the like of the decrease in arousal level. Whether the arousal level reduction notification device 140 that receives such an instruction notifies the driver, the passenger, or the like of the decrease in arousal level by using at least one of the speaker 153 and the display 154 and prompts a break. Indicates that the operation cannot be transferred.

As described above, according to the first embodiment, the level of the driver's arousal level can be estimated from the state of the posture holding function. In addition, the driver, passengers, and the like can be notified according to the estimation result.

In the first embodiment, since the level of arousal level is estimated by evaluating the state of the posture holding function of the head from the coordinates of the head and the inclination angle of the head, awakening even when the eyes are hidden. The level of degree can be estimated.
In addition, since a sensor other than a camera is not required to estimate the arousal level, cost reduction and miniaturization are possible.
Further, since the inclination angle of the head is used for estimation, it is possible to distinguish whether the movement of the head is due to the turning of the face or the vibration of the vehicle. As a result, it is possible to prevent erroneous detection due to the movement of the head caused by turning around at the time of safety confirmation.

Embodiment 2.
FIG. 6 is a block diagram schematically showing the configuration of the information processing device 200, which is the alertness estimation device according to the second embodiment.
As shown in FIG. 6, the information processing device 200 is connected to the image pickup device 130 and the alertness decrease notification device 140. The image pickup device 130 and the alertness decrease notification device 140 in the second embodiment are the same as the image pickup device 130 and the alertness decrease notification device 140 in the first embodiment.

The information processing device 200 according to the second embodiment is also connected to the CAN 270. "CAN" is an abbreviation for Controller Area Network, and is a network corresponding to a standard for connecting electronic circuits and various devices developed as a communication technology between in-vehicle devices.

The information processing device 200 includes an input I / F unit 201, a posture holding function evaluation unit 102, an alertness estimation unit 206, an output I / F unit 107, a PERCLOS calculation unit 208, and a yawn detection unit 209. ..
The posture holding function evaluation unit 102 includes a head coordinate detection unit 103, a head inclination angle detection unit 104, and an evaluation unit 105.

The posture holding function evaluation unit 102 and the output I / F unit 107 in the second embodiment are the same as the posture holding function evaluation unit 102 and the output I / F unit 107 in the first embodiment.
However, the head coordinate detection unit 103 of the posture holding function evaluation unit 102 transmits the face area information indicating the detected face detection frame to the head inclination angle detection unit 104, as well as the PERCLOS calculation unit 208 and the yawn detection unit 209. give.

The input I / F unit 201 receives the input of image data from the image pickup apparatus 130. The input I / F unit 201 gives the input image data to the head coordinate detection unit 103, the head inclination angle detection unit 104, the PERCLOS calculation unit 208, and the yawn detection unit 209.
Further, the input I / F unit 201 receives input from CAN270 of vehicle information indicating the state of the vehicle driven by the driver, such as the steering angle of the steering wheel, the position of the gear, or the vehicle speed. The input I / F unit 201 gives the input vehicle information to the alertness estimation unit 206.

The PERCLOS calculation unit 208 detects the degree of eye opening, which is the rate at which the driver's eyes are open, from the image indicated by the image data given from the input I / F unit 201, and according to the degree of eye opening, the driver at a fixed time. Is an eye closure ratio calculation unit that calculates the eye closure ratio, which is the ratio of the time that the eye is closed.
Here, the PERCLOS calculation unit 208 calculates the driver's PERCLOS. PERCLOS is an abbreviation for percentage of eye closure, which is the ratio of eye closure time within a certain period of time. PERCLOS is known to have a high correlation with the subjective drowsiness of the subject (here, the driver), and can be used as a drowsiness index.

For example, the PERCLOS calculation unit 208 has a face area image corresponding to the face detection frame indicated by the face area information given by the head coordinate detection unit 103 among the images indicated by the image data given by the input I / F unit 201. In, the degree of eye opening is calculated from the distance between the upper and lower eyelids. Specifically, the PERCLOS calculation unit 208 finds the midpoint between the outer and inner corners of the eyes on the arc of the upper and lower eyelids, and determines the ratio of the distance between the upper and lower eyelids to the distance between the upper and lower eyelids at the time of maximum eye opening. The degree of eye opening.
The distance between the upper and lower eyelids with respect to the distance between the upper and lower eyelids at the time of maximum eye opening shall be predetermined. Therefore, the definition of the degree of eye opening is calculated for each frame, with the degree of eye opening when the eyes are closed as 0% and the degree of eye opening when awake as 100%.

Then, the PERCLOS calculation unit 208 calculates PERCLOS from the calculated degree of eye opening. The PERCLOS calculation unit 208 determines whether the eyes are open or closed for each frame with respect to the calculated degree of eye opening, and when it is determined that the eyes are closed, the calculation is performed as an eye closing frame. For example, when the PERCLOS calculation unit 208 is provided with a buffer for 1 minute and calculates PERCLOS from an image taken by an image pickup device 130 at 10 fps, the number of eye-closure frames accumulated in 1 minute is divided by 6000 frames. , PERCLOS can be calculated.

The yawn detection unit 209 detects the opening degree, which is the ratio at which the driver's mouth is open, from the image indicated by the image data given from the input I / F unit 201, and the driver yawns according to the opening degree. Detects whether or not it is done.

For example, the lips detection unit 209 is a face area image corresponding to the face detection frame indicated by the face area information given by the head coordinate detection unit 103 among the images indicated by the image data given by the input I / F unit 201. In, the degree of opening of the mouth is detected from the feature points of the face from the distance between the upper and lower lips. Specifically, the accretion detection unit 209 obtains the midpoint between the left and right mouth corners on the arc of the upper lip and the lower lip, respectively, and sets the distance between the upper and lower midpoints as the opening degree with respect to the distance between the upper and lower midpoints at the time of maximum opening. calculate.

Then, the yawn detection unit 209 detects the driver's yawn based on the calculated opening degree. The characteristics of yawning are that the mouth opens continuously for a certain period of time, the nostrils widen, tears come out, or inhalation is performed before yawning occurs, but here the mouth opens continuously for a certain period of time. The case where the feature is used will be described.

Specifically, the yawn detection unit 209 determines that the corresponding frame is a yawn frame when the calculated opening degree becomes equal to or higher than a predetermined threshold value. Then, the yawn detection unit 209 determines that the yawn is yawn when the yawn frame continues for a predetermined time, which is a threshold value or longer. Further, since yawning does not occur continuously in a short time, the yawning detection unit 209 does not detect yawning for a certain period of time after detecting yawning once. This makes it possible to prevent erroneous detection.

The alertness estimation unit 206 has vehicle information given by the input I / F unit 201, an evaluation value given by the evaluation unit 105, PERCLOS given by the PERCLOS calculation unit 208, and the presence or absence of yawning given by the yawn detection unit 209. From, the level of alertness of the driver is estimated.

For example, the arousal level estimation unit 206 refers to the vehicle state indicated by the vehicle state information, and estimates the driver's arousal level when the vehicle state does not correspond to a predetermined condition.
Here, the predetermined conditions are when back or parking is selected as the gear, when the vehicle speed is below the predetermined threshold value, or when the steering angle is above the predetermined threshold value. Is. When none of these conditions are met, the arousal level estimation unit 206 estimates the level of arousal level.

By considering the above predetermined conditions, in the second embodiment, the alertness estimation unit 206 advances straight forward toward the front when the vehicle state indicated by the vehicle state information is equal to or higher than the predetermined vehicle speed. If so, estimate the level of alertness of the driver.

Then, when the arousal level estimation unit 206 determines that the driver's arousal level is to be estimated, the vehicle information given by the input I / F unit 201, the evaluation value given by the evaluation unit 105, and the PERCLOS calculation unit. The level of alertness of the driver is estimated from the PERCLOS given by 208 and the presence or absence of yawn given by the yawn detection unit 209.

The arousal level estimation unit 206 includes vehicle information given by the input I / F unit 201, an evaluation value given by the evaluation unit 105, PERCLOS given by the PERCLOS calculation unit 208, and yawns given by the yawn detection unit 209. The level of alertness corresponding to each is estimated from each of the presence or absence. Then, the arousal level estimation unit 206 adopts the lowest arousal level at the same timing as the arousal level at that timing.

Here, as vehicle information, a change over time in the steering angle of the steering wheel is used. It is known that the steering angle of the steering changes finely as the level of alertness decreases, and this index is used. That is, the level of arousal level may be associated in advance according to the magnitude of the change in steering angle with time.
Further, for PERCLOS, the level of arousal level may be associated with each value in advance, and the level of arousal level may be associated with each value in advance with respect to the presence or absence of yawning.

Here, the level of arousal level corresponding to the evaluation value is the first level, the level of the arousal level corresponding to the state of the vehicle indicated by the vehicle information is the second level, and the level of the arousal level corresponding to PERCLOS is the third level. And the level of alertness corresponding to whether or not yawning is the fourth level, the alertness estimation unit 206 has a first level, a second level, a third level, and a third level. Identify the final level of alertness from level 4. However, if the third level cannot be obtained due to reasons such as wearing sunglasses, the final level of alertness is specified from the first level, the second level, and the third level. .. Here, among these levels, the one with the lowest arousal level is specified as the final arousal level.

As the level of alertness, the NEDO evaluation method may be used as in the first embodiment, but any level of alertness may be used. It is desirable to calculate the level of alertness together with the reliability for each index. The reliability is an index indicating whether the output value is a plausible value. The value output when the reliability is high is a plausible value, and the value output when the reliability is low is adjusted so that it is often incorrect.

As a method for estimating the arousal level, the lowest level among the indicators whose reliability exceeds a predetermined threshold value may be adopted. As a result, even a driver who does not have a characteristic of opening and closing eyes can correctly determine the level of arousal level by yawning or variation in steering angle.

That is, it is known that a method of estimating the level of the driver's arousal level from the degree of eyelid opening is effective, and that the PERCLOS increases as the level of the driver's arousal level decreases. A method of detecting an open / closed eye using an image pickup device 130 that is sensitive to infrared rays is common, but the state of the eyes cannot be detected when wearing sunglasses having low infrared transmittance. However, according to the above method, the level of arousal level by PERCLOS is not estimated when wearing sunglasses, but the level estimated by another method can be applied.

FIG. 7 is a schematic view showing an example of arrangement of the information processing device 200 according to the second embodiment in the vehicle 150.
Inside the vehicle 150, there are an in-vehicle camera 151 that functions as an image pickup device 130, an information processing device 200, a storage device 152, a speaker 153 and a display 154 that function as an alertness decrease notification device 140, and a vehicle speed sensor 255. The gearbox 256 and the steering 257 are arranged.

The vehicle-mounted camera 151, the storage device 152, the speaker 153, and the display 154 in the second embodiment are the same as the vehicle-mounted camera 151, the storage device 152, the speaker 153, and the display 154 in the first embodiment.
In the second embodiment, the information that can be acquired from the vehicle speed sensor 255, the gearbox 256, and the steering 257 is sent to the information processing apparatus 200 as vehicle information via the CAN 270.

As shown in FIG. 4, the information processing apparatus 200 according to the second embodiment also includes a CPU 160, a program memory 161 and a data memory 162, an interface 163, and a bus 164 connecting them.
An in-vehicle camera 151, a speaker 153, and a display 154 are connected to the interface 163.
In the second embodiment, the interface 163 is also connected to the CAN 270.

FIG. 8 is a flowchart showing the operation of the information processing apparatus 200 according to the second embodiment.
The flowchart shown in FIG. 8 includes an image data acquisition step (S20), a face detection step (S21), a head coordinate detection step (S22), a head coordinate distribution calculation step (S23), and a head. Tilt angle detection step (S24), head tilt angle dispersion calculation step (S25), eye opening degree detection step (S26), PERCLOS calculation step (S27), opening degree detection step (S28), and accretion detection step. (S29), an evaluation value calculation step (S30), an arousal degree estimation step (S31), and an arousal degree decrease notification step (S32).

In the image data acquisition step S20, the input I / F unit 201 acquires image data of an image of the upper body including the driver's head. The image shown by the image data is an RGB image, an IR image, or an IR image, if it is possible to calculate the head coordinates, the head inclination angle, the PERCLOS, and the yawn detection to be performed in the subsequent processing. It may be any image such as a distance image.

In the face detection step S21, the head coordinate detection unit 103 detects the driver's face from the image indicated by the acquired image data. The head coordinate detection unit 103 gives face area information indicating a detection frame, which is a region including the detected face, to the head inclination angle detection unit 104, the PERCLOS calculation unit 208, and the yawn detection unit 209.

In the head coordinate detection step S22, the head coordinate detection unit 103 detects the head coordinates from the face detection frame, which is an area including the face detected in the face detection step S21.
In the head coordinate variance calculation step S23, the evaluation unit 105 calculates the variance of the head coordinates.

In the head tilt angle detection step S24, the head tilt angle detection unit 104 detects the head tilt angle from the face detection frame, which is a region including the face detected in the face detection step S21.
In the head tilt angle dispersion calculation step S25, the evaluation unit 105 calculates the variance of the head tilt angle detected in the head tilt angle detection step S24.

In the eye opening degree detection step S26, the PERCLOS calculation unit 208 uses the feature points of the driver's face from the face detection frame, which is an area including the face detected in the face detection step S21, to determine the degree of eye opening. Is detected.
In the PERCLOS calculation step S27, the PERCLOS calculation unit 208 calculates PERCLOS from the eye opening degree detected in the eye opening degree detection step S26.

In the opening degree detection step S28, the yawn detection unit 209 uses the feature points of the driver's face from the face detection frame, which is an area including the face detected in the face detection step S21, to open the mouth. Detect the degree.
In the yawn detection step S29, the yawn detection unit 209 detects the presence or absence of yawn by using the opening degree detected in the opening degree detection step S28.

In the evaluation value calculation step S30, the evaluation unit 105 calculates an evaluation value indicating the state of the posture holding function from the dispersion of the head coordinates and the dispersion of the head inclination angle.

In the arousal level estimation step S31, the awakening level estimation unit 206 refers to the vehicle state indicated by the vehicle state information, and estimates the driver's arousal level when the vehicle state does not correspond to a predetermined condition. .. For example, the alertness estimation unit 206 has vehicle information given by the input I / F unit 201, an evaluation value given by the evaluation unit 105, PERCLOS given by the PERCLOS calculation unit 208, and yawn given by the yawn detection unit 209. The level of alertness of the driver is estimated from the presence or absence of.

In the arousal level reduction notification step S32, the arousal level estimation unit 206 uses the output I / F unit 107 via the output I / F unit 107 when the level of the arousal level estimated in the arousal level estimation step S31 falls below a predetermined threshold value. The arousal level decrease notification device 140 is instructed to notify the driver and the like of the decrease in arousal level. Whether the arousal level reduction notification device 140 that receives such an instruction notifies the driver, the passenger, or the like of the decrease in arousal level by using at least one of the speaker 153 and the display 154 and prompts a break. Indicates that the operation cannot be transferred.

As described above, according to the second embodiment, the evaluation can be performed by an index suitable for evaluating the level of the driver's arousal level.
Further, by controlling the speaker 153 and the display 154 according to the evaluation result, the driver and the passenger can be notified of the arousal level.

100,200 Information processing device, 101,201 input I / F unit, 103 head coordinate detection unit, 104 head tilt angle detection unit, 105 evaluation unit, 106,206 arousal degree estimation unit, 107 output I / F unit, 208 PERCLOS calculation unit, 209 accretion detection unit, 130 imaging device, 140 arousal reduction notification device, 150 vehicle, 151 in-vehicle camera, 152 storage device, 153 speaker, 154 display, 160 CPU, 161 program memory, 162 data memory, 163 Interface, 164 buses.

The information processing apparatus according to one aspect of the present invention includes a head coordinate detection unit that detects head coordinates, which are the coordinates of the driver's head, from an image including the driver's head, and the image. From the head tilt angle detection unit that detects the head tilt angle, which is the tilt angle of the driver's head, and the variation of the head coordinates and the head tilt angle in a predetermined period, the driver takes a posture. It is characterized by including an evaluation unit that evaluates the state of the posture holding function, which is a function of holding the above, and an arousal degree estimation unit that estimates the level of the arousal degree of the driver by using the evaluation.

In the program according to one aspect of the present invention, the computer is subjected to a head coordinate detection unit that detects head coordinates, which are the coordinates of the driver's head, from an image including the driver's head. The head tilt angle detection unit that detects the head tilt angle, which is the tilt angle of the driver's head, the head coordinates in a predetermined period, and the variation of the head tilt angle, the driver determines the posture. It is characterized in that it functions as an evaluation unit that evaluates the state of the posture holding function, which is a function to be held, and an arousal degree estimation unit that estimates the level of the arousal degree of the driver by using the evaluation.

In the information processing method according to one aspect of the present invention, the head coordinate detection unit detects the head coordinates, which are the coordinates of the driver's head, from the image including the driver's head, and the head inclination angle. The detection unit detects the head tilt angle, which is the tilt angle of the driver's head, from the image, and the evaluation unit detects the variation of the head coordinates and the head tilt angle in a predetermined period. , The state of the posture holding function, which is a function of the driver to hold the posture, is evaluated, and the arousal degree estimation unit estimates the level of the driver's arousal degree by using the evaluation. ..

The information processing apparatus according to one aspect of the present invention includes a head coordinate detection unit that detects head coordinates, which are the coordinates of the driver's head, from an image including the driver's head, and the image. The head tilt angle detection unit that detects the head tilt angle, which is the tilt angle of the driver's head, the dispersion of the head coordinates in a predetermined period, and the head tilt in the predetermined period. Arousal estimation that estimates the level of arousal of the driver using the evaluation unit and the evaluation unit that evaluates the state of the attitude holding function, which is the function of the driver to hold the posture, from the dispersion of the angles. It is characterized by having a part and.

In the program according to one aspect of the present invention, the computer is subjected to a head coordinate detection unit that detects head coordinates, which are the coordinates of the driver's head, from an image including the driver's head. The head tilt angle detection unit that detects the head tilt angle, which is the tilt angle of the driver's head, the dispersion of the head coordinates in a predetermined period, and the head tilt angle in the predetermined period. and a dispersion of the evaluation unit for evaluating the driver attitude holding function is a function of holding the posture state, and, using the evaluation, arousal level wakefulness estimation to estimate of the driver It is characterized by functioning as a department.

In the information processing method according to one aspect of the present invention, the head coordinate detection unit detects the head coordinates, which are the coordinates of the driver's head, from the image including the driver's head, and the head inclination angle. The detection unit detects the head inclination angle, which is the inclination angle of the driver's head, from the image, and the evaluation unit determines the dispersion of the head coordinates in a predetermined period and the predetermined determination. From the dispersion of the head inclination angle during the period, the state of the posture holding function, which is the function of the driver to hold the posture, is evaluated, and the arousal degree estimation unit uses the evaluation to awaken the driver. It is characterized by estimating the level of degree.

Here, the level of arousal level corresponding to the evaluation value is the first level, the level of the arousal level corresponding to the state of the vehicle indicated by the vehicle information is the second level, and the level of the arousal level corresponding to PERCLOS is the third level. And the level of alertness corresponding to whether or not yawning is the fourth level, the alertness estimation unit 206 has a first level, a second level, a third level, and a third level. Identify the final level of alertness from level 4. However, if the third level cannot be obtained due to reasons such as wearing sunglasses, the final level of alertness is specified from the first level, the second level, and the fourth level. .. Here, among these levels, the one with the lowest arousal level is specified as the final arousal level.

Claims (9)

A posture holding function evaluation unit that evaluates the state of the posture holding function, which is a function for the driver to hold the posture, from an image including the driver's head.
An information processing device including an arousal level estimation unit that estimates the level of arousal level of the driver using the evaluation. The posture holding function evaluation unit
From the image, a head coordinate detection unit that detects the head coordinates, which are the coordinates of the driver's head,
From the image, a head tilt angle detection unit that detects the head tilt angle, which is the tilt angle of the driver's head,
The information processing apparatus according to claim 1, further comprising an evaluation unit that performs the evaluation based on variations in the head coordinates and the head inclination angle in a predetermined period. The evaluation unit divides the variance of the head coordinates in the predetermined period by the variance of the head inclination angle in the predetermined period, or the head in the predetermined period. The information processing apparatus according to claim 2, wherein the value obtained by dividing the variance of the inclination angle by the variance of the head coordinates in the predetermined period is calculated as an evaluation value. The information processing apparatus according to claim 3, wherein the level corresponding to the evaluation value is specified in advance. An input unit that receives input of vehicle state information indicating the vehicle state, which is the state of the vehicle driven by the driver.
From the image, the eye opening rate, which is the rate at which the driver's eyes are open, is detected, and the eye closing rate, which is the rate at which the driver closes the eyes at a certain time, is calculated according to the eye opening degree. Percentage calculation unit and
From the image, a yawn detection unit that detects the opening degree, which is the ratio at which the driver's mouth is open, and detects whether or not the driver is yawning according to the opening degree, is further provided.
The first level of the alertness corresponding to the evaluation, the second level corresponding to the vehicle state, the third level of the alertness corresponding to the eye closure rate, and whether or not the person is yawning. The third level of the arousal level corresponding to is predetermined.
Any of claims 1 to 3, wherein the alertness estimation unit identifies the level from the first level, the second level, the third level, and the fourth level. The information processing device according to the first paragraph. The arousal level estimation unit is characterized in that, among the first level, the second level, the third level, and the fourth level, the one with the lowest arousal level is specified as the level. The information processing apparatus according to claim 5. 5. The information processing apparatus according to 6. Computer,
From the image including the driver's head, the posture holding function evaluation unit that evaluates the state of the posture holding function, which is the function of the driver holding the posture, and the posture holding function evaluation unit.
A program characterized by functioning as an arousal level estimation unit that estimates the level of arousal level of the driver using the evaluation. The posture holding function evaluation unit evaluates the state of the posture holding function, which is a function for the driver to hold the posture, from the image including the driver's head.
An information processing method characterized in that the arousal level estimation unit estimates the level of arousal level of the driver using the evaluation.

Images