JP5782726B2 - Arousal reduction detection device - Google Patents

Description

  The present invention relates to a wakefulness detection device.

  Steering angle detection means for detecting the steering angle, steering angle estimation means for estimating the current steering angle when it is assumed that steering is smoothly performed based on the detected steering angle value, the estimated steering angle value, and the steering angle A vehicular driving operation monitoring device comprising steering error detecting means for detecting an error from a detected value and unstable state detecting means for detecting an unstable state of a driving operation based on the steepness of a steering error distribution is known. (Patent Document 1).

JP-A-11-227491

  However, there is a problem that it is difficult to determine whether the error between the estimated steering angle value and the detected steering angle value is due to a decrease in the driver's arousal level or other factors, and the estimation accuracy of the arousal level is low. It was.

  The problem to be solved by the present invention is to provide a wakefulness detection device that can improve the estimation accuracy of the wakefulness drop level.

  The present invention provides a first arousal level calculating means for calculating a first arousal level of a driver, and a second arousal level for calculating a second arousal level based on a secondary operation of the driver in the initial state of a reduced arousal level. The above-mentioned problem is solved by including two arousal level calculation means and awakening reduction level estimation means for estimating the driver's arousal reduction level using the first awakening level and the second awakening level.

  According to the present invention, since the wakefulness reduction level is estimated by combining different indexes of the first wakefulness level and the second wakefulness level, it is possible to improve the estimation accuracy of the wakefulness reduction level.

It is a block diagram which shows schematic structure of the arousal fall detection apparatus to which one embodiment of this invention is applied. It is a block diagram which shows the control apparatus, vehicle information detection part, and imaging device of FIG. FIG. 2 shows the characteristics of the steering angle per time, where (a) shows the characteristics of the input signal of the steering angle signal processing section of FIG. 2, and (b) shows the characteristics of the output signal of the steering angle signal processing section of FIG. It is a graph. It is a figure explaining the image after being processed by the image signal processing part 14 of FIG. FIG. 5 is a diagram for explaining the correspondence between the feature points in FIG. 4 and actual faces. FIG. 3 is a control block diagram of a primary arousal level calculation unit in FIG. 2. FIG. 7 is a diagram for explaining the characteristics of an index used in the primary arousal level calculation unit in FIG. 6, and shows the detected steering angle characteristics per time (graph a) and the predicted steering angle characteristics (graph b). It is a figure for demonstrating the steering amplitude and steering interval calculated by the steering amplitude and steering interval calculation function of FIG. It is a frequency distribution table calculated by the distribution calculation function of FIG. FIG. 3 is a control block diagram of a secondary arousal level calculation unit in FIG. 2. It is a figure for demonstrating the image of the eye determined by the opening-and-closing eye determination function of FIG. 10, (a) shows an image of an open eye, (b) shows an image of a closed eye. It is a figure for demonstrating the opening / closing rate calculated by the opening / closing rate calculating function of FIG. It is a control block diagram of the 2nd awakening degree calculating part of FIG. It is a figure for demonstrating the yawn detection function of FIG. 13, Comprising: It is an enlarged view of the image of the mouth of a face. It is a figure for demonstrating the motion detection function of the neck of FIG. 13, Comprising: (a) shows the face image of the state which inclined the neck, (b) is the mouth of (a) at the time (t-1). An enlarged image is shown, and (c) shows an enlarged image of the mouth among face images in which the neck is tilted to the left at time (t). It is a figure for demonstrating the motion detection function of the hand of FIG. It is a figure for demonstrating the awakening degree calculating function of FIG. It is a control block diagram of the arousal reduction level of FIG. It is a figure explaining the map stored in the arousal level-decrease level map of FIG. (A) is a graph which shows the characteristic of the arousal fall level per time in the observation evaluation of dozing, and (b) is a graph which shows the characteristic per time of the calculated arousal level. FIG. 20 is a map for explaining the correspondence between the arousal reduction level and the stimulation degree on the map of FIG. 19. It is a map stored in the stimulus control part of FIG.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

<< First Embodiment >>
FIG. 1 is a block diagram showing a schematic configuration of a wakefulness detection apparatus to which an embodiment of the present invention is applied. The arousal reduction detection device 1 according to this embodiment is provided in a vehicle and includes a control device 10, a vehicle information detection unit 20, an imaging device 31, a monitor 40, a speaker 41, and an air conditioner 42. The control device 10 is a control unit for calculating a driver's arousal level based on the vehicle information or the driver's information, and estimating a decrease in arousal according to the calculated arousal level. The vehicle information detection unit 20 is a detection unit for detecting information related to a driving state and a vehicle by a driver. The imaging device 31 is a camera that captures a driver's face image, and is provided at a position suitable for surrounding imaging including the driver's face. The monitor 40 is a display that gives a visual stimulus to the driver by displaying an image based on an output signal from the control device 10. The speaker 41 is a speaker that gives an audible stimulus to the driver by outputting a sound based on an output signal from the control device 10. The air conditioner 42 stimulates the driver by adjusting the temperature in the passenger compartment based on the output signal from the control device 10.

  Next, the control device 10, the vehicle information detection unit 20, and the imaging device 31 of the wakefulness decrease detection device 1 of this example will be described with reference to FIG. FIG. 2 is a block diagram showing in more detail the control device 10, the vehicle information detection unit 20, and the imaging device 31 of FIG.

  The vehicle information detection unit 20 includes a vehicle speed detection unit 21 that detects the traveling speed of the vehicle, and a steering angle detection unit 22 that detects the steering angle of the steering. The vehicle speed detection unit 21 is provided for detecting vehicle information indicating the running state of the vehicle, and the steering angle detection unit 22 is provided for detecting the driving state of the driver. The corner is detected. A signal including information on the steering angle detected by the steering angle detection unit 22, a signal including information on the vehicle speed detected by the vehicle speed detection unit 21, and a signal including a captured image of the imaging device are output to the control device 10. Is done.

  The imaging device 31 is provided to detect the driver's driving state by detecting the state of the driver's face by image analysis described later, and images the face and its surroundings at a predetermined sampling period. . The image data captured by the imaging device 31 is output to the control device 10.

  The control device 10 includes a first arousal level calculation unit 11, a second arousal level calculation unit 12, a steering angle signal processing unit 13, an image signal processing unit 14, a wakefulness reduction level estimation unit 15, and a stimulation control unit 16. With. Hereinafter, each structure of the control apparatus 10 is demonstrated.

  The steering angle signal processing unit 13 receives the detection data of the steering angle detection unit 22 as input, performs a smoothing process on the data as shown in FIG. 3, and outputs the data to the first arousal level calculation unit 11. FIG. 3 shows the characteristics of the angle of the steering angle per time, (a) shows the data signal of the steering angle before being processed by the steering angle signal processing unit 13, and (b) shows the steering angle signal processing unit 13. 2 shows a steering angle data signal after being processed by. The steering angle signal processing unit 13 performs a smoothing process by, for example, providing a window for a predetermined time on the time axis and obtaining a moving average of the output for the predetermined time. Further, the steering angle signal processing unit 13 may perform a smoothing process by removing high-frequency components by passing a low-pass pass filter or a pass filter through the input signal. As shown in FIG. 3A, in the signal before processing, the processing load increases in the awakening degree calculation process described later, but by performing smoothing processing as shown in FIG. 3B. Since the frequency of the angle change is suppressed, the processing load in the arousal level calculation step can be suppressed. Then, the steering angle signal processing unit 13 outputs the processed steering angle data to the first awakening level calculation unit 12 as a processed steering angle signal.

  The image signal processing unit 14 performs image processing in order to detect facial features from the facial image captured by the imaging device 31. FIG. 4 is a diagram showing an image after being processed by the image signal processing 14. As shown in FIG. 4, the image signal processing unit 14 extracts, as extraction points, pixels whose density is locally high from each image row in the vertical direction of the image, and connects the extraction points of adjacent pixel rows. Then, a group of curves extending in the horizontal direction is extracted. Then, the eyes, nose, mouth, and the like that are facial features are extracted from the curve group. In FIG. 5, the curve group G1 corresponds to the left eyebrow, the curve group G2 corresponds to the right eyebrow, G3 corresponds to the left eye, G4 corresponds to the right eye, G5 corresponds to the nose, and G6 corresponds to the mouth. Thereby, the position and shape of each part of the face which is the feature point of the face are specified.

  Further, the image signal processing unit 14 can extract facial features as shown in FIG. 5 by extracting the upper, lower, left and right end points of each of the curve groups G1 to G6 as feature points. For example, the eye opening / closing can be detected by detecting the movement of the feature points G31, 32, 41, 42 arranged in the vertical direction among the feature points corresponding to the eye position in time series. Similarly, with respect to other facial parts, the movement can be detected by detecting the variation of the feature points. Then, the image signal processing unit 14 transmits the processed image signal to the first awakening level calculation unit 11 and the second awakening level calculation unit 12.

  The first arousal level calculation unit 11 includes a primary arousal level calculation unit 111 and a secondary awakening level calculation unit 112. First, the primary arousal level calculation unit 111 will be described with reference to FIGS. FIG. 6 shows a control block diagram of the primary arousal level calculation unit 111, FIG. 7 shows a detected steering angle characteristic (graph a) and a predicted steering angle characteristic (graph b) per time, and FIG. FIG. 9 shows a frequency distribution with respect to the steering angle error.

  The primary arousal level calculation unit 111 has a steering error calculation function (F1), a steering amplitude / steering interval calculation function (F2), a steering distribution calculation function (F3), and a wakefulness level calculation function (F4). Based on the steering angle detected by the steering angle detection unit 32 and the vehicle speed detected by the vehicle speed detection unit 21, the awakening level of the primary stage is calculated. Since the arousal drop is likely to occur in a monotonous driving environment, in this example, the arousal level is calculated based on the vehicle speed and the steering angle.

The primary arousal level calculation unit 111 receives the processed steering angle signal from the steering angle signal processing unit 13, calculates the predicted steering angle by the steering error calculation function (F1), and calculates the current steering angle and the predicted steering angle. To calculate the error. The predicted steering angle is calculated from the transition of the steering angle on the assumption that the steering is performed smoothly from the past data during a predetermined period from the data of the latest steering angle. That is, in a state where the steering is smoothly performed, the transition of the steering angle may differ depending on the skill of the driver, the habit, etc. A proper steering angle is calculated as an expected steering angle. The primary arousal level calculation unit 111 calculates a prediction error by taking a difference between the calculated predicted steering angle and the current steering angle detected by the steering angle detection unit 32. As shown in FIG. 7, when the steering angle detected by the steering angle detection unit 32 is remained as the graph a, the calculated expected steering angle is remained as the graph b, at time t _d, the expected The error is d. The smaller the prediction error, the smoother the driving state.

The primary arousal level calculation unit 111 calculates the steering amplitude and the steering interval by the steering amplitude / steering interval calculation function (F2). The primary arousal level calculation unit 111 obtains the difference between the latest two maximum values and the minimum value from the peak values (maximum value and minimum value) of the steering angle signal output from the steering angle signal processing unit 13. The steering amplitude A is calculated. Further, the primary arousal level calculation unit 111 calculates the steering interval T by taking the difference between the time of the two maximum values and the time of the minimum value. In the example shown in FIG. 8, the processed steering angle signal characteristic has a maximum value A ₁ at time t ₁ and a minimum value A ₂ at time t ₂ , and the steering amplitude A is calculated by A ₁ -A _2. The steering interval T is calculated by T ₁ -T ₂ .

Then, the primary arousal level calculation unit 111 calculates the respective frequency distributions of the predicted steering angle, the steering amplitude, and the steering interval as shown in FIG. 9 by the steering distribution calculation function (F3). FIG. 9 is a graph showing a frequency distribution of prediction errors calculated by the steering distribution calculation function (F3). The frequency distribution is calculated by dividing the prediction error by a predetermined value and taking the number (frequency) of divided prediction error data. In FIG. 9, Q _M is a graph with the maximum frequency. That is, when the prediction error is calculated as described above in a normal driving state (a state where the driver's arousal level is high), the prediction error belongs to the graph A with a high probability. Similarly, a frequency distribution of steering amplitude and steering interval is also created. These frequency distributions are stored in a storage unit such as a memory (not shown) and updated as needed. The primary awakening degree calculation unit 111 is within a certain period from when the vehicle speed becomes equal to or higher than a predetermined threshold speed from the start of driving or when the steering angle becomes equal to or higher than the predetermined steering angle from the start of driving. The frequency distribution is updated using the calculated predicted steering angle, steering amplitude, and steering interval.

Next, the primary arousal level calculation unit 111 calculates the awakening level at the initial stage by the arousal level calculation function (F4) using the frequency distribution calculated by the steering distribution calculation function (F3). First, the primary arousal level calculation unit 11 calculates a prediction error from the difference between the current steering angle and the predicted steering angle, and identifies which graph the prediction error belongs to on the frequency distribution of FIG. For example, when the prediction error calculated at the current value (X ₁ ) is x ₁ , the prediction error x ₁ belongs to the graph Q ₁ on the frequency distribution of FIG. 9, and the current value (X ₂ ). expected error if the calculated expected error is x ₂ is x ₂ belongs to the graph Q ₂ when. When the difference values (Δx ₁ and Δx ₂ ) between the prediction error corresponding to the maximum frequency graph Q _M and the prediction error x ₁ and the prediction error x ₂ are calculated, the difference value (Δx ₁ ) is the difference. Greater than the value (Δx ₂ ). The difference value indicates how much the current prediction error deviates from the maximum frequency of prediction error. The smaller the difference value (Δx), the closer the operation is to smooth operation at normal times. It will be. Then, the primary arousal level calculation unit 111 calculates a low arousal level as the time in which the difference value (Δx) is smaller than a predetermined value serving as a threshold value continues longer. That is, the state in which the difference value (Δx) is smaller than the predetermined threshold value is a state in which the driver is smoothly driving. If the state continues for a long time, the driver neglects the steering operation and becomes doze. There is a possibility. For this reason, in this example, the arousal level is calculated to be low in this state. Since the change in the steering angle per hour is small at low speeds, the primary arousal level calculation unit 111 performs the above calculation when the vehicle speed is higher than a predetermined vehicle speed.

  Further, as described above, the primary arousal level calculation unit 111 calculates the steering amplitude and the steering interval within a predetermined period including the current detection time. Next, the position where the current steering amplitude and steering interval are located on the frequency distribution is grasped, and the difference with respect to the maximum frequency graph is calculated. The primary arousal level calculation unit 111 calculates the initial level of arousal level using the difference value based on the prediction error, the difference value based on the steering amplitude, and the difference value based on the steering interval on each frequency distribution. To do.

  Next, the secondary arousal level calculation unit 112 will be described with reference to FIGS. FIG. 10 is a control block diagram of the secondary arousal level calculation unit 111, and FIG. 11 is an enlarged view of one eye part of FIG. 5, showing an open eye state (a) and a closed eye state (b). 12 is a diagram for explaining the transition of the opening and closing eyes per time.

  The secondary arousal level calculation unit 112 has an open / closed eye determination function (F11), an open / close rate calculation function (F12), and a wakefulness level calculation function (F13), and the post-processing output from the image signal processing unit 14 Based on the image signal, the secondary level arousal level is calculated.

The secondary arousal level calculation unit 112 receives the processed image signal from the image signal processing unit 14, and determines the open / closed eye by the open / closed eye determination function (F11). The opening / closing of the eyes is determined from the movements of the feature points of the eyes and the corners of the eyes. That is, as shown in FIGS. 11A and 11B, when the eye is closed, the distance between the eye feature point and the eye feature point (d _b ) is the distance between the feature points in the vertical direction when the eye is opened (d _a ) Shorter. Therefore, the secondary arousal level calculation unit 112 calculates the interval from the position of the pixel corresponding to the eye feature point, and the calculated interval corresponds to the feature point interval when the eye is closed. When it becomes smaller than the threshold, it is determined that the eyes are closed. The secondary arousal level calculation unit 112 performs open / closed eye determination at a predetermined sampling period.

The secondary arousal level calculation unit 112 determines the open / closed eye by the open / close rate determination function (F12). In the secondary arousal level calculation unit 112, an eye opening rate calculation section (d _c ) is set in advance, and the time during which one eye is closed for a certain time (t _c ) or longer is specified as the eye closing time. Then, the secondary arousal level calculation unit 112 calculates the ratio of the eye closing time per eye opening rate calculation section as the eye closing rate. For example, as shown in FIG. 12, when there are three eyes closed in the eye opening rate calculation interval (d _c ), the time when the eyes are closed for the first time (t _c1 ) and the time when the eyes are closed for the second time (t _c2 ) Is shorter than the preset eye closure time (t _c ) and is not treated as the eye closure time. On the other hand, since the time (t _c3 ) when the eye is closed for the third time is longer than the eye closing time (t _c ), it is treated as the eye closing time. Then, the closed eye rate is calculated based on the ratio of the closed eye time (t _c3 ) to the open eye rate calculation section (d _c ).

  Then, the secondary arousal level calculation unit 112 calculates the secondary level of the awakening level based on the closed eye rate by the awakening level calculation function (F13). The higher the open / close ratio, the longer the time during which the eyes are closed. Therefore, the awakening level in the secondary stage has a correlation with the opening / closing rate, and the secondary awakening level calculating unit 112 calculates the awakening level from the opening / closing rate.

Here, in this example, when the driver shifts from a normal driving state to a dozing state,
Assuming that the driving state with smooth steering continues for a long time as the primary stage and the eye closing time becomes long as the secondary stage, the arousal level based on the steering angle is the primary level awakening level, and the eyes are opened and closed. The awakening level based on is the secondary level awakening level.

  The first arousal level calculation unit 11 transmits a signal including the calculated primary level arousal level and secondary level arousal level to the arousal reduction level estimation unit 15.

  Next, the 2nd arousal level calculating part 12 is demonstrated using FIGS. FIG. 13 is a control block diagram of the second arousal level calculation unit 12, FIG. 14 is a diagram for explaining yawn detection, FIG. 15 is a diagram for explaining the movement of the neck, and FIG. FIG. 17 is a diagram for explaining the frequency of the secondary operation of the driver in the initial stage when the arousal level is lowered.

  As shown in FIG. 13, the second arousal level calculation unit 12 includes a yawning detection function (F21), a neck motion detection function (F22), a hand motion detection function (F23), and a wakefulness level calculation function (F24). ) And the awakening degree is calculated based on the processed image signal output from the image signal processing unit 14 and the vehicle speed detected by the vehicle speed detection unit 21. Drivers often take several actions to avoid falling asleep if they are sleepy while driving. For example, yawn, bend or turn the neck, or rub dry eyes with your hand. Such an operation is also called a secondary operation in the initial state when the arousal level is decreasing. Therefore, the second arousal level calculation unit 12 calculates a level of arousal different from that of the first awakening calculation unit 11 by detecting a secondary motion.

  The second arousal level calculation unit 12 detects the driver's yawn by the yawn detection function (F12). When the driver yawns, the mouth opens wide in the vertical direction. Since the change in the vertical direction of the mouth corresponds to a vector in the vertical direction calculated from the feature point of the mouth, the yawn is detected from the change in the vector in the vertical direction. In order to calculate the vector in the vertical direction, first, the second arousal level calculation unit 12 extracts facial feature points from the output signal of the image signal processing unit 14, and performs normalization processing of the extracted feature points. Then, the feature vector in the vertical direction is calculated by combining the distances between the feature points after the regularity. As shown in FIG. 14, this calculation process extracts vectors f0, f1,... M0, m1,... That are considered to be strongly reflected in the direction of the vertical motion, and these are extracted into one vector P = {f0, f1,. m0, m1,. Further, a difference vector X = P−Pave is calculated by setting an average value of the past n vectors P as an average vector Pave. The average vector Pave is used to ignore a slight difference. The difference vector X thus calculated becomes a vertical feature vector. Yawns stay open for a long time. Therefore, the second arousal level calculation unit 12 determines that yawning occurs when the scalar of the vertical feature vector is greater than a predetermined value and the state continues for a predetermined period or longer. On the other hand, the second arousal level calculation unit 12 determines that the vertical feature vector scalar is smaller than a predetermined value or the vertical feature vector scalar is larger than a predetermined value for a predetermined period or less. Does not judge yawning. The predetermined value may be calculated from the driver's face image, and the predetermined period may be set in advance.

  The second arousal level calculation unit 12 determines the movement of the neck by the neck motion detection function (F12). When the driver bends or turns his neck or head to the left or right (see FIG. 15 (a)), as shown in FIGS. 15 (b) and 15 (c), a straight x connecting both ends of the mouth among facial features. The inclination θ with respect to the axis changes with time. FIG. 15A shows a face image with the neck tilted to the right, (b) shows an enlarged image of the mouth of (a) at time (t−1), and (c) shows time (t Among the face images with the neck tilted to the left in), an enlarged image of the mouth is shown. The second arousal level calculation unit 12 calculates the tilt angle θ from the feature points at both ends in the left-right direction of the mouth in a predetermined cycle, and the difference between the tilt angles per cycle (Δθt) and the angular difference between the predetermined threshold values. And compare.

In FIG. 15 (b) and (c), the difference in inclination (Δθt) includes a tilt angle theta _t-1 time (t-1), is calculated from the difference between the tilt angle theta _t time (t) The Then, when the difference in inclination angle (Δθt) is equal to or greater than a predetermined threshold, the second arousal level calculation unit 12 determines that the driver is bending or turning his / her neck left and right to wake up to sleepiness. When the difference in inclination angle (Δθt) is smaller than a predetermined threshold value, it is determined that the movement is not a neck motion for waking up sleepiness.

  The second arousal level calculation unit 12 determines whether or not the driver's hand has touched his / her eyes or face by the hand motion detection function (F13). When the driver touches the eyes or face from the state where the driver does not touch the eyes or face, the shade of the area around the face in the captured image changes according to the movement of the hand. Therefore, the second arousal level calculation unit 12 detects hand movement by the optical flow method. First, as shown in FIG. 16, detection areas D1 to D4 are set on four sides around the face image. In addition, in the same figure, the symbols of the dots and arrows shown outside the screen are described outside the box in order to show the detection results of these detection regions D1 to D4 in an easily understandable manner.

  When an object crosses the detection areas D1 to D4, the direction is detected by the optical flow method. For example, the hand in the detection region D3 in FIG. 16 shows an optical flow that moves from the outside of the screen toward the face. In each detection area, the second arousal level calculation unit 12 sets flag 1 when an optical flow in a direction approaching the face is detected, flag 0 when an optical flow in a direction away from the face is detected, and flag n when neither is detected. Set. When the driver's hand does not touch the face, and the hand leaves the face after the hand touches the face, the driver's hand once passes the detection areas D1 to D4 from the outside to the inside. After passing, the hand passes from inside to outside. The flag at this time changes from flag 1 to flag 0. Therefore, the second awakening level calculation unit 12 determines that the hand has touched the face when the flag changes from 1 to 0.

The second arousal level calculation unit 12 calculates the frequency of secondary motion detected by each detection function by the arousal level calculation function (F24), and calculates the arousal level. In the second arousal level calculation unit 12, a secondary action frequency calculation section (d _f ) is set in advance, and the number of yawns, the number of neck movements, and the number of hand movements per section. Is calculated. In the example shown in FIG. 17, the number of yawns is one during the section (d _f ), the number of neck movements is one during the section (d _f ), and the number of hand movements is the section (d f _f ) 2 times in between. Then, the second arousal level calculation unit 12 calculates the frequency of the secondary action by adding the weights to the frequency of each action and adding the weights and dividing by the section (d _f ). Regarding the weight, in the initial state when the degree of arousal is decreasing, there are frequent behaviors and behaviors that do not occur frequently depending on the type of secondary behavior. Therefore, in this example, the secondary behavior increases the calculation accuracy of the arousal level by giving a weight according to the type of the secondary behavior. The weight may be set for each driver.

  And the 2nd awakening degree calculating part 12 calculates awakening degree based on the frequency of secondary action. The arousal level and the frequency of the secondary action have a correlation, and the second awakening level calculation unit 12 calculates the awakening level to be lower as the frequency of the secondary action is higher.

  Next, the arousal reduction level estimation unit 15 will be described with reference to FIGS. 18 shows a control block diagram of the arousal reduction level estimation unit 15, FIG. 19 shows a diagram for explaining the correlation (map) between the arousal level and the arousal reduction level, and FIG. 20 shows the calculated arousal level. It is a figure explaining the relationship between the transition of and a parameter | index of observation evaluation of dozing. FIG. 21 is a diagram for explaining the classification of the arousal level lowering level on the map of FIG.

  As shown in FIG. 18, the arousal reduction level estimation unit 15 includes an arousal level-decrease level map 151 and a wakefulness level decrease level estimation function (F31). Here, hereinafter, the arousal level calculated by the primary arousal level calculation unit 111 is defined as a primary level arousal level, the arousal level calculated by the secondary arousal level calculation unit 112 is defined as a secondary level arousal level, and a first arousal level calculation is performed. The awakening level calculated by the unit 11 is defined as a first awakening level, and the awakening level calculated by the second awakening level calculating unit 12 is defined as a second awakening level.

  In the arousal level-decrease level map, as shown in FIG. 19, the correlation between the primary level arousal level, the secondary level arousal level, the second arousal level, and the arousal level decrease level is stored in advance as a map. The arousal reduction level estimation unit 15 estimates the arousal reduction level with reference to the input primary stage awakening degree, secondary stage awakening degree, and second awakening degree, and the map. Here, the arousal reduction level estimated by the arousal reduction level estimation unit 15 indicates the degree of arousal to the driver as in the above arousal level, but the arousal level estimated from at least two or more indicators. The degree of decrease is shown.

  First, the map will be described with reference to FIG. The vertical axis shows the driver's arousal level, the horizontal axis is pleasant / uncomfortable, and shows the level of arousal corresponding to the level of the level of arousal downward from the origin. Let's say that it is 1 for a dull state, 2 for a sleepy state, and 3 for a dozing state. Further, the coordinates of the portion surrounded by a circle are equivalent to (value on the horizontal axis, value on the vertical axis).

  As for the vertical axis, the primary stage arousal level represents the initial stage of dozing (numerical stage) as a numerical value, and the secondary stage awakening level represents the awakening level of the next stage of dozing as a numerical value. The following conditions are set in the arousal reduction level estimation unit 15 so that the stage awakening level corresponds to the arousal reduction level 1 to 2 and the secondary stage arousal level corresponds to the awakening reduction level 2 to 3. Yes. The primary stage awakening degree, the secondary stage awakening degree, and the second awakening degree are values obtained by normalizing the awakening degree that is an output of each calculation unit.

なお、上記条件において、縦軸の覚醒低下レベルを推定する際に、横軸の覚醒レベルを推定するための指標である第２覚醒度を条件に加えている。運転手が眠くなった場合に、第１覚醒度と第２覚醒度はそれぞれ完全に独立した指標ではないため、縦軸の覚醒低下レベルを推定する際に、第２覚醒度を指標に加えることで推定精度を高めている。

In addition, in the said conditions, when estimating the arousal fall level of a vertical axis | shaft, the 2nd arousal level which is the parameter | index for estimating the arousal level of a horizontal axis is added to the conditions. When the driver becomes sleepy, the first wakefulness and the second wakefulness are not completely independent indexes. Therefore, when estimating the wakefulness reduction level on the vertical axis, the second wakefulness is added to the index. To improve the estimation accuracy.

  Next, on the horizontal axis, the second arousal level represents the level of discomfort experienced by the driver as a result of the driver trying to wake up as much as possible by struggling with or struggling with the sleeper. The higher the degree, the higher the degree of discomfort. The following conditions are set in the awakening reduction level estimation unit 15.

なお、平常(覚醒低下レベル０)時と居眠り(覚醒低下レベル３)の時には、通常、眠気との葛藤は見られないが、計算を簡略化するために設定する（図１９の点線で囲った丸部分に相当）。

In normal times (awakening reduction level 0) and dozing (awakening reduction level 3), there is usually no conflict with sleepiness, but it is set to simplify the calculation (enclosed by the dotted line in FIG. 19). Equivalent to the round part).

  In addition, in the arousal level-decrease level map 151, as shown by the respective coordinates in FIG. 19, the arousal decrease estimated for each coordinate point corresponding to the arousal decrease level on the vertical axis and the horizontal axis calculated under the above conditions. A level is assigned. For ease of explanation, FIG. 19 shows 12 discrete areas. However, the present invention is not limited to 12 areas, and is not limited to 12 areas. The level is set. Then, the arousal reduction level corresponding to each coordinate in FIG. 19 is the arousal reduction level estimated by the arousal reduction level estimation function (F31). Within the area indicated by each coordinate, the arousal reduction level sets the coordinate (3, 0) to the highest value, (2, 2), (2, 1), (2, 0), (1, 1) Set to be lower in the order of (1, 0), and set (0, 0) to the lowest value. For example, the level of wakefulness corresponding to coordinates (3, 0) indicates a decrease in wakefulness in a dozing state, the level corresponding to (2, 2) indicates a level of wakefulness that is sleepy and quite difficult, and (2, 1) Corresponding levels indicate sleepiness and a little bit of arousal decline, levels corresponding to (2, 0) indicate not sleepy, but indicate sleepiness, levels corresponding to (1, 0) indicate sleepless and dull state The level corresponding to (0, 0) indicates a normal state. Moreover, the arousal reduction level of the coordinates (1, 2), (1, 1), (2, 2), and (2, 1) indicates a state of conflict with sleepiness. Of the regions shown in FIG. 19, the regions indicated by coordinates (3, 2), (0, 2), and (0, 1) are provided for the sake of simplification of calculations as described above. No arousal reduction level has been assigned.

  Thereby, the arousal fall level estimation part 15 respond | corresponded to the primary stage awakening degree, the secondary stage awakening degree, and the 2nd awakening degree by the awakening degree degradation level estimation function (F31) using the arousal degree-decrease level map. Calculate the wakefulness level on the vertical and horizontal axes, and identify the coordinates corresponding to the estimated wakefulness level from the calculated wakefulness level on the vertical and horizontal axes, to estimate the final wakefulness level To do.

  Here, with reference to FIG. 20, when the driver actually changes from a normal state to a dozing state, the transition of the arousal reduction level based on the driver's subjectivity (graph b) and the arousal reduction level based on the evaluation of the observer Transition (graph a), subsidiary behavior transition (c), steering amplitude transition (d), and closed eye rate transition (graph e). FIG. 20 is a graph showing a time series of each of these indices. Note that the level of arousal reduction by the observer's evaluation is evaluated by looking at the driver's face image. The observer evaluates the facial image as 0 for normal, 1 for blurred, 2 for sleepy, 3 for sleepy, and 0-2 depending on the driver's degree of sleep pain. Each was evaluated as an observation evaluation index (degree of arousal reduction, degree of distress for sleep).

  As shown in the graphs a and b in FIG. 20A, the driver's subjectivity and the observer's rating are approximately the same, and it can be said that the process of the level of wakefulness reduction from normal to sleep is expressed. In the observation evaluation, it is considered that the disturbance in the waveform as compared with the subjectivity is caused by the driver's self-awakening. Since self-awakening is an event that occurs simultaneously with the driver's sleepiness, it is unlikely to appear in the driver's subjectivity.

  Then, as shown in FIG. 20B, the waveform obtained by superimposing the indexes shown in graphs c to e is roughly the waveform of the arousal reduction level by observation evaluation shown in graphs a and b in FIG. You can see that they match. Thereby, it can confirm that this example has improved the estimation precision of the wakefulness fall level by combining a different parameter | index and estimating a wakefulness fall level.

  Returning to FIG. 2, the control content of the stimulus control unit 16 will be described with reference to FIGS. FIG. 21 is a diagram for explaining the degree of stimulation with respect to the estimated wakefulness reduction level, and corresponds to FIG. FIG. 20 shows a map stored in the stimulus control unit 16.

  The stimulation control unit 16 sets a corresponding stimulation degree according to the arousal reduction level estimated from the arousal reduction level 15, and controls the monitor 40, the speaker 41, and the air conditioner 42 according to the stimulation degree. The degree of stimulation is shown in five stages of A to E, the arousal reduction level is the order from A to E, and E has the highest arousal reduction level.

  As shown in FIG. 21, areas A to E are divided according to the arousal reduction level corresponding to the map stored in the arousal level-decrease level map 151. Region A is a region for normal operation. Area B is an area where the driver is vague and where efforts are made to get up. Area C is an area in which the driver is aware of drowsiness and strives to get up. Region D is a region where efforts to occur are not allowed even though arousal reduction has occurred. A region E is a region close to drowsy driving.

  The stimulus control unit 16 stores the map shown in FIG. 21, and the stimulus control unit 16 sets the corresponding degree of stimulation from the estimated wakefulness reduction level. For example, when the estimated arousal reduction level is a level represented by coordinates (2, 2), the stimulus control unit 16 sets the degree of stimulation to C.

  The stimulation control unit 16 stores a control map according to the degree of stimulation (A to E) as shown in FIG. 22, and the stimulation control unit 16 refers to the map to stimulate the driver. To give control. At the stimulation degree A, the driver is in a normal state, so the stimulation control unit 16 does not perform any control. Alternatively, the estimated arousal reduction level may be displayed on the monitor 40. At the stimulus level B, the driver is vague but makes an effort to get up, so the stimulus control unit 16 displays a suggestion of a break on the monitor 40 or controls the air conditioner to lower the temperature in the passenger compartment. Or Or you may control to put music.

  At the stimulation degree C, the stimulation control unit 16 controls to display a break recommendation on the monitor 40 or to speak from the speaker 41. Or you may control the apparatus (not shown) which produces the scent of an awakening effect. At the degree of stimulation D, the driver feels dull or drowsy but does not make any effort to get up. Therefore, the stimulation control unit 16 displays on the monitor 40 that a break is instructed or a communication device (not shown) To inform the outside of the vehicle. Or you may control the apparatus (not shown) which produces the scent of an awakening effect. At the stimulation degree E, the stimulation control unit 16 issues a dozing alarm by using the monitor 40 and the speaker 41, or notifies the outside of the vehicle using a communication device (not shown). Alternatively, the driver may be directly stimulated by a vibrator device (not shown).

  In this way, the stimulus control unit 16 is set to control such that the strength of the stimulus given to the driver is weakened when the stimulus level is low, and the stimulus given to the driver when the stimulus level is high. The control is set to increase the strength of. Then, the stimulus control unit 16 controls the monitor 40 and the like according to the set degree of stimulation, and performs control so that the driver is awakened.

  The control device 10 calculates each arousal level by the first awakening level calculation unit 11 and the second awakening level calculation unit 12, estimates the arousal reduction level by the arousal reduction level estimation unit 15, and performs stimulation control by the stimulation control unit 16. Are performed in a predetermined cycle. When the controller 10 does not improve the arousal reduction level even when the driver is stimulated, in other words, the arousal reduction level after the stimulation is applied becomes higher than the arousal reduction level before the stimulation is applied. If this happens, the stimulus level increases, so a stimulus different from the previous stimulus is given. Thereby, it can prevent that a driver | operator's arousal fall level progresses.

  As described above, this example uses the first arousal level calculation unit, the second arousal level calculation unit different from the first arousal level calculation unit, the first awakening level and the second awakening level, An awakening reduction level estimation unit for estimating a driver's arousal reduction level; Unlike this example, for example, when the level of wakefulness reduction is estimated by only one of the first wakefulness level and the second wakefulness level, is the wakefulness decrease due to dozing or other factors, There is a possibility that it cannot be determined. In this example, since the lower level of wakefulness is estimated by combining different indexes of the first wakefulness and the second wakefulness, the estimation accuracy of the lowered level of wakefulness can be improved, and as a result, the state of the driver Can be grasped accurately. In addition, by using different indicators, not only the initial stage of wakefulness decline, but also the detection range of the dozing state is expanded, the wakefulness decline level is calculated after grasping the whole process of wakefulness drop, and the calculation accuracy is improved. be able to.

  Also, in this example, the arousal level is calculated based on at least one index of the error between the detected steering angle and the predicted steering angle, the steering amplitude calculated by the primary arousal level calculation unit 111, or the steering interval. Then, the arousal level is calculated from the closed eye rate based on the driver's face image. Thereby, the driving information of the driver can be accurately detected, and the calculation accuracy of the arousal level can be improved. In addition, since the arousal level is calculated using not only the vehicle information but also the steering angle and the eye closing rate, the calculation accuracy can be improved.

  This example also calculates the driver's primary level arousal level (primary level arousal level) based on the steering rate, which is vehicle information, and based on the driving state information such as the closed eye rate, the driver's secondary level Calculate the degree of awakening. Thus, at the primary stage, it is possible to calculate the decrease in driving performance by evaluating the movement of the vehicle due to the driver's action, and at the secondary stage, it is possible to doze from the normal state by evaluating the possibility of falling asleep. The state can be estimated using numerical values.

  Further, in this example, at least one of the driver's mouth movement, the neck movement, and the hand movement is detected, and the second arousal level is calculated based on the at least one movement. As a result, it is possible to detect a driver's pain, conflict, or effort to relieve drowsiness caused by a decrease in arousal.

  Moreover, this example gives the driver a stimulus different from the stimulus given to the driver when the arousal reduction level after giving the driver a stimulus is equal to or higher than the awakening reduction level giving the driver a stimulus. Thereby, control which gives another stimulus can be selected, and progress of a fall of a driver's arousal can be controlled.

  In this example, the primary stage arousal level, the secondary stage arousal level, and the second arousal level are calculated to estimate the arousal drop level, but first, only the primary stage arousal level is calculated, When the arousal level is lower than a predetermined arousal level threshold, the secondary level arousal level and the second arousal level may be calculated to estimate the arousal reduction level. This makes it possible to determine whether or not the decrease in wakefulness that occurred in the initial stage of the dozing state is due to dozing, so that the calculation level of the wakefulness decrease can be increased.

In the case where arousal decrease level is lowered as compared with the previous arousal decrease levels irritating is a period longer than a predetermined period to stimulate, stimulate the driver or to the driver Stimulation may be stopped. Accordingly, it is possible to prevent the driver from being unnecessarily stimulated and uncomfortable even though the driver is awake.
Note that, in the steering amplitude / steering interval calculation function (F2) of the primary arousal level calculation unit 111, the steering interval and the steering amplitude are discrete values without a peak during steering to the right or left. Alternatively, it may be converted into a continuous value by performing interpolation or extrapolation by a method such as linear interpolation, and calculation may be performed. Further, the primary arousal level calculation unit 111 updates each frequency distribution at the time when the data amount of the predicted steering angle, the steering amplitude, and the steering interval reaches a predetermined data amount in the steering distribution calculation function (F3), for example. Alternatively, it may be updated using the accumulated data of the predicted steering angle, steering amplitude, and steering interval at the end of driving. In addition, when the primary arousal level calculation unit 111 has a personal authentication function for identifying the individual driver, the frequency distribution may be calculated for each driver.

Note that the secondary arousal level calculation unit 112, for example, when looking at the feature points of the eyes and the feature points of the eyes in time series, if the feature points are not in the same position, the face is facing front. It may be determined that the opening / closing eye is not determined. Further, the secondary arousal level calculation unit 11
If the vehicle speed signal or the steering angle signal is lower than the threshold speed or low, or if the steering angle is larger than the threshold steering angle, it is determined that the face position or orientation is not stable, and the You may make it not determine.

  In this example, the awakening level to be calculated is not necessarily divided into two stages. For example, one awakening level may be calculated based on the steering angle data and the eye opening / closing data. Further, in this example, as shown in FIG. 19, the arousal reduction level is estimated by a map that correlates between the primary and secondary arousal levels and the second arousal level. A map that correlates between the first arousal level and the second arousal level may be used to estimate the arousal reduction level. Since the primary stage awakening level and the secondary stage awakening level indicate the level of arousal reduction indicated by the vertical axis in FIG. 19, the first awakening level calculation unit 11 uses the primary stage awakening level and the secondary stage awakening level. Thus, the wakefulness level corresponding to the wakefulness reduction level on the vertical axis may be calculated as the first wakefulness level.

  Further, in this example, the arousal reduction level estimation unit 15 calculates the arousal reduction level on the vertical axis of the map of FIG. 19 from the primary level awakening level, the secondary level awakening level, and the second awakening level, based on the above conditions. Although the arousal decrease level on the horizontal axis is estimated, the present invention is not necessarily limited to the above conditions. For example, the arousal decrease level on the vertical axis and the arousal decrease level on the horizontal axis may be estimated under the following conditions.

第１覚醒低下レベルは図１９のマップの縦軸の覚醒低下レベルに相当し、第２覚醒低下レベルは図１９のマップ横軸の覚醒低下レベルに相当する。ａ１〜ａ５及びｂ１〜ｂ５は、重みを示す係数であり、予め運転手の顔画像を観て、観測者が評定した第１覚醒低下レベルと第２覚醒低下レベルとを正解データとしてできるだけ一致するように、予め演算された値である。なお、当該係数は、統計的な学習方法により演算されてもよい。

The first arousal reduction level corresponds to the arousal reduction level on the vertical axis of the map of FIG. 19, and the second arousal reduction level corresponds to the arousal reduction level on the horizontal axis of the map of FIG. a1 to a5 and b1 to b5 are coefficients indicating weights, and the first arousal reduction level and the second arousal reduction level evaluated by the observer are matched as much as possible as correct data by looking at the driver's face image in advance. Thus, it is a value calculated in advance. Note that the coefficient may be calculated by a statistical learning method.

  The first arousal level calculating unit 11 corresponds to a “first arousal level calculating unit” according to the present invention, and the second arousal level calculating unit 12 is added to the “second arousal level calculating unit”. 15 corresponds to “awakening reduction level estimation means”, the steering angle detection unit 22 corresponds to “steering angle detection means”, the imaging device 31 corresponds to “imaging means”, and the stimulus control unit 16 corresponds to “driver awakening means”. Further, the primary stage awakening degree, the secondary stage awakening degree, and the first awakening degree correspond to the “first awakening degree” according to the present invention, and the second awakening degree corresponds to the “second awakening degree according to the present invention. Corresponds to "degree".

DESCRIPTION OF SYMBOLS 1 ... Arousal fall detection apparatus 10 ... Control apparatus 11 ... 1st arousal degree calculating part 111 ... Primary arousal degree calculating part 112 ... Secondary arousal degree calculating part 12 ... 2nd awakening degree calculating part 13 ... Steering angle signal processing part 14 ... Image signal processing unit 15 ... Arousal reduction level estimation unit 151 ... Arousal level-decrease level map 16 ... Stimulation control unit 20 ... Vehicle information detection unit 21 ... Steering angle detection unit 22 ... Vehicle speed detection unit 31 ... Imaging device 40 ... Monitor 41 ... Speaker 42 ... Air conditioner

Claims (7)

A first arousal degree calculating means for calculating a first arousal degree of the driver based on the vehicle information and information indicating the driving state of the driver;
Unlike the first wakefulness calculating means, second wakefulness calculating means for calculating a second wakefulness based on the driver's secondary action in the initial state of wakefulness reduction;
Waking reduction level estimation means for estimating the driver's waking reduction level,
The first awakening level includes a primary stage awakening level and a secondary stage awakening level,
The first arousal level calculating means includes:
By evaluating based-out vehicles movement in the vehicle information, the alertness of the initial stage of the driver, the computed as the primary stage alertness, and the initial phase on the basis of information of the operating conditions Calculate the awakening level in the next-stage dozing state as the secondary-level awakening level,
The arousal reduction level estimation means includes:
On the addition of the second degree of awakening to the primary stage alertness and the secondary stage alertness, it calculates the first arousal decrease the level of the driver, based on the second wakefulness, the first A second arousal reduction level that is an index different from the one arousal reduction level and represents the discomfort experienced by the driver is calculated, and the driver's final level is calculated using the first awakening reduction level and the second arousal reduction level. An arousal reduction detection apparatus characterized by estimating a low arousal reduction level. A steering angle detection means for detecting a steering angle included in the vehicle information;
The first arousal level calculating means includes:
It is calculated from the error between the steering angle detected by the steering angle detection means and the predicted steering angle predicted from the past steering angle detected by the steering angle detection means, and the steering angle detected by the steering angle detection means. The first arousal level is calculated on the basis of at least one index of a steering amplitude calculated from a steering amplitude detected by the steering angle detection unit or a steering interval calculated from a steering angle detected by the steering angle detection unit. 1. The arousal reduction detection device according to 1. Further comprising imaging means for imaging an image of at least the eyes of the driver;
The first arousal level calculating means includes:
From the image captured by the imaging means, the closed eye rate of the driver's eyes is detected as information on the driving state,
The wakefulness reduction detection apparatus according to claim 1 or 2, wherein the first wakefulness level is calculated based on the eye-closing rate. It further has an imaging means for capturing an image of the driver's face,
The second awakening degree calculating means includes:
From the image picked up by the image pickup means, as the driving state, detect the movement of the mouth of the driver, the movement of the neck of the driver, or the movement of the hand of the driver,
The wakefulness drop detection device according to any one of claims 1 to 3, wherein the second wakefulness level is calculated based on the at least one action. The arousal reduction level estimation means includes:
When the first awakening level in the initial stage dozing state is lower than a predetermined awakening level threshold, the first awakening level and the second awakening level in the next stage dozing state are used to The wakefulness drop detection apparatus according to claim 1, wherein the drop level is estimated. In accordance with the arousal reduction level estimated by the arousal reduction level estimation means, further comprising driver awakening means for controlling the vehicle so as to stimulate the driver to awaken the driver,
The stimulus is a first stimulus and a second stimulus having a higher degree of stimulation than the first stimulus,
The driver awakening means includes
When the level of wakefulness after giving the first stimulus to the driver is equal to or higher than the level of wakefulness giving the first stimulus to the driver, the second stimulus is used as the driving The arousal reduction detecting device according to any one of claims 1 to 5, wherein the device is applied to a hand. In accordance with the arousal reduction level estimated by the arousal reduction level estimation means, it further comprises driver awakening means for controlling the vehicle so as to stimulate the driver and awaken the driver at a predetermined cycle,
The driver awakening means includes
When the arousal reduction level is lower than the arousal reduction level before applying the stimulus, the driver is stimulated with a cycle longer than the predetermined cycle, or the stimulus to the driver is stopped. The wakefulness drop detection device according to any one of claims 1 to 5, wherein:

Images