JP2018183248A - Drowsiness determination device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a drowsiness determination device capable of accurately determining drowsiness of each object person.SOLUTION: A drowsiness determination device 100 includes: an imaging camera 110 for imaging a driver; an image processing part 120 for generating eyelid information on opening and closing of the eyelids of the driver from the imaged data; and a control part 130 for selecting a pattern on the opening and closing of the eyelids most suitable for the driver based on the eyelid information from the image processing part 120, and setting a threshold for drowsiness determination based on the selected pattern. The control part 130 determines the drowsiness of the driver using the set threshold.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a drowsiness determination device that determines drowsiness of a driver or the like.

  There is a system in which an imaging camera is installed in front of the driver, sleepiness is determined by checking the degree of opening / closing of the driver's bag, and a warning is issued. For example, Patent Document 1 discloses a measurement system that quantitatively measures sleepiness based on the blinking opening / closing time on the assumption that there is a correlation between the blinking opening / closing time and sleepiness. Patent Document 2 discloses a sleepiness state determination device that more appropriately determines the driver's sleepiness level by providing a plurality of threshold values for the driver's eyes closed. Patent Document 3 discloses a biological state determination device that estimates the presence or absence of drowsiness from a driver's face image, brain waves, or the like.

JP 2009-95387 A JP-A-44922652 Japanese Patent Laid-Open No. 5423787

  One conventional drowsiness determination device calculates a score of the degree of opening / closing of a heel by analyzing data obtained by imaging a driver's face, and determines drowsiness based on the score. FIG. 8A shows an example in which the score of the opening / closing degree of the heel is high, and FIG. 8B shows an example in which the score of the opening / closing degree of the heel is low. The numerical values of the scores (49, 52) are the open / closed degrees of the right eye and the left eye, and the higher the score, the more open the eyes, and the lower the score, the closer the eyes.

The drowsiness was determined uniformly by comparing the length of time that the driver's eyelids were closed, the number of blinks, and the score with a threshold value.
(Judgment of normal state)
When the average score of the degree of open / closed wings over an arbitrary number of seconds is equal to or greater than a certain value, it is determined that there is no sleepiness, that is, a normal state. For example, when the average score (including the right eye and the left eye) of the eyelid opening / closing degree for 10 seconds is 40 or more, the normal state is estimated.
(Determination of sleeping state)
A sleep state is determined when the average score of the degree of open / closed occupancy for an arbitrary number of seconds is equal to or less than a certain value. For example, when the average score (including the right eye and left eye) of the heel opening / closing degree for 10 seconds is 15 or less, it is determined that the person is sleeping.
(Determination of depression and depression)
The depressed state is a state in which the user is likely to fall asleep, that is, a state in which a transition is made from a normal state to a sleeping state. If the score of the eyelid opening / closing degree is equal to or smaller than a certain value and the blinking time is equal to or larger than a certain value within an arbitrary number of seconds, it is determined that the state of depression is in a state of depression. For example, when there are three or more blinks in which the value of the eyelid opening / closing degree score of 30 or less continues for 500 ms or more in 10 seconds, it is determined that the subject is in a depressed state.

  However, since the manner of opening and closing the bag when feeling sleepy varies among individuals, the uniform determination method as described above cannot accurately determine sleepiness for each individual. FIG. 9 is an example of opening and closing patterns of some heels when drowsiness is felt. FIG. 9A shows a human pattern 1 in which when the user feels drowsiness, the degree of opening and closing of the heel decreases, that is, becomes thin. In pattern 1, the score is relatively small. FIG. 9 (B) shows a pattern 2 of a person who tends to lengthen the time for opening and closing one bag. FIG. 9C shows a person's pattern 3 in which only the number of blinks increases. FIG. 9D shows a pattern 4 of a person whose number of blinks and time for opening and closing one eyelid are long. As you can see, there are actually variations in the work when you feel sleepy. Therefore, if the threshold value for determining sleepiness is determined uniformly as in the prior art, there is a possibility that an erroneous determination will occur in the sleepiness determination for different persons.

  This invention solves such a conventional subject, and it aims at providing the sleepiness determination apparatus which can perform the determination of sleepiness for every subject accurately.

  The drowsiness determination device according to the present invention includes a storage unit that stores a plurality of pieces of pattern information relating to opening / closing of a heel when drowsiness, an estimation unit that estimates that the subject feels drowsiness, Acquisition means for acquiring eyelid information related to opening / closing of the subject's eyelid when the person feels sleepy, and selection of selecting pattern information corresponding to the eyelid information acquired by the acquisition means from among the plurality of pattern information Means, setting means for setting a threshold for sleepiness determination based on the selected pattern information, and sleepiness determination means for determining sleepiness of the subject using the set threshold.

  Preferably, the estimation means estimates that the subject feels drowsy when the subject has been driving for a certain time or longer. Preferably, the estimation means estimates that the subject feels sleepy when the subject yawns. Preferably, the estimation means estimates that the subject feels drowsy when sudden braking is activated without the subject seeing. Preferably, the estimation means compares the heart rate pattern of the subject with the heart rate pattern at the time of falling asleep, and estimates that the subject feels sleepy when there is a correlation between the two. Preferably, the drowsiness determination device further includes an identification unit for identifying the subject, and a storage unit for storing a threshold value in association with the identified subject, wherein the setting unit is the subject identified by the identification unit. Is read from the storage means, and the read threshold is set. Preferably, the sleepiness determination device further includes warning means for issuing a warning when the sleepiness determination means determines that there is sleepiness.

  According to the present invention, it is assumed that the subject feels drowsy, and pattern information relating to the opening / closing of the eyelid is selected from among the plurality of pattern information based on the eyelid information relating to the opening / closing of the subject's eyelid. Therefore, it is possible to automatically set a threshold for determining sleepiness that is optimal for the target person, which makes it possible to determine sleepiness with high accuracy in consideration of individual differences in how to open and close the eyelid when feeling sleepiness It can be performed.

It is a block diagram which shows the structure of the drowsiness determination apparatus which concerns on 1st Example of this invention. It is a figure which shows the example of attachment of an imaging camera. It is a figure which shows the functional structure of the sleepiness determination program which concerns on 1st Example of this invention. It is an example of the table which prescribes | regulates the relationship between a driver | operator and the selected basic pattern. It is a figure which shows the flow of the drowsiness determination operation | movement which concerns on the Example of this invention. It is a figure which shows the structure of the 3rd Example of this invention. It is a figure which shows the structure of the 4th Example of this invention. It is a figure explaining the conventional sleepiness determination method. It is a figure which illustrates the opening-and-closing pattern of the bag when feeling sleepy.

  Next, embodiments of the present invention will be described in detail with reference to the drawings. The drowsiness determination device of the present invention is preferably mounted on a vehicle such as an automobile and determines the drowsiness of the driver. However, this is only an example, and the sleepiness of other persons may be determined. In addition, the sleepiness determination device of the present invention may be implemented alone, or may cooperate or link with other devices, for example, an in-vehicle device equipped with a driving support function or the like. .

  FIG. 1 is a block diagram showing the configuration of the drowsiness determination device according to the first embodiment of the present invention. The sleepiness determination apparatus 100 includes an imaging camera 110 that captures a driver's face, an image processing unit 120 that processes image data from the imaging camera 110, a control unit 130 that performs sleepiness determination, and a storage unit 140 that stores various data. , And an external interface (I / F) 150 that enables transmission / reception of data to / from the outside.

  The imaging camera 110 is installed at a position where the driver's face can be imaged. For example, as shown in FIG. 2, the imaging camera 110 is attached on the column cover 112 and images the face of the driver 116 through the steering 114. The imaging camera 110 may be a normal camera, or may include an infrared LED that irradiates the face of the driver 116 and an infrared camera that receives the reflected light. Data captured by the imaging camera 110 is provided to the image processing unit 120.

  The image processing unit 120 performs image analysis on the received imaging data, extracts a driver's face image, and extracts feature points related to wrinkles included in the extracted face image. For example, the area or shape of the black or white part of the left or right eye is extracted from the face image, or the distance between the upper eyelid and the lower eyelid is extracted. The wrinkle information related to the opening and closing of the wrinkles is provided to the control unit 130.

  The control unit 130 calculates a score of the heel opening / closing degree based on the heel information provided from the image processing unit 120, and determines the driver's sleepiness based on the score. The score of the eyelid opening / closing degree is a value that quantitatively represents the degree of opening of the eyelid of each of the right eye and the left eye. The larger the eyelid opening / closing score, the more the eyelid opens, and the smaller the eyelid opening / closing score, the more the eyelid opens. Means closed. In a preferred embodiment, the control unit 130 includes a microcontroller or a microprocessor including a ROM / RAM and performs sleepiness determination by executing a program stored in the ROM / RAM.

  The storage unit 140 stores several pieces of basic pattern information related to the patterns 1 to 4 in FIG. The basic pattern 1 is a wrinkle opening / closing pattern in which a score below a certain value decreases by a certain value within a certain period. For example, the average value of the left eye and right eye scores is 40 or less, and this is the opening and closing of the eyelids that decreases to 30 or less within a certain period. The basic pattern 2 is a cocoon opening / closing pattern in which the cocoon opening / closing time is extended once within a certain period. For example, opening / closing of a cocoon such that a cocoon opening / closing time of 0.5 seconds would be 1 second or longer. The basic pattern 3 is an eyelid opening / closing pattern in which the number of blinks exceeds a certain value within a certain period. For example, the opening and closing of the kite so that the number of blinks is 5 times becomes 7 times. The basic pattern 4 is an open / close pattern of the eyelid that combines the basic pattern 2 and the basic pattern 3. The pattern information is not limited to the four basic patterns shown in FIG. 9 and may include other basic patterns.

  The external I / F 150 enables connection to an external device or an external information source, and transmits / receives data to / from the outside. Further, the external I / F 150 can include an externally connectable memory, communication means, user input, audio output device, and the like.

  FIG. 3 is a diagram showing a functional configuration of the drowsiness determination program according to the first embodiment of the present invention. The sleepiness determination program 200 includes a sleepiness state estimation unit 210, a heel opening / closing pattern analysis unit 220, a basic pattern selection unit 230, a threshold setting unit 240, a sleepiness determination unit 250, and a sleepiness warning unit 260.

  The sleepiness state estimation unit 210 estimates a situation in which the driver feels sleepiness. In the present embodiment, the situation where the driver feels sleepy is when the driver drives the vehicle for a certain time or more. For example, the sleepiness state estimation unit 210 determines whether or not the driver has driven for a predetermined time or more based on information acquired from the external I / F 150, and based on the determination result, whether or not the driver is sleepy. Guess. For example, when a navigation device or other in-vehicle device is connected via the external I / F 150, the drowsiness state estimation unit 210 is configured to turn on / off information on the ignition switch of the vehicle, vehicle speed pulse information, distance information, and navigation device. The driving time of the driver is determined from the operation information. When the sleepiness state estimation unit 210 estimates the sleepiness state, the sleepiness state estimation unit 210 causes the bag opening / closing pattern analysis unit 220 to analyze the driver's bag information, causes the basic pattern selection unit 230 to select a basic pattern, and The threshold setting unit 240 sets a threshold based on the selected basic pattern.

  The heel opening / closing pattern analysis unit 220 analyzes the driver's heel information provided from the image processing unit 120 when it is estimated that the driver feels sleepy. In a preferred embodiment, the eyelid opening / closing pattern analyzing unit 220 calculates the eyelid opening / closing degree score from the eyelid information, and obtains the eyelid opening / closing time, the number of blinks, etc. within a certain period. The opening / closing time of the heel and the number of blinks can be obtained from the score of the heel opening / closing degree.For example, when a plurality of local minimum and maximum values are included in the score calculated from the heel information sampled within a certain period, The opening / closing time of one kite is the time from the minimum value to the maximum value, the time from the maximum value to the minimum value, the time from the minimum value to the next minimum value, or from the maximum value to the next maximum value. The number of blinks can be determined by how many times the opening / closing time of one kite is performed.

  The basic pattern selection unit 230 compares the eyelid opening / closing pattern analyzed by the eyelid opening / closing pattern analysis unit 220 with the basic pattern stored in the storage unit 140, and selects a basic pattern that matches or approximates the analyzed eyelid opening / closing pattern. To do. The analysis result of the heel opening / closing pattern analysis unit 220 is analysis of heel information when the sleepiness state estimation unit 210 estimates that the driver feels drowsiness. Choose a basic pattern that suits how you want to open and close your heels when you feel sleepy.

  The threshold setting unit 240 sets a threshold for determining sleepiness based on the basic pattern selected by the basic pattern selection unit 230. When the basic pattern 1 is selected, the threshold value setting unit 240 sets a threshold value that can be used to determine the opening / closing of the cocoon whose score less than or equal to a certain value decreases within a certain period. When the basic pattern 2 is selected, the threshold value setting unit 240 sets a threshold value that can be used to determine whether to open / close the kite that has a long kite opening / closing time within a certain period. When the basic pattern 3 is selected, a threshold value is set that can determine whether the eyelid is opened or closed within a certain period of time when the number of blinks exceeds a certain value.

  If it is possible to identify a person when analyzing the open / close pattern, it is possible to store the identification information of the person and the basic pattern selection information in association with the identification information. Then, the next time the same person gets on board, the basic pattern selected for the person can be set without analyzing the heel opening / closing pattern. For example, as shown in FIG. 4, the relationship between the driver identification information and basic pattern selection information or threshold setting information when the driver is drowsy is stored in the storage unit 140.

  For example, a face image captured by the imaging camera 110 can be used as the driver identification information. In this case, it is determined whether or not the driver is the same driver by pattern matching the face image stored as the driver identification information and the driver's face image captured by the imaging camera 110 this time. In addition to the identification information, the user may input information for identifying who the driver is from the external I / F 150.

  The sleepiness determination unit 250 determines sleepiness based on the driver's habit information provided from the image processing unit 120 using the threshold set by the threshold setting unit 240. The sleepiness determination unit 250 determines a normal state, a sleeping state, a depressed state, and the like. For example, it is assumed that the basic pattern 3 is selected by the basic pattern selection unit 230, and the threshold value for determining the depressed state is set to 7 blinks in 10 seconds. At this time, if the analysis result by the eyelid opening / closing pattern analysis unit 220 is that there are 8 blinks in 10 seconds, the drowsiness determination unit 250 determines that the driver is in a depressed state, that is, in a state of feeling drowsiness. .

  The sleepiness warning unit 260 issues a warning to the driver when the sleepiness determination unit 250 determines that the patient is sleeping or depressed. Although the warning method is not particularly limited, for example, a warning message can be displayed on the display unit via the external I / F 150 or a voice warning can be issued from the voice output unit.

  Next, the operation of the drowsiness determination device according to the present embodiment will be described with reference to the flowcharts of FIGS. FIG. 5A shows an example of setting a threshold when the driver identification function is not provided. First, it is estimated by the sleepiness state estimation unit 210 whether or not the driver feels sleepiness (S100). When it is estimated that the user feels sleepy, the wrinkle information at that time is analyzed by the wrinkle opening / closing pattern analysis unit 220 (S102), a basic pattern that matches or approximates the analysis result is selected (S104), and the threshold setting unit 240 sets a threshold corresponding to the selected basic pattern (S106). Thereafter, drowsiness determination based on the driver's habit information is performed using the set threshold value. In this case, the threshold value is set every time the operation of the drowsiness determination device 100 is started or whenever an instruction for setting the threshold value is received from the user.

  FIG. 5B is an example of setting a threshold when a driver identification function is provided. First, the driver is identified (S120). As described above, for example, a driver's face image can be used as the identification method. Next, the threshold value setting unit 240 searches whether or not the driver has been registered in the past (see FIG. 4), and if registered (S122), selection information or setting information about the driver. And the threshold value is set (S124). If the driver is not registered, a threshold value for the driver is set according to the steps shown in FIG. 5A, and the threshold value is stored in the storage unit 140 in association with the driver.

  According to the present embodiment, a basic pattern for opening and closing a plurality of eaves is prepared, and a basic pattern suitable for the driver is selected based on the analysis result of the eaves information of the driver when it is estimated that he / she feels sleepy Since this is done, it is possible to set a threshold for sleepiness determination according to the driver, and to perform accurate sleepiness determination.

  Next, a second embodiment of the present invention will be described. In the first embodiment, it is assumed that the driver feels drowsy when the driver has driven for a certain period of time or longer, but in the second embodiment, the driver yawns when the driver yawns. Guess you feel sleepy.

  The image processing unit 120 extracts feature points representing the mouth or lip outline of the face image and feature points of white teeth, and provides the face image including these feature points to the control unit 130. The sleepiness state estimation unit 210 determines whether or not the driver yawns based on the face image provided from the image processing unit 120. When it is determined by the sleepiness state estimation unit 210 that the driver yawns, it is estimated that the driver feels sleepy. Note that the number of times the driver yawns may be considered. For example, it may be estimated that the driver feels sleepy when the number of yawns is more than once. The other operations are the same as those in the first embodiment. When the driver's sleepiness is estimated, the sleepiness state estimation unit 210 causes the basic pattern selection unit 230 to select a basic pattern, and The threshold setting unit 240 sets a threshold based on the selected basic pattern.

  According to the second embodiment, it can be estimated whether or not the driver is drowsy based on the driver's biological information, so that the determination accuracy when selecting the basic pattern can be improved. It is to be noted that when both the estimation of the sleepiness state according to the first embodiment and the estimation of the sleepiness state according to the second embodiment are combined, that is, when the driver drove for a certain time and yawns, the driver You may make it guess that you are feeling.

  Next, a third embodiment of the present invention will be described. In the third embodiment, the sleepiness state estimation unit 210 estimates that the driver feels sleepy when automatic braking (or sudden braking) is performed even though the driver is not looking away. To do. In other words, it is assumed that the driver is drowsy because the driver should have looked ahead but the vehicle was automatically braked or suddenly stopped. Note that the automatic braking is performed when the distance from the front object becomes equal to or less than a predetermined value by the obstacle sensor mounted on the vehicle.

  FIG. 6 is a configuration example of the third embodiment. The drowsiness determination device 100 is connected to the vehicle information providing unit 300 via the external I / F 150. The vehicle information providing unit 300 includes, for example, an in-vehicle bus that acquires vehicle information, and the vehicle information includes vehicle speed information, brake information, steering information, blinker information, and the like. The drowsiness determination device 100 checks whether or not the driver is looking forward from the angle of the driver's face image provided from the image processing unit 120 or the driver's gaze direction included in the face image. Whether or not the vehicle is suddenly braked or stopped based on the vehicle information provided by the vehicle information providing unit 300, and whether the driver feels drowsy based on these determination results Guess whether or not. For example, if the driver is not looking away from the driver's face image when the automatic braking is performed, the sleepiness state estimation unit 210 estimates that the driver feels sleepy. Further, whether or not the user is looking away can also be determined based on whether or not the driver's line-of-sight direction is gazing at the side mirror or the rearview mirror. Other operations are the same as those in the first and second embodiments. It is also possible to combine the estimation of the sleepiness state of the third embodiment with the estimation of the sleepiness state of the first and second embodiments. For example, if the driver has been driving for a certain period of time, the driver is yawning, and is not looking away, the driver is in a sleepy state when sudden braking or stopping is performed. You may do it.

  Next, a fourth embodiment of the present invention will be described. In the fourth example, when the sleepiness state estimation unit 210 compares the driver's heartbeat pattern with the heartbeat pattern at the time of falling asleep as biometric information and determines that there is a correlation, the driver feels sleepy. Infer.

  FIG. 7 is a configuration example of the fourth embodiment. The drowsiness determination device 100 is connected to the wearable end 400 via the external I / F 150. Wearable terminal 400 is worn by the driver, measures the driver's biological information, such as heart rate and blood pressure, and provides the measurement result to drowsiness determination device 100. The sleepiness state estimation unit 210 stores the heart rate pattern information at the time of falling asleep in advance, the heart rate of the driver from the wearable terminal 400 or the pattern thereof, and the heart rate at the time of sleep or the pattern prepared in advance. When it is determined that there is a correlation between the two, it is determined that the driver feels sleepy. Other operations are the same as those in the first to third embodiments. It is also possible to combine the estimation of the sleepiness state of the fourth embodiment with the estimation of the sleepiness state of the first to third embodiments.

  As another aspect, the wearable terminal 400 may store a heartbeat pattern at the time of sleep. For example, the driver sleeps with the wearable terminal 400 worn, and the wearable terminal 400 stores the heart rate at the time of falling asleep. When the user himself / herself inputs the sleep time, the heart rate or the pattern at the time of sleep can be extracted. Then, the driver wearing the wearable terminal 400 starts driving, and then the wearable terminal 400 compares the heartbeat pattern during sleep with the heartbeat pattern during driving, and provides the correlation result to the sleepiness determination apparatus 100. You may do it. The sleepiness state estimation unit 210 estimates whether the driver feels sleepy based on the correlation result. In the present embodiment, the example in which the wearable terminal 400 measures the heart rate of the driver is shown, but the present invention is not limited to this, and a heart rate sensor that detects the heart rate can also be used.

  In any of the first to fourth embodiments, the basic pattern related to the optimal lid opening / closing when the driver feels sleepy is selected from the saddle information when the driver is estimated to feel sleepy. As a result, the optimal threshold for sleepiness determination according to the driver can be set, and highly accurate sleepiness determination can be performed for each driver.

  The preferred embodiment of the present invention has been described in detail above. However, the present invention is not limited to the specific embodiment, and various modifications and variations are possible within the scope of the gist of the invention described in the claims. It can be changed.

100: Sleepiness determination device 110: Imaging camera 120: Image processing unit 130: Control unit 140: Storage unit 150: External I / F
200: Sleepiness determination program 300: Vehicle information providing unit 400: Wearable terminal

Claims (7)

Storage means for storing a plurality of pieces of pattern information relating to opening and closing of the heel when drowsiness,
A guessing means to guess that the subject feels sleepy,
Acquisition means for acquiring wrinkle information related to opening and closing of the subject's heel when the subject feels sleepy by the estimation means;
Selecting means for selecting pattern information corresponding to the wrinkle information acquired by the acquiring means from the plurality of pattern information;
Setting means for setting a threshold for sleepiness determination based on the selected pattern information;
Sleepiness determination means for determining sleepiness of a subject using the set threshold;
A drowsiness determination device having The sleepiness determination apparatus according to claim 1, wherein the estimation means estimates that the subject feels drowsy when the subject is driving for a predetermined time or longer. The sleepiness determination apparatus according to claim 1, wherein the estimation means estimates that the subject feels sleepy when the subject yawns. The sleepiness determination apparatus according to any one of claims 1 to 3, wherein the estimation means estimates that the subject feels drowsy when a sudden brake is activated without the subject looking aside. 5. The method according to claim 1, wherein the estimating means compares the heart rate pattern of the subject and the heart rate pattern at the time of falling asleep, and estimates that the subject feels drowsy when there is a correlation between the two. The drowsiness determination device described. The drowsiness determination device further includes an identification unit that identifies the subject, and a storage unit that stores a threshold in association with the identified subject, and the setting unit corresponds to the subject identified by the identification unit. The drowsiness determination device according to claim 1, wherein a threshold value to be read is read from the storage unit, and the read threshold value is set. The drowsiness determination device according to any one of claims 1 to 6, further comprising a warning unit that issues a warning when the drowsiness determination unit determines that there is drowsiness.

Images