JP2010049337A - Blinking condition detecting device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a blinking condition detecting device capable of detecting driver's blinking condition with high precision even if detection of change with time of degree of eye opening based on image data is interrupted. <P>SOLUTION: A blinking condition detection ECU 1 detects driver's degree of eye opening detected from the image data imaged by a face image capturing camera 2. Here, the blinking condition detecting device interpolates a missing part if missing occurs in the detection of degree of eye opening. When interpolating the missing part, change with time of driver's degree of eye opening is stored as history of degree of eye opening for 10 minutes, the sections being similar to front and rear parts of the missing part are searched from the history of degree of eye opening, and the missing part is interpolated, based on waveforms in front and rear parts of the similar sections. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a blink state detection device, and more particularly to a blink state detection device suitable for use in detecting the drowsiness state of a vehicle driver.

  Conventionally, by detecting the drowsiness state of the driver of the vehicle, it has been used to prevent the drowsy driving. In addition, in order to prevent a drowsy driving, a wakefulness detection method that detects a driver's drowsiness (wakefulness state) based on a change over time of a driver's blink has been conventionally disclosed (for example, Patent Document 1). reference).

In this awakening state detection method, the face of the vehicle driver is imaged by a CCD camera, and the blink observation pattern on the face of the vehicle driver is observed. From the observation of the observation pattern of the blink of the vehicle driver, a change with time of the blink of the vehicle driver is obtained. This temporal change is converted into a voltage waveform, and the awakening state of the vehicle driver is detected from the converted voltage waveform.
JP-A-11-147428

  By the way, in the wakefulness detection method disclosed in the above-mentioned Patent Document 1, detection of change with time of blinking based on image data obtained by imaging with a CCD camera due to disturbance such as illuminance change or vibration in the vehicle while the vehicle is running. There may be interruptions. As described above, when detection of a change with time of blinking (degree of eye opening) is interrupted, a part of the voltage waveform is lost and a continuous voltage waveform cannot be obtained. As a result, the detection accuracy of the blinking state of the vehicle driver is reduced. This decrease in the detection accuracy of the blinking state leads to a decrease in the monitoring accuracy of the dozing state.

  Therefore, an object of the present invention is to provide a blink state detection device that can detect a blink state with high accuracy even when detection of a change with time in the degree of eye opening based on image data is interrupted.

  A blink state detection device according to the present invention that has solved the above-described problems includes eye-opening degree detection means for detecting the degree of eye opening of a driver from image data obtained by photographing a driver's face in a vehicle. A blink state detection device that detects a driver's blink state based on a change in the degree of eye opening over time, the eye opening degree change storage unit that stores the change over time of the driver's eye opening degree, and the driver by the eye opening degree detection unit Eye opening degree detection missing detecting means for detecting a lack of eye opening degree detection, and when the eye opening degree detecting means detects a lack of eye opening degree detection of the driver, the eye opening degree stored in the eye opening degree change storage means over time Interpolating means for interpolating a temporal change in the degree of opening of the eye when the lack of eye opening degree detection of the driver is detected based on the change is provided.

  The blink state detection device according to the present invention detects a driver's eye opening degree based on a temporal change of the eye opening degree stored in the eye opening degree change storage means when a lack of eye opening degree detection of the driver is detected. Interpolate the change over time in the degree of eye opening at the time when the omission was detected. Since the driver's blink pattern is often similar for each driver, the eye opening degree over time based on the image data is interpolated by interpolating the missing portion based on the time change of the eye opening degree stored in the eye opening degree change storage means. Even when the change detection is interrupted, the blinking state can be detected with high accuracy.

  Here, the interpolation means is similar to the temporal change of the eye opening degree before and after the time when the lack of detection of the eye opening degree of the driver is detected from the temporal change of the eye opening degree stored in the eye opening degree change storage means. The time variation of the degree is extracted, and based on the extracted time variation of the eye opening degree, the time variation of the eye opening degree at the time when the lack of eye opening degree detection of the driver is detected is interpolated. Can do.

  In this way, by extracting the temporal change of the eye opening degree similar to the temporal change of the eye opening degree before and after the time when the lack of eye opening degree detection of the driver is detected, the similarity of the shape after the remaining part of the missing part is extracted. The shape to be interpolated can be extracted. Therefore, the missing portion can be interpolated with higher accuracy.

  According to the blink state detection device according to the present invention, it is possible to detect the blink state with high accuracy even when the detection of the temporal change of the eye opening degree based on the image data is interrupted.

  Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the description of the drawings, the same elements are denoted by the same reference numerals, and redundant description is omitted. For the convenience of illustration, the dimensional ratios in the drawings do not necessarily match those described.

  FIG. 1 is a block diagram of a dozing prevention apparatus including a blink state detection apparatus according to the present embodiment. The dozing prevention device is a device that is provided in a vehicle and prevents a driver from falling asleep. As shown in FIG. 1, the dozing prevention device includes a blink state detection ECU (Electronic control unit) 1 that is a blink state detection device of the present invention. . The dozing prevention device includes a face image capturing camera 2, a face image processing ECU 3, and a dozing prevention ECU 4. Further, the dozing prevention ECU 4 is provided with a stimulus applying means for applying a stimulus to the driver in the dozing state.

  The blinking state detection ECU 1 is a computer mounted on a vehicle, and includes a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), an input / output interface, and the like. The blink state detection ECU 1 includes an eye opening degree calculation unit 11, an eye opening degree history storage unit 12, a loss detection unit 13, a loss interpolation unit 14, and a blink state detection unit 15.

  The face image capturing camera 2 is installed on a column cover, for example, and acquires a driver's face image. The face image capturing camera 2 is, for example, a near-infrared camera, and captures the driver's face by receiving reflected light reflected by the near-infrared ray irradiated from a near-infrared irradiation device (not shown) on the driver's face. For example, the face image capturing camera 2 can acquire image data of 30 frames per second. The face image capturing camera 2 transmits the acquired image data to the face image processing ECU 3.

  The face image processing ECU 3 is electrically connected to the face image capturing camera 2. The face image processing ECU 3 performs image processing on the image data transmitted from the face image capturing camera 2 to recognize the driver's face image. In the face image processing ECU 3, for example, features such as face contour edge extraction processing for detecting the contour of the driver's face, face orientation angle determination processing for determining the driver's face orientation angle, facial parts (eyes, nose, mouth), etc. Facial feature point extraction processing for extracting points is executed. After performing these image processes, the face image processing ECU 3 detects the eyelid position, and transmits the eyelid position information related to the eyelid position to the eye opening degree calculation unit 11 in the blinking state detection ECU 1.

  The eye opening degree calculation unit 11 calculates the eye opening degree of the driver based on the eyelid position information transmitted from the face image processing ECU 3. At this time, when the degree of eye opening cannot be calculated, the degree of eye opening cannot be calculated. The eye opening degree calculation unit 11 performs a fifo on the calculated eye opening degree and stores the eye opening degree history storage unit 12 as an eye opening degree history. Here, the eye opening degree history storage unit 12 stores the eye opening degree for 10 minutes as the eye opening degree history.

  Further, the eye opening degree calculation unit 11 stores the eye opening degree history per unit time (hereinafter referred to as “unit time eye opening degree history”) in the eye opening degree history storage unit 12. The unit time here is set to 10 seconds. For this reason, the eye opening degree for 300 frames is stored as the unit time eye opening degree history. The eye opening degree history storage unit 12 constitutes an eye opening degree change storage unit.

  The missing detection unit 13 reads the unit time eye opening degree history from the eye opening degree history storage unit 12, and detects the missing position where the eye opening degree cannot be calculated in the unit time eye opening degree history and the duration (missing time) of the missing position. The missing detection unit 13 adds missing information regarding the detected missing position to the unit time eye opening degree history and outputs the missing information to the missing interpolation unit 14. The missing detection unit 13 constitutes an eye opening degree detection missing detection unit.

  The missing interpolation unit 14 interpolates the opening degree of the missing position in the unit time opening degree history output from the missing detection unit 13. The missing interpolation unit 14 reads the eye opening degree history from the eye opening degree history storage unit 12 and interpolates the missing position by referring to the eye opening degree change with time when interpolating the missing position. After interpolating the missing position, the missing interpolation unit 14 outputs a unit time eye opening degree history to the blinking state detecting unit 15. The missing interpolation unit constitutes an interpolation unit.

  The blink state detection unit 15 detects the driver's blink state based on the unit time eye opening degree history output from the missing interpolation unit 14. The blink state detection unit 15 obtains the number of blinks per unit time of the driver based on the detected blink state. Based on the number of blinks, the driver's sleepiness level is estimated, and when the driver's sleepiness level is equal to or greater than a predetermined threshold, it is determined that the driver is sleeping. The blink state detection unit 15 transmits dozing information to the dozing prevention ECU 4 when it is determined that the driver is dozing.

  When the dozing prevention ECU 4 receives the dozing information transmitted by the blinking state detection unit 15, the dozing prevention ECU 4 controls the stimulus applying means to give the driver a stimulus and prevent the driver from falling asleep.

  Next, a processing procedure of the dozing prevention apparatus according to the present embodiment will be described. FIG. 2 is a flowchart showing a processing procedure of the dozing prevention apparatus according to the present embodiment.

  As shown in FIG. 2, in the dozing prevention apparatus according to the present embodiment, first, a face image of the driver is photographed by the face image capturing camera 2 (S1). Image data of the captured face image is transmitted to the face image processing ECU 3.

  Subsequently, the face image processing ECU 3 performs image processing such as template matching on the image data transmitted from the face image capturing camera 2, and recognizes the driver's face image (S2). Then, the face image processing ECU 3 detects the driver's eyelid position from the recognized face image (S3). The eyelid position information regarding the eyelid position detected by the face image processing ECU 3 is transmitted to the eye opening degree calculation unit 11 in the blink state detection ECU 1.

  Subsequently, the eye opening degree calculation unit 11 calculates the eye opening degree of the driver based on the transmitted eyelid position information (S4). Here, the degree of eye opening of the driver is the distance between the upper eyelid and the lower eyelid based on the eyelid position information with respect to the distance when the distance between the upper eyelid and the lower eyelid is the longest. Is calculated as

  After calculating the eye opening degree, the eye opening degree calculation unit 11 records the calculated eye opening degree in the eye opening degree history storage unit 12 (S5). Here, the eye opening degree history storage unit 12 stores a unit time eye opening degree history for 10 seconds and an eye opening degree history for 10 minutes. Then, the omission detection unit 13 determines whether or not the unit time eye opening degree history for 10 seconds is stored in the eye opening degree history storage unit 12 (S6). As a result, if it is determined that the unit time eye opening degree history for 10 seconds is not stored in the eye opening degree history storage unit 12, the unit time eye opening degree history for detecting the blinking state is not obtained. The process by the dozing prevention device is terminated.

  On the other hand, when it is determined that the unit time eye opening degree history for 10 seconds is stored in the eye opening degree history storage unit 12, the omission detection unit 13 reads the unit time eye opening degree history from the eye opening degree history storage unit 12. Then, it is determined whether the missing portion of the read unit time eye opening degree history is 50% or more (S6).

  As a result, when it is determined that the missing portion of the unit-time eye opening degree history read from the eye opening degree history storage unit 12 is 50% or more, it is determined that sufficient conditions for detecting the blink state cannot be obtained. Undetected processing is performed (S8), and the process proceeds to step S20. On the other hand, if it is determined that the missing portion of the unit time eye opening degree history read from the eye opening degree history storage unit 12 is not 50% or more, the missing part of the unit time eye opening degree history read from the eye opening degree history storage unit 12 is 10 It is determined whether or not it is greater than or equal to% (S9).

  As a result, when it is determined that the missing portion of the unit time eye opening degree history is not 10% or more, the missing time of the missing portion is considered to be short. In this case, the missing interpolation unit 14 performs linear interpolation before and after the missing part (S10), and proceeds to step S19. When performing linear interpolation, it is possible to connect the immediately before and immediately after the missing portion in a curved line based on the amount of change in the eye opening degree immediately before the missing portion and the amount of change in the eye opening degree immediately after the missing portion. Alternatively, the portion immediately before and after the missing portion can be connected in a straight line.

  On the other hand, if it is determined that the missing portion of the unit time eye opening degree history is 10% or more, the process proceeds to the flow shown in FIG. FIG. 3 is a flowchart showing a procedure following FIG. As shown in FIG. 3, as a result, after determining that the missing portion of the unit time eye opening degree history is 10% or more, the missing interpolation unit 14 calculates the parameters of the first half of the missing portion by calculation (S11). As the parameters of the first half of the missing portion extracted here, for example, those of 8 items shown in FIG. 4 can be cited. The items shown in FIG. 4 are “range”, “mode”, “average”, “median”, “increase speed”, “decrease speed”, “number of blinks”, and “closed eyes for a long time”. The parameters for all these 8 items are obtained.

  Subsequently, the missing second half parameter is calculated by calculation (S12). The parameters of the same item as the parameters of the first half of the missing part are also obtained for the parameters of the second half of the missing part. Here, the first half of the missing part is a part in which the degree of eye opening is calculated before the missing part in the unit time eye opening degree history, and the second half of the missing part is the eye opening after the missing part in the unit time eye opening degree history. This is the part where the degree was calculated.

  Then, a frame process is performed for each part of the first half of the past past data (S13). Thereafter, a similar waveform (hereinafter referred to as “missing first half similar waveform”) of the first half of the missing portion is extracted from the first half portion that has undergone frame processing (S14). A method such as pattern matching can be considered as a reference for extracting the missing first half similar waveform, but since a dozing prevention device is mounted on a vehicle, it is preferable to use a simple numerical calculation capable of high-speed processing. . In addition, since it is considered for the purpose of preventing dozing when the driver's drowsiness level increases while driving, the main purpose is to detect the similarity of the situation where the blinking state presents drowsiness.

  Therefore, in the present embodiment, the missing first half similar waveform is extracted based on the items calculated in step S11 and step S12. FIG. 5 shows a criterion for determining that the waveform is a missing first half similar waveform in each item. For example, regarding the “range” item, when the difference of the standard deviation 1σ of the difference between the maximum value and the minimum value is within 10%, it is determined that they are similar.

  In the frame processing for each piece of past data in the first half of the missing part in step S13, the similarity judgment of the item calculated in step S11 among these items is performed for the first half of the missing part. Then, as the extraction of the missing first half similar waveform in step S14, a highly similar portion is extracted from the eye opening degree history for 10 minutes.

  Then, in the same procedure as in steps S13 and S14, frame processing is performed for each missing second half part past data for the missing second half part (S15), and then the missing second half part for which frame processing has been performed is performed. A partial similar waveform (hereinafter referred to as “missing second half similar waveform”) is extracted (S16).

  Then, time-weighted synthesis is performed on the extracted first half-similar waveform and second half-similar waveform that have been extracted (S17). After performing the time weighting synthesis, the unit time eye opening degree history is recombined (S18), and the missing portion is interpolated.

  The steps from step S11 to step S18 will be described with reference to FIG. FIG. 6A is a diagram showing an example of the unit time eye opening degree history, and a missing portion can be seen at the center portion. In step S11, a missing first half waveform Wf included in the missing first half is calculated. In step S12, a missing second half waveform Wr is calculated.

  Subsequently, in step S13, frame processing is performed for each past data in the first half of the missing part. In step S14, the result of this frame processing is compared with the eye opening degree history, and the missing first half similar waveform is calculated from the eye opening degree history. Rf is extracted.

  As the eye opening degree history, the eye opening degree for the past 10 minutes is stored. The missing first half waveform Wf obtained in step S13 and the waveform for the duration of this waveform are extracted from the eye opening degree history and compared with the first half waveform Wf. Here, the items shown in FIG. 4 are compared, and the degree of similarity is determined. The criteria for determining whether or not they are similar are the criteria shown in FIG.

  Then, after determining whether or not each item serving as a criterion is similar, a missing first half similar waveform Rf similar to the waveform of the missing first half waveform Wf is extracted as shown in FIG. 6B. In extracting the missing first half similar waveform Rf, first, of the above eight items, a waveform corresponding to a prescribed number or more is extracted as a missing first half similar waveform candidate. The specified number here may be one or a plurality of two or more.

  If there is only one type of missing first half similar waveform candidate, that missing first half similar waveform candidate is determined as the missing first half similar waveform Rf. When there are two or more missing first half similar waveform candidates, the missing first half similar waveform candidate having the largest number of items satisfying the criterion among the eight items serving as the determination criteria is determined as the missing first half similar waveform Rf.

  Further, when the number of items satisfying the standard is the same, the sum of errors between the missing first half waveform Wf and the missing first half similar waveform candidate in each item is obtained, and the missing first half similar waveform candidate having a small sum of errors is missing. It is determined as the first half similar waveform Rf. If the sum of the errors matches, a waveform obtained by averaging the missing first half similar waveform candidates having the same error is determined as the missing first half similar waveform Rf.

  When the missing first half similar waveform Rf is determined, in step S15, frame processing is performed for each past data in the second half of the missing portion, and in step S16, the result of this frame processing is compared with the eye opening degree history for the past 10 minutes. The missing second half similar waveform Rr is extracted from the minute eye opening degree history. The extraction of the missing second half similar waveform Rr is performed by the same procedure as that for extracting the missing first half similar waveform Rf.

  When the missing first half similar waveform Rf and the missing second half similar waveform Rr are thus determined, a waveform corresponding to the same time as the missing portion (hereinafter referred to as “first half corresponding missing partial waveform” immediately after the missing first half similar waveform Rf from the eye opening degree history). And a waveform for the same time period as the missing portion (hereinafter referred to as “latter half-corresponding missing partial waveform”) immediately before the missing second half similar waveform Rr. Thereafter, in step S17, the first half correspondence missing partial waveform and the second half correspondence missing partial waveform are synthesized. This synthesis is time-weighted synthesis in which time weighting is performed.

  Specifically, weighting is performed between the first half corresponding missing partial waveform and the second half corresponding missing partial waveform based on the ratio between the occupation time of the missing first half similar waveform Rf and the occupation time of the missing second half similar waveform Rr. By this weighting, as shown in FIG. 6B, a corresponding partial missing waveform L1 that largely reflects the corresponding missing partial waveform corresponding to the missing partial similar waveform having a long occupation time is generated.

  Thereafter, in step S18, the unit time eye opening degree history is recombined. The re-synthesis of the unit time eye opening degree history is performed by pasting the corresponding partial missing waveform L1 between the missing first half waveform Wf and the missing second half waveform Wr. In this way, the missing portion is interpolated.

  Subsequently, the driver's sleepiness level is estimated based on the unit time eye opening degree history generated in step S10 and step S18 (S19). Then, when the sleepiness level is equal to or higher than the predetermined threshold value and it is determined that the driver is dozing, the dozing information is transmitted to the dozing prevention ECU 4 (S20). Then, the 10-second data based on the unit time eye opening degree history is released (S20), and the process ends. If it is determined that the driver is not dozing, the 10-second data based on the unit-time eye opening degree history is released without transmitting the dozing information (S20), and the process is terminated.

  Thereafter, when the dozing prevention ECU 4 receives dozing information, the dozing information is controlled to actuate the stimulus applying means to give the driver a stimulus and prevent the driver from falling asleep.

  As described above, in the snoozing prevention apparatus according to the present embodiment, when a missing part occurs in the unit time eye opening degree history used for the determination of dozing, the missing part is interpolated. In performing the interpolation of the missing portion, when the missing portion corresponds to 50% or more of the unit time eye opening degree history, the missing portion similar waveform in the eye opening degree history for 10 minutes is used. For this reason, even if a missing portion occurs in the unit time eye opening degree history, the blink state detection unit 15 can detect the blink state with high accuracy.

  Further, in obtaining the missing portion similar waveform, the missing first half similar waveform Rf similar to the missing first half waveform Wf and the missing second half similar waveform Rr similar to the missing second half waveform Wr before and after the missing portion are used. For this reason, it is possible to interpolate a missing portion similar waveform close to an actual missing portion waveform. At this time, weighting is performed between the first half corresponding missing partial waveform and the second half corresponding missing partial waveform based on the ratio of the occupation time of the first half missing partial similar waveform and the occupation time of the second half missing partial similar waveform. . For this reason, it is possible to interpolate a missing portion similar waveform with higher accuracy.

  Next, another example in the case of interpolating a missing part will be described. In the above embodiment, when the missing partial waveform is interpolated, the corresponding partial missing waveform L1 is obtained based on the missing first half waveform Wf and the missing second half waveform Wr. However, the corresponding part is obtained based on the missing first half waveform Wf or the missing second half waveform Wr. Missing waveforms can also be obtained.

  In this case, as shown in FIG. 7A, first, when a missing portion is generated in the unit time eye opening degree history, a missing first half waveform Wf for the missing portion is obtained. The method of obtaining the missing first half waveform Wf can be based on the determination criteria of the eight items shown in FIG. When the missing first half waveform Wf is obtained, as shown in FIG. 7B, the first half corresponding missing partial waveform Lf is obtained together with the missing first half similar waveform Rf from the eye opening degree history for 10 minutes. Then, as shown in FIG. 7C, the missing portion can be interpolated by synthesizing the missing portion waveform Lf corresponding to the first half with the missing portion of the unit time eye opening degree.

Further, in determining the similarity between the missing first half waveform Wf or the missing second half waveform Wr and the waveform extracted from the eye opening degree history, an index is provided for the unit time eye opening degree history and the eye opening degree history, and the missing is based on the transition of this index. It is also possible to obtain the similarity between the first half waveform Wf or the missing second half waveform Wr and the eye opening degree history. For example, as shown in FIG. 8, a plurality of predetermined indexes are set for the degree of eye opening.
Here, 60%. Although 70% and 80% are used as indexes, other indexes may be used. The degree of similarity between the missing first half waveform Wf or the missing second half waveform Wr and the waveform extracted from the eye opening degree history can be determined by the number of intersections when the eye opening degree history intersects with the index.

  Furthermore, instead of showing the eye opening degree as a waveform when storing the change in the eye opening degree over time, the eye opening state is set, and the degree of similarity between the eye opening degree history and the unit time eye opening degree history is set based on the degree of coincidence of the eye opening state. It can also be calculated. As the degree of eye opening, an eye opening state such as eye opening (opening degree 100%), middle (opening degree 50%), closed eye (opening degree 0%) is set, and a missing portion is interpolated from a change in the eye opening state. You can also. In this case, a similar pattern can be sufficiently obtained even when the eye opening degree is grasped not at the waveform but at the state level such as the above-described eye opening state. In this way, the calculation amount can be reduced by setting only the open, middle, and closed eyes as the open state.

  The degree of similarity with the unit-time eye-opening degree history in the eye-opening degree history when the eye-opening degree is expressed in the eye-opening state is a portion having the same length as the unit-time eye-opening degree history (10 seconds) in the eye-opening degree history for 10 minutes. It can be determined by the number of eye open states that match. Here, FIG. 9 shows an example of the temporal change of the eye opening degree set in the eye open state. In FIG. 9, a portion similar to the unit time eye opening degree history in the eye opening degree history is shown as a reference history, and the unit time eye opening degree history is shown as a history with omission. Further, as the eye open state, the open eye is indicated as “o”, the middle is indicated as “m”, the closed eye is indicated as “c”, and the defect that becomes the missing part is indicated as “l”.

  As shown in FIG. 9, in this example, 27 eye-opening states are represented in the reference history and the history with loss, but 25 eye-opening states out of the 27 eye-opening states coincide with each other. . Comparing the reference history with the deficient history, it can be seen that the reference history matches the deficient history except for the missing “l” portion in the deficient history. Here, the deficiency “l” in the deficit history can be estimated to be in the same open eye state as the corresponding part in the reference history. By applying the open eye state in the reference history to the missing part of the history with the defect, the missing part of the history with the defect can be interpolated.

  In this case, the open eye state can be expressed by 2 bits per frame including the missing information, and the matching calculation can be performed by a simple shift operation, so that the calculation load can be reduced. Furthermore, since there is no need to use a window, the possibility of finding a similar pattern can be increased.

  Further, instead of recording the eye opening degree in the fifo type, a unit time eye opening degree history pattern may be associated and stored as a feature pattern for each sleepiness degree. In this case, for example, a unit time eye opening degree history pattern corresponding to the sleepiness degree is stored. Specifically, the unit time eye opening degree history pattern when the sleepiness degree D = 0 is estimated is a001 to a100, the unit time eye opening degree history pattern is estimated when the sleepiness degree D = 1 is b001 to b100, and the sleepiness degree D is The unit time eye opening degree history pattern when estimated as = 2 is stored as c0011 to c100... Unit time eye opening degree history pattern when the sleepiness degree D = 5 is estimated as f001 to f100, respectively.

  Then, when the eye opening degree is lost, a matching unit time eye opening degree history pattern is searched based on the sleepiness degree determined immediately before, for example, 10 seconds before. Here, when a matching unit time eye opening degree history pattern is not found, the sleepiness degree is lowered by one step, and a unit time eye opening degree history pattern matching the sleepiness degree is searched again. It can also be set as the aspect which interpolates a missing part with a unit time eye opening degree history pattern.

  The preferred embodiment of the present invention has been described above, but the present invention is not limited to the above embodiment. For example, in the above-described embodiment, when obtaining a similar waveform before and after the loss, the parameters for all eight items such as “range” and “mode” are obtained. However, an arbitrary number of parameters is obtained from these parameters. It can also be. In the above embodiment, the eye opening degree history is continuously stored for 10 minutes. However, the eye opening degree history may be extracted and stored only in the blinking state. In this case, it is possible to reduce the storage amount without greatly reducing the accuracy of the blink state detection.

  Furthermore, in addition to the degree of eye opening, it is also possible to detect the blinking state from the eye opening degree and the face direction angle by detecting the face direction angle. According to this aspect, the blinking state can be detected with higher accuracy.

It is a block block diagram of the dozing prevention apparatus containing a blink state detection apparatus. It is a flowchart which shows the process sequence of a dozing prevention apparatus. It is a flowchart which shows the process sequence following FIG. It is a table | surface which shows the parameter used for judgment whether a waveform is similar. It is a table | surface which shows the judgment criteria of whether a waveform is similar. (A) is a graph showing an example of the unit time eye opening degree history, (b) is a graph showing a part similar to the unit time eye opening degree history in the eye opening degree history, and (c) is a unit time obtained by interpolating the missing part. It is a graph which shows an eye opening degree log | history. (A) is a graph showing another example of the unit time eye opening degree history, (b) is a graph showing a part similar to the unit time eye opening degree history in the eye opening degree history, and (c) is an interpolation of the missing part. It is a graph which shows a unit time eye-opening degree log | history. It is a graph which shows a time-dependent change of an eye opening degree. It is a figure which shows a reference log | history and a log | history with a missing part.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 ... Blink state detection ECU, 2 ... Face image pick-up camera, 3 ... Face image processing ECU, 4 ... Dozing prevention ECU, 11 ... Eye opening degree calculation part, 12 ... Eye opening degree log | history storage part, 13 ... Missing detection part, 14 ... Missing interpolation unit, 15 ... blink state detecting unit, L1 ... corresponding portion missing waveform, Rf ... missing first half similar waveform, Rr ... missing second half similar waveform, Wf ... missing first half waveform, Wr ... missing latter waveform.

Claims (2)

Eye opening degree detecting means for detecting the degree of eye opening of the driver from image data obtained by photographing the driver's face in the vehicle, and based on a change with time of the driver's eye opening degree, A blink state detection device for detecting a blink state,
Eye opening degree change storage means for storing changes over time of the driver's eye opening degree;
Eye-opening degree detection missing detection means for detecting a lack of eye opening degree detection of the driver by the eye-opening degree detection means;
When the lack of eye opening degree detection of the driver is detected by the eye opening degree detection means, the eye opening degree detection of the driver is performed based on the temporal change of the eye opening degree stored in the eye opening degree change storage means. Interpolating means for interpolating the change over time of the degree of eye opening of the time when the omission is detected;
A blinking state detection device comprising: The interpolating means is similar to the temporal change in the degree of eye opening before and after the time when the lack of eye opening degree detection of the driver is detected from the temporal change in the degree of eye opening stored in the eye opening degree change storage means. Extract the change over time,
The blink state detection device according to claim 1, wherein, based on the extracted temporal change of the eye opening degree, the blinking state detection apparatus interpolates the temporal change of the eye opening degree at a time when the lack of eye opening degree detection of the driver is detected.

Images