RU2413632C2 - Method to prevent driver falling asleep - Google Patents

Abstract

FIELD: transport.

SUBSTANCE: invention relates to control over driver state, namely, to prevention of driver falling asleep. Proposed method comprises generating face image and detecting areas of eyes. Besides, pattern of driver pupil is generated proceeding from general description for every human being, intermittent illumination of driver face by IF radiation, detecting image area including face image, defining limits of pupil motion, determining frequency and directions of eyes motion, determining blinking frequency, determining duration wherein driver eyes are closed, comparing parametres characterising driver state with reference states of falling asleep and sleeping. Using the results of comparison, decision is made to signal the driver falling asleep.

EFFECT: simplified method, simple implementation, and insensitivity to outer illumination.

1 dwg

Description

The invention relates to computer technology and can be used to prevent the driver of a vehicle from falling asleep by detecting signs of fatigue and closing his eyes in the image of his face.

One of the causes of traffic accidents is the drowsiness of the driver. External manifestations of falling asleep of the driver (both the process of incomplete and complete - sleep) are:

- reduction in the frequency of eye blinks;

- a decrease in the frequency of eye movements (change of direction of observation) and long-term fixation of the gaze in one direction;

- closing eyes for a long time.

A known method of preventing the driver from falling asleep and the automobile system “Attention Assist” that implements it (see. Attention Assist system http://mb-market.ru/content/view/437/129) [1] integrated into Mercedes-Benz cars ", Designed to prevent a driver falling asleep on the basis of recognition of signs of fatigue in the movements and reactions of the driver. If signs of fatigue are detected, the system signals a need for rest. The system constantly monitors the driver’s movements and, based on the data received, models the behavior typical of him personally. Next, the system creates a profile that takes into account the time of day, the duration of the trip without stops and the reaction of the driver. Having processed the received data, the system notifies the driver about the need to take a break. The system registers and analyzes the following data: wheel rotation angle, speed, acceleration, use of indicators and depressing the pedals, as well as external factors, such as wind or uneven road surface. If the system detects a change in driver behavior, it gives a signal notifying the driver about the need to rest. In addition to natural fatigue from lack of sleep, the device identifies the monotony of unchanging road conditions during a long trip and the absence of other vehicles on the road, especially in the dark, because it is because of these factors that the driver gets tired and loses vigilance.

The main disadvantages of this system is that its operation is based on the recognition of signs of fatigue of a particular driver, as well as the complexity of its training and adaptation for each driver. In addition, this system is not suitable for use in vehicles used by more than one driver. The system cannot also be used in cars of other brands, as receives information about the driver’s condition from auxiliary sensors (steering wheel position, gas pedal, etc.), which may be incompatible with other car brands. Another disadvantage of the system is its high complexity, cost, limiting its use in domestic cars, and high structural and functional redundancy.

A known method for the automatic detection of closed eyes in digital images of arbitrary nature with a complex background (see application for invention of the Russian Federation 2007107099) [2], which consists in the following operations: detect faces / faces in a digital image; for each detected face, localization of the eyes is performed; carry out the selection of features of closed eyes, combining a set of related pixels characteristic of a given defect; analyze two types of characteristic features, namely: horizontal and vertical features; highlighted characteristic features are represented in the form of curves approximated by parabolas based on the least squares method; make a conclusion about the presence or absence of closed eyes in the digital image based on data on the directions of the protuberances of the parabolas, approximating these features.

The disadvantage of this method is the significant computational complexity that impedes its implementation in the on-board computer computing facilities of a vehicle that do not have sufficient processing power; an increase in the time to receive a signal about falling asleep due to the relatively long process of eye detection.

A known method of recognition by the iris of the eye (see RF patent 2331107) [3], comprising the steps of acquiring an image of at least one human eye in the infrared spectrum and processing the obtained image to determine identification signs based on the iris limited by the outer boundary on the eye, the method comprises the following steps: obtaining an image of the eye in the visible spectrum almost simultaneously with obtaining its image in the infrared spectrum; determination of the outer border of the iris based on the image made in the visible spectrum, before processing the image obtained in the infrared spectrum.

The disadvantage of this method lies in the complexity of implementation (due to the resource consumption of the method) in the built-in computing means of the operation of the analysis of the iris. Other disadvantages are the inability to track eye movements and the inability to detect closed eyes in the considered manner, which is fundamentally necessary for a traffic accident prevention system.

There is a method of determining the position of the eye and a device for its implementation (see US patent No. 7,130,453) [4], which consists in obtaining images of the face in the near infrared light, preprocessing the image and increasing the contrast of the sclera and iris based on the construction of a histogram of brightness and the subsequent determination of the estimated ratios for correcting brightness by a histogram, determining the correspondence of a gradient image and an eye pattern.

The disadvantage of this method is the ability to detect only open eyes and the inability to analyze eye movements in dynamics.

A known method for the safe determination of eyes (see US patent No. 7,430,365) [5], including estimating the distance to the eye, iterative change in radiation power depending on the eye, eye detection.

The disadvantage of this method is the need for multiple estimates of the distance to the eye and eye detection, which increases the total execution time of the method and does not provide timely eye detection.

A known method for detecting the eyes of humans and animals (see RF patent No. 2223516) [6]. The method irradiates the located volume of space with pulsed scanning radiation in the range of 450-760 nm of the electromagnetic spectrum, receives reflected scanning radiation, at least two measurements of the radiation energy reflected from the located volume of space, each in the wavelength ranges of 450-500 nm and 500 -570 nm - q _b1 , q _g1 and q _b2 , q _g1, respectively, and the determination of the presence of people and animals in the located volume of space is carried out by the presence of rhodopsin by comparing the ratios k ₁ = q _b1 / q _g1 and k ₂ = q _b2 / q _g2 among themselves - at k ₂ ≠ k ₁ rhodopsin is present, at k ₂ = k _{1 there is} no rhodopsin. The localized volume of space is irradiated with scanning radiation of the visible electromagnetic wavelength range should be carried out at the minimum possible distance from the optical axis of the reception of reflected radiation, providing the possibility of maximum amplification of the red eye effect. The localized volume of space can be irradiated with pulsed scanning radiation using a color digital camera, while photographing with a color digital camera during the flash exposure takes at least two photographs in the range 450-570 nm and 520-760 nm.

The disadvantages of the method are:

- obtaining at least two frames during the duration of the flash, which is not always possible due to the insufficiently high speed of the used optical-electronic sensor (for example, for large images) and the short duration of the flash;

- the location of the radiation source at a close distance from the optical axis of the reception of the reflected radiation is not possible for the system of preventing falling asleep of the driver;

- Using the flash is completely unacceptable due to the visual disturbance to the driver.

Closest to the proposed one is a method of detecting closed eyes in a system for preventing falling asleep of a driver (see US Pat. No. 6,130,617) [7], which consists in obtaining a face image from a CCD camera, highlighting pairs of areas supposedly containing eye images, detecting areas of the eyes and nose according to the smallest isosceles triangle compiled by them.

The disadvantage of this method is the lack of analysis of eye movements, the lack of measurement of parameters of movements and blinks.

The task to be solved by the claimed invention is to develop a method for preventing falling asleep of a vehicle driver, different from the known ones:

- the possibility of use in a vehicle of any type,

- low computational complexity sufficient for implementation on low-budget computing modules,

- simplicity of practical implementation,

- the ability to analyze both signs of the process of falling asleep and fatigue of the driver, and the fact of falling asleep,

- invariance to changes in external lighting.

The technical result is achieved due to the fact that in the known method [7] for detecting closed eyes in the system for preventing falling asleep of a driver, including obtaining a face image, detecting areas presumably containing eyes, detecting eye areas, operations have been introduced to form a pupil pattern of the current driver based on a common for any person descriptions, periodic illumination of the driver’s face with infrared light, detection of the image area containing the face, determination of the boundaries of the area of movement of the pupil, determination of the frequency and directions of eye movement, determination of the frequency of blinks, determination of the length of the period of time during which the eyes are closed, comparison of parameters characterizing the state of the driver with the reference ones for the state of falling asleep and the state of sleep, deciding on the need for signaling about falling asleep of the driver.

The invention can be used to improve the safety of driving a vehicle by timely preventing the driver from falling asleep and meets the criterion of "industrial applicability".

The invention is illustrated in the drawing, which shows the algorithm for implementing the method.

The sequence of operations in the method of preventing falling asleep of the driver is as follows:

- getting a face image,

- the formation of the standard pupil of the current driver based on a description common to any person,

- detection of the image area containing the face,

- detection of areas allegedly containing eyes,

- detection of eye areas and determination of the boundaries of the area of movement of the pupil,

- determination of the current coordinates of the pupils,

- determination of the frequency, amplitude and directions of eye movement,

- determination of the frequency and duration of blinks,

- determination of the duration of the period of time during which the eyes are closed,

- a comparison of the parameters characterizing the condition of the driver, with the reference for the state of falling asleep and the state of sleep,

- decision making on the need for signaling that the driver is falling asleep

Consider a way to prevent the driver from falling asleep.

The method consists in automatically performing the following groups of operations (see the algorithm in figure 1):

- setting the parameters of the method for detecting the eyes of the current driver (one-time recognition of the eyes of the current driver is carried out on the basis of a general description of the characteristics of the eye for the subsequent use of the eye standard and the pupil of the current driver; this significantly reduces the computational complexity of the subsequent repeatedly performed eye detection procedure used in the analysis of the trajectory eye movements), blocks 1-3 of the algorithm in figure 1,

- analysis of the current state of the eyes (open, closed, position of the pupils), blocks 4-11 of the algorithm,

- assessing the condition of the driver and making a decision about the need for signaling about falling asleep of the driver, blocks 12-20 of the algorithm.

Obtaining a face image is carried out by using a small-sized optical-electronic sensor fixed in front of the driver's face. Optoelectronic sensor (OED) is mounted on the dashboard of a car or above the windshield. To illuminate and obtain a contrasting image of the face, regardless of the external illumination, a small-sized infrared illuminator is used, located similarly to the OED opposite the face.

On the basis of a priori information about the image of the pupil and eye (the pupil is almost round, two areas are distinguished on its image - the iris and the pupil itself, the pupil has a color different from the skin and the eyeball, the borders of the lower and upper eyelids are located above and below the eye, the corners of the eyes have an acute angle) eye recognition is performed (block 2 of the algorithm, Fig. 1). The recognition method used to form the standard allows one to reliably distinguish eyes in the image (see. Principles of functioning of a binocular optoelectronic device for diagnosing deviations of the oculomotor apparatus / V.N. Gridin, BC Titov, M. I. Trufanov // Izvestiya Vuzov. Instrument-making, - 2008, - No. 2, - S.48-53 [8]; A method for detecting eyes in an image and recognizing their state / BCTitov, M.I. Trufanov // Bulletin of TulSU. Series Radio engineering and radio electronics. T.IX. - Tula: Publishing house of TulSU, 2007, - S.133-138 [9]).

As a result, the eye pattern of the current driver is formed, which is a set of characteristics (block 3, Fig. 1):

- the length and width of the rectangles described around each eye;

- the diameter of the pupils and iris,

- the ratio of the color of the pupil to the color of the iris (the color in all operations of the method is presented in shades of gray),

- distance between the eyes.

Further, according to the proposed method, a group of operations is performed that provides an analysis of the current state of the eyes (open / closed) and determination of the coordinates of the pupils for open eyes.

For this, face images are obtained with the infrared (IR) illuminator turned off and then on. Each frame of the image is divided into small areas. For each region, the average value of the brightness of the pixels that make it up is obtained. Analysis of the average brightness of the frame areas with the IR source turned off and on allows you to detect a face based on the following features. When IR radiation is on, the skin of the face has an almost white color, while the background located farther from the OED than the face is characterized by a significantly lower brightness. In addition, an additional feature is used to detect faces: when the infrared radiation is turned off and on, the brightness of the part of the frame corresponding to the skin of the face will differ significantly, and the background brightness behind the face will only slightly change. Thus, the combination of these features allows you to detect the driver’s face with a procedure that is simple from a computational point of view. The final operation of the second group of operations is the detection of eyes by the presence of pupils in the image based on the use of the following features and measurement of the two-dimensional coordinates of the pupils:

- under IR illumination, the pupil remains almost black in the image, and the rest of the face is light as well as a round pupil;

- known (previously defined in blocks 1-3 of the algorithm, figure 1) the diameter of the pupils,

- the distance between the eyes (also defined earlier in blocks 1-3 of the algorithm).

To detect the eyes, determine their condition and measure the coordinates of the pupils in the case of open eyes, produce:

1) obtaining a frame I ₁ image when the infrared radiation is off;

2) an estimate of the average brightness in the regions P _{1ij of the} frame, which are rectangles with side sizes n times smaller than the corresponding frame sizes (length and width).

3) irradiation of the object with infrared radiation and obtaining a frame I ₂ image when the infrared radiation;

4) an estimate of the average brightness in the above areas P _2ij frame when the source of infrared radiation;

5) the identification of the fact that there is a change in the brightness of the areas of the frame, such that: a) the brightness of the areas on the left and right sides of the frame have changed slightly; b) the brightness of the central areas have changed significantly, and these areas are a connected area;

6) a decision is made when condition 5 is met about the presence of a person in the frame or about the absence thereof if condition 5 is not met;

7) frame analysis I ₂ . Formation of a set of image areas with brightness close to the minimum for a given image frame, and determination of the degree of roundness (see Digital image processing methods: a training manual Part 1. Grif UMO / Degtyarev SV, Sadykov SS, Tevs SS, Shirabakina T.A. - Kursk: Kursk, state technical university, 2001, - 167 p.) [10] for each of them;

8) the search for at least two circular regions from the set formed in claim 7, having similar diameters and areas;

9) determination of contrast in the corresponding region of P _2i containing the circular region of claim 8;

10) the decision to find the pupils (and eyes) in the presence of at least two areas according to claim 8, having a contrast measured in clause 9, exceeding a predetermined threshold, and located at a distance of not less than m * d (where m - constant, d is the diameter of the regions defined in clause 8).

Consider the operations listed above in more detail.

Obtaining an image frame consists in reading pixel brightness values from a matrix receiver of an optoelectronic sensor and recording the brightness values in matrix I, the dimension of which is determined by the image frame size (see Methods of computer image processing / Edited by V.A.Soyfer. - M. : Fizmatlit, - 2001, - 784 p.) [11]. Two image frames are formed - when the IR radiation is off I ₁ and when I _{2 is on} .

Next, each image frame is divided into n ² areas (P _1ij - areas of the first frame, P _2ij - areas of the second frame, indices i, j determine the position of the area). If the original image is color, then it is converted to monochromatic or further image analysis is performed only on the red channel. For each region, the average value of the brightness of the pixels that make it up is obtained.

Analysis of the average brightness of the regions P _1ij and P _2ij frames with the IR source turned off and on allows you to detect a face based on the features discussed above.

The final operation is the detection of pupils in the image. It is known (see [8]) that under IR illumination the pupil remains almost black in the image, and the rest of the face is light. Moreover, for an open eye, the pupil will have a shape close to a circle (eyelashes of the upper eyelid can slightly distort the shape). Only dark nostrils may appear in the image of the face with other dark areas. For the selection of pupils from the nostrils, a sign is used that is associated with the distance between them, many times greater than the distance between the nostrils. Other dark spots on the face are unlikely. These signs, with sufficient reliability for practical use, make it possible to detect pupils and determine their coordinates, to reveal the fact of the presence of eyes and their condition (open, closed).

The obtained current data on the state of the eyes are stored for a given period of time and are used to assess the condition of the driver and make a decision about the need for signaling that the driver is falling asleep (blocks 12-20 of the algorithm, FIG. 1).

Consider the final third group of operations of the proposed method.

In block 12 of the algorithm (Fig. 1), the number n of eye blinks is calculated (as the transition from the open state to the closed state and back to the open state in a short time interval) and their average duration t is determined.

In block 13 of the algorithm determines the average amplitude α and the frequency of rapid eye movements when changing directions of observation when driving.

In block 14, the time interval T is determined during which the eyes are in a closed state (if their current state is “closed”).

The decision on the fatigue of the driver and the possible start of falling asleep is carried out:

- to reduce the number of blinks n and increase their average duration t. Verification of these conditions is performed by comparing the values of n, t with a priori given values of n _e , t _e in block 15;

- to reduce the average amplitude A of the movements of the pupils when changing directions of observation and reducing the frequency f of changing directions (eye movements). Checking these conditions is performed in block 17 by comparing the values of A, f with a priori specified values of _AE , _fE .

When the conditions of the blocks of algorithm 15 and 17 are met, a warning sound signal is generated when the driver begins to fall asleep.

Falling asleep as a final state is determined in block 19 by detecting a prolonged closure of both eyes (closing the eyes by an amount greater than the reference T _E ). In this case, an acoustic signal is generated about the danger, which does not stop until the eyes open.

In the proposed method, an IR LED can be used as a source of infrared radiation. As an OED - a small-sized video camera. The operations of blocks 4-21 of the algorithm are characterized by low computational complexity and can be implemented on low-budget programmable logic integrated circuits and a microcontroller or microcomputer. The operation of block 2 (initial eye recognition) is computationally complex, however, unlike operations 4-21, it can be performed in real time, which allows it to be implemented on the low-cost computing tools mentioned above.

The invention improves road safety through the timely detection of falling asleep and is characterized by the possibility of application in a vehicle of any type, low computational complexity, sufficient for implementation on low-budget computing facilities, ease of practical implementation, the ability to analyze both signs of the process of falling asleep and driver fatigue, and its fact falling asleep.

Claims (1)

A method for preventing falling asleep of a vehicle driver, including obtaining a face image, detecting areas presumably containing eyes, detecting eye areas, further comprising generating a pupil pattern of the current driver based on a description common to any person, periodic illumination of the driver’s face with infrared light, detection areas of the image containing the face, determining the boundaries of the area of movement of the pupil, determining the frequency and directions of eye movement, op edelenie blink frequency, determining the length of time during which the eyes are closed, a comparison of the parameters characterizing the state of the driver with reference to the sleep state and the sleep state, a decision about the need to signal the driver falling asleep.

Images