KR102161908B1 - Driver status monitor method and apparatus - Google Patents

Abstract

The present invention discloses a driver condition monitoring method and apparatus. According to the present invention, a driver condition monitoring device includes a processor and a memory connected to the processor, wherein the memory acquires an image of the driver through a camera installed in the vehicle, and has a face having a preset size and ratio. A face candidate region is determined from the acquired image using a feature extraction filter, the driver's eye region is searched in consideration of whether glasses are worn in the determined face candidate region, and the driver is drowsy using the searched eye region. A driver condition monitoring device is provided that stores program instructions executable by the processor to determine whether it is in a state or a negligent forward-looking state.

Description

Driver status monitor method and apparatus TECHNICAL FIELD

The present invention relates to a method and apparatus for monitoring driver conditions.

Traffic accidents frequently occur due to the driver's drowsiness, carelessness or poor visibility, and violation of the obligatory forward look of the vehicle behind the vehicle.

To prevent this, vehicles are recently equipped with Advanced Driver Assistance Systems (ADAS).

The Intelligent Driver Assistance System uses advanced detection sensors, GPS, communication, and intelligent video equipment to recognize some situations while driving by itself to determine the situation, control the car or detect dangers in advance by sound and light. It is a driver assistance system that informs you by vibration, etc.

Among them, the Driver Status Monitoring (DSM) system determines whether the driver is drowsy or is in a state of neglect of looking forward by processing the driver's image by a camera inside the vehicle.

The driver condition monitoring system must quickly determine a case of drowsiness or negligent looking forward and provide an alarm to the driver. However, the conventional system has a problem in that it cannot provide a fast alarm due to a slow recognition speed.

Registered Patent 10-1551262

In order to solve the above problems of the prior art, a method and apparatus for monitoring a driver's condition capable of increasing the recognition speed of a driver's condition is proposed.

In order to achieve the above object, according to an embodiment of the present invention, a driver condition monitoring apparatus, comprising: a processor; And a memory connected to the processor, wherein the memory acquires an image of a driver through a camera installed in the vehicle, and uses a face feature extraction filter having a preset size and ratio to obtain a face from the acquired image. To determine a candidate region, to search for an eye region of the driver in consideration of whether or not to wear glasses in the determined face candidate region, and to determine whether the driver is in a drowsy state or a negligent forward gaze state by using the searched eye region, A driver condition monitoring device is provided that stores program instructions executable by the processor.

The program instructions randomly generate a plurality of facial feature extraction filters having different sizes and different horizontal and vertical ratios at an initial point of view, and apply the randomly generated plurality of facial feature extraction filters to the obtained image. By applying the facial feature extraction filter having the highest face search success rate, a facial feature extraction filter may be determined, and a face candidate region in the obtained image may be determined using the determined facial feature extraction filter.

The program instructions set an eye area search area and a bridge search area in the determined face candidate area, determine whether a bridge exists in the bridge search area, and if the bridge exists, based on the searched bridge. By searching for an eyeglass frame, comparing a first value that considers the brightness and position in a preset number of frames of the eye region selected based on the position of the eyeglass frame and a second value that considers the brightness and position of the eye region in the current frame You can explore the eye area.

The first calculated value is an average brightness value, a brightness variance value, a position average value, and a variance value of the eye region of each of the preset frames, and the second calculated value is a luminance value and a luminance variance value of the eye region in the current frame. , Position values, and variance values thereof.

The program instructions search for an eyelid from the eye area, determine whether the driver's eyes are closed through the state of the searched eyelid, and determine whether the driver's eyes are closed for a preset time, and determine as a drowsy state and an alarm. Can be created.

The program instructions, search the nose from the eye area, calculate the rotation angle of the face through the difference between the position of the glabellar and the horizontal axis of the nose, and the state where the rotation angle is greater than a preset threshold continues for a preset time. In this case, an alarm can be generated by determining that the forward gaze is negligent.

According to another aspect of the present invention, there is provided a driver condition monitoring method, comprising: determining a facial feature extraction filter having a size and ratio suitable for a driver's face; Obtaining an image of a driver through a camera installed in the vehicle; Determining a face candidate region from the obtained image using the determined facial feature extraction filter; Searching for an eye area of a driver in consideration of whether glasses are worn in the determined facial candidate area; And determining whether the driver is in a drowsy state or a negligent forward-looking state by using the searched eye area.

According to the present invention, since a facial feature extraction filter suitable for the driver's face is used in advance, the computation speed can be further improved.

In addition, according to the present invention, since the eye area is detected by determining whether glasses are worn, there is an advantage in that the driver's state can be accurately monitored.

1 is a view showing the configuration of a driver condition monitoring apparatus according to an embodiment of the present invention.
2 is a diagram illustrating facial feature extraction filters having different sizes and ratios according to the present embodiment.
3 is a flowchart illustrating a process of generating a filter for extracting facial features according to the present embodiment.
4 is a flowchart of an eye search process in consideration of whether or not to wear glasses according to an embodiment of the present invention.
5 is a view for explaining whether glasses are worn and an eye search process according to the present embodiment.
6 is a diagram for explaining an eyelid search process according to the present embodiment.
7 is a diagram for explaining a process of detecting a negligent forward gaze state according to the present embodiment.

In the present invention, various modifications may be made and various embodiments may be provided, and specific embodiments will be illustrated in the drawings and described in detail.

However, this is not intended to limit the present invention to a specific embodiment, it is to be understood to include all changes, equivalents, and substitutes included in the spirit and scope of the present invention.

The present invention acquires an image of a driver through a camera installed inside a vehicle, quickly searches for a face and an object (part) within the face from the acquired image to monitor the driver's state, and provides an alarm if necessary.

1 is a view showing the configuration of a driver condition monitoring apparatus according to an embodiment of the present invention.

As shown in FIG. 1, the driver condition monitoring apparatus according to the present embodiment may include a processor 100, a memory 102, a camera 104, and an alarm unit 106.

The processor 100 may include a central processing unit (CPU) capable of executing a computer program or a virtual machine.

The memory 102 may include a nonvolatile storage device such as a fixed hard drive or a removable storage device. The removable storage device may include a compact flash unit, a USB memory stick, or the like. Memory 102 may also include volatile memories such as various random access memories.

Program instructions executable by the processor 100 are stored in the memory 102.

Program commands according to the present embodiment detect a face from an image input through the camera 104 installed in the vehicle, and determine whether the driver is in a drowsy state or a negligent state of looking forward by detecting eye, nose, and face rotation angles. do.

In addition, when the drowsy state or the negligent looking forward state continues for a preset time, an alarm is output through the alarm unit 106.

Here, the camera may include both an optical camera or an infrared camera.

Since the driver's face varies depending on the installation location of the camera, the type of vehicle, and the driving posture, it may take a long time to detect a feature corresponding to the driver's face.

To this end, according to a preferred embodiment of the present invention, facial feature extraction filters of various sizes and ratios for detecting facial features are used.

2 is a diagram illustrating facial feature extraction filters having different sizes and ratios according to the present embodiment.

The driver condition monitoring device searches for a driver's face using a face feature extraction filter having various sizes and ratios of width and height as shown in FIG. 2 at the beginning of the driver condition monitoring execution, and determines a face search success rate for each size and ratio. do.

The driver condition monitoring apparatus determines a face feature extraction filter having a high face search success rate by performing a scan through a face feature extraction filter having different sizes and ratios in one frame at an initial time point of execution.

In this case, the driver condition monitoring apparatus determines whether or not a feature corresponding to a face is detected in each filter to determine a face search success rate, and determines a face feature extraction filter close to the driver's face size and ratio.

According to the present embodiment, since a facial feature extraction filter having a ratio suitable for the driver is determined in advance and the driver's face is searched, the amount of computation can be significantly reduced.

The driver condition monitoring apparatus searches for a face from images continuously input through the camera 104 using the facial feature extraction filter determined as described above.

The Haar Adaboost algorithm may be applied to the facial feature search according to the present embodiment, and the facial feature extraction filter may be a Haar filter.

Hereinafter, the filter according to the present embodiment will be described in detail.

Preferably, the camera 104 may be an infrared camera, and the driver condition monitoring device generates an integral image from the black-and-white image input through the infrared camera, and faces through a strong classifier composed of a weak classifier using a Haar filter based on the difference in shadow. Search for location candidates. This can be defined as a facial feature search process using a cascade classifier.

Haar filters are created through reinforcement learning.

3 is a flowchart illustrating a process of generating a filter for extracting facial features according to the present embodiment.

In more detail, the driver condition monitoring apparatus randomly generates a plurality of Haar filter masks including information on the location (x, y), width (w), height (h), and features to be used (step 300). Here, the location, width, and height may be defined as a size and a ratio.

The driver condition monitoring apparatus stores a filtering result value by applying each Haar filter mask generated in step 300 to the positive sample and the negative sample (step 302).

Thereafter, a Haar filter mask is additionally generated by changing the position, width, and height of each filter mask (step 304), and a filtering result value for the additionally generated Haar filter mask is additionally calculated (step 306).

Next, the Haar filter mask generated in step 300 and the Haar filter mask additionally generated in step 304 are evaluated through an evaluation function (step 308).

The evaluation function is expressed as the following equation.

는 포지티브 샘플에 대한 필터링 결과값의 분산이고,

는 네거티브 샘플에 대한 필터링 결과값의 분산이며, k는 현재 필터 마스크의 인덱스이다. here,

Is the variance of the filtering result for the positive sample,

Is the variance of the filtering result for negative samples, and k is the index of the current filter mask.

The driver condition monitoring device replaces the filter mask with the filter mask determined to be excellent by the evaluation function based on the value of the Score function (step 310).

Thereafter, step 308 is repeatedly performed with the replaced filter mask as an initial value, and the repetition is stopped when a certain number of times is satisfied or all filter masks are not changed.

The above-described facial feature extraction filter having a predetermined size and ratio is a set of Haar filters determined through the process of FIG. 3.

When a face candidate region is determined through a facial feature extraction filter having a preset size and ratio, the driver condition monitoring apparatus searches for eyes by setting an eye search region in consideration of the body ratio.

When the driver wears glasses or the like in the search of the eye area, an error may occur in face recognition due to reflection of light.

According to the present embodiment, it is first determined whether the driver is wearing glasses so as to increase the reliability of driver condition monitoring.

4 is a flowchart of an eye search process in consideration of whether or not to wear glasses according to an embodiment of the present invention.

Referring to FIG. 4, an eye search area is set in a face candidate area (step 400).

5 is a view for explaining whether glasses are worn and an eye search process according to the present embodiment.

As shown in FIG. 5, an area having a predetermined size in the face candidate area may be set as the eye search area 500.

In order to determine whether to wear glasses, the driver condition monitoring device sets a bridge search area 502 in the eye search area 500 (step 402).

The bridge search area 502 may be set to a predetermined size in the center of the eye search area 500.

The driver condition monitoring apparatus determines whether or not a bridge exists through a change in brightness of the vertical/horizontal boundary in the bridge search area 502 (step 404).

When the bridge 504 is searched, the spectacle frame is searched based on the searched bridge 504 (step 406).

When the spectacle frame 506 is searched, the driver condition monitoring apparatus includes an average brightness value, a luminance dispersion value, and a position average value (rectangle) in the last predetermined number of frames of the eye region 508 selected based on the position of the spectacle frame 506 A center point) and a first value including a variance value of each of the above items is calculated (step 408). Here, step 408 is a process of calculating the cumulative value of the brightness and eye position of each eye area of a preset number of frames adjacent to the current frame.

Thereafter, the accumulated value is compared with the second values calculated for the eye area of the current frame to calculate the reliability (step 410).

Here, the second value may be a brightness value, a brightness dispersion value, a position value (a center point of the eye region), and a dispersion value of the eye region of the current frame.

The equation for calculating the reliability is as follows. .

Here, zscore means a z score value, and w is a weight controlling each zscore. When the r value for each eye is calculated, the eye area is determined through the ratio of the r values for both eyes. .

If the bridge 504 is not searched, the eye region is searched using an existing algorithm.

After the eye area is searched, the driver condition monitoring device searches the eyelid from the eye area.

6 is a diagram for explaining an eyelid search process according to the present embodiment.

Referring to FIG. 6, an image is vertically moved downward and upward to search for an eyelid, and a vertical boundary component is searched.

Through the above-described search process, the difference between the upper and lower eyelids is calculated, and through this, it is determined whether the eyes are closed.

The driver condition monitoring device outputs an alarm through the alarm unit 106 when the closed condition continues for a preset time.

In addition, the driver condition monitoring apparatus according to the present exemplary embodiment determines whether the driver is in a negligent forward-looking state by calculating nose and face angles.

7 is a diagram for explaining a process of detecting a negligent forward gaze state according to the present embodiment.

Referring to FIG. 7, a lower portion of the face that does not overlap with the eye region 700 calculated as described above is set as the nose search region 702.

A classifier using a Haar filter based on a horizontal boundary and a vertical boundary can be used for nose search.

The driver condition monitoring device calculates the difference between the position of the eyebrows 704 and the horizontal axis of the nose, and calculates the rotation angle of the face assuming that the position of the nose and the center of the face 706 are at a certain distance.

When the rotation angle of the face is greater than the threshold for a preset time, the driver condition monitoring device determines that the driver is in a negligent forward-looking state and generates an alarm.

The above-described embodiments of the present invention have been disclosed for the purpose of illustration, and those skilled in the art who have ordinary knowledge of the present invention will be able to make various modifications, changes, and additions within the spirit and scope of the present invention, and such modifications, changes and additions It should be seen as belonging to the following claims.

Claims (7)

As a driver condition monitoring device,
Processor; And
Including a memory connected to the processor,
The memory,
Acquires an image of the driver through a camera installed in the vehicle,
Determine a face candidate region from the obtained image using a facial feature extraction filter having a preset size and ratio,
In the determined face candidate region, the driver's eye region is searched in consideration of whether glasses are worn,
To determine whether the driver is drowsy or neglected to look forward using the searched eye area,
Store program instructions executable by the processor,
The program instructions,
Randomly generate a plurality of facial feature extraction filters having different sizes and different horizontal and vertical ratios at the initial viewpoint,
A facial feature extraction filter having the highest face search success rate is determined by applying the plurality of randomly generated facial feature extraction filters to the acquired image,
A driver condition monitoring device that determines a face candidate region from the obtained image using the determined facial feature extraction filter. delete The method of claim 1,
The program instructions,
Set an eye area search area and a bridge search area in the determined face candidate area,
It is determined whether or not a bridge exists in the bridge search area,
If the bridge is present, the eyeglass frame is searched based on the searched bridge,
The eye region is determined by comparing a first calculated value that considers brightness and position in a preset number of frames of the eye region selected based on the position of the spectacle frame and a second calculated value that considers the brightness and position of the eye region in the current frame. Driver condition monitoring device to navigate. The method of claim 3,
The first calculated value is an average brightness value, a brightness dispersion value, a position average value, and a dispersion value thereof of the eye region of each of the preset frames,
The second calculated value is a brightness value, a brightness dispersion value, a position value, and a dispersion value of the eye area in the current frame. The method of claim 1,
The program instructions,
Search for the eyelid from the eye area,
Determines whether the driver's eyes are closed based on the detected eyelid condition,
A driver condition monitoring device that determines that the eyes are closed and generates an alarm if the condition continues for a preset period of time. The method of claim 1,
The program instructions,
Search the nose from the eye area,
The rotation angle of the face is calculated through the difference between the position of the eyebrows and the horizontal axis of the nose,
A driver condition monitoring device configured to generate an alarm by determining that the rotation angle is greater than the preset threshold and continues for a preset period of time. As a driver condition monitoring method,
(a) determining a facial feature extraction filter having a size and ratio suitable for the driver's face;
(b) obtaining an image of the driver through a camera installed in the vehicle;
(c) determining a face candidate region from the obtained image using the determined facial feature extraction filter;
(d) searching for a driver's eye area in consideration of whether glasses are worn in the determined face candidate area; And
(e) determining whether the driver is in a drowsy state or a negligent forward-looking state by using the searched eye area,
In the step (a), a plurality of facial feature extraction filters having different sizes and different horizontal and vertical ratios at an initial viewpoint are randomly generated,
A facial feature extraction filter having the highest face search success rate is determined by applying the plurality of randomly generated facial feature extraction filters to the acquired image,
In the step (b), a driver condition monitoring method of determining a face candidate region in the obtained image using the determined facial feature extraction filter.

Images