KR20200084860A - System for monitoring the driver status and method thereof - Google Patents

Abstract

Disclosed are a driver status monitoring system and an operating method thereof. According to one aspect of the present invention, the driver status monitoring system includes: a near infrared (NIR) control part turning on NIR lighting towards a driver; an image obtaining part alternately obtaining a first face image frame where the face of the driver is photographed while the NIR lighting is on and a second face image frame where the face of the driver is photographed while the NIR lighting is off, or selectively obtaining only the first face image frame; an image processing part creating a difference in the image frame by calculating an image difference between the first face image frame and the second face image frame; and a driver status monitoring part confirming whether the driver properly keeps his or her eyes forward by using the difference in the image frame or the first face image frame. Therefore, the driver status monitoring system is capable of reducing light reflection caused by an accessory such as glasses or the like.

Description

System for monitoring the driver status and method thereof

The present invention relates to a driver condition monitoring system, and more particularly, to a driver image acquisition technology for a driver condition monitoring system.

In modern society, the number of deaths due to various traffic accidents is increasing along with the explosion of vehicles. In order to reduce the fatalities of traffic accidents as much as possible, technologies for preventing accidents have been developed in order to improve automobile stability and protect pedestrians. For example, for safe driving, the vehicle is equipped with a safe driving assistance system such as a drowsy driving warning system, a night obstacle detection system, and a dangerous state warning system of a vehicle.

Among the safe driving assistance systems, the Driver Status Monitoring (DSM) system is a system that warns the driver of neglect due to drowsiness or unnecessary actions not related to driving. As shown in FIG. 1, the conventional driver condition monitoring system 10 is composed of an NIR camera 11 and an ECU 12. In the conventional driver condition monitoring system 10, the NIR camera 11 projects an infrared ray onto the driver's face using the NIR LED, and acquires an image of the driver's face on which infrared rays are projected. The ECU 12 detects eyes, noses, and mouths by image processing the driver's face image obtained from the NIR camera 11 and checks the direction of the driver's gaze and whether the eyes are opened or closed. Through the direction of the driver's gaze and whether the eyes are opened or closed, the ECU 12 can grasp the driver's condition, and accordingly, can determine whether the driver is looking ahead. When it is determined that the driver's negligence is forward, the ECU 12 warns the driver by outputting a warning signal through an output device such as a buzzer.

However, the conventional driver condition monitoring system 10 has a problem in that the image pattern behind the reflective object due to the infrared component of sunlight is saturated when the driver wears glasses or other ornaments that reflect natural light (sunlight). . For example, the conventional driver condition monitoring system 10 has difficulty in recognizing a face pattern (particularly, an eye pattern) as the face image pattern of a driver wearing glasses is saturated as illustrated in FIG. 2. exist.

An object of the present invention is to provide a technical method for obtaining a driver image for preventing a pattern recognition degradation due to sunlight reflection in a driver condition monitoring system.

An object of the present invention is to provide a technical method to obtain a driver image for preventing a pattern recognition deterioration due to solar reflection in a driver condition monitoring system according to an aspect of the present invention for achieving the above-described problems The driver condition monitoring system includes an NIR control unit that lights up an NIR (Near Infrared) light toward the driver, a first face image frame in which the driver's face is photographed while the NIR light is on, and the NIR light is off. An image acquisition unit that alternately acquires a second face image frame in which a driver's face is photographed or selectively acquires only the first face image frame, and calculates an image difference between the first face image frame and the second face image frame And an image processing unit for generating a vehicle image frame, and a driver status monitoring (DSM) performing unit for determining whether the driver is neglecting forward using the vehicle image frame or the first face image frame.

Meanwhile, a method of operating a driver status monitoring (DSM) system including a driver status monitoring (DSM) system according to another aspect of the present invention for achieving the above-described problems may include determining whether it is a current daytime time zone or a nighttime time zone, and the result of the time zone determination. Accordingly, a first face image frame in which a driver's face is photographed while the NIR (Near Infrared) light is on and a second face image frame in which the driver's face is photographed when the NIR light is off are obtained, or Alternatively, selectively obtaining only the first face image frame, and when the first and second face image frames are obtained, calculate a difference in image between the first face image frame and the second face image frame to calculate a difference image frame And generating a driver's negligence by using the car image frame or the first face image frame.

According to the present invention, a face image frame reflecting sunlight and a face image frame reflecting infrared and sunlight are obtained, and whether a driver's driver is neglected to look ahead by using a difference image frame generated according to an image difference between the two images By confirming, light reflection by ornaments that reflect sunlight such as glasses can be reduced, and accordingly, a decrease in performance of recognizing a pattern of faces and eyes in a driver's face image can be prevented.

1 is a diagram illustrating a conventional driver condition monitoring system.
2 is a diagram illustrating an image in which a face image pattern is saturated by light reflection.
Figure 3 is a driver state monitoring system configuration according to an embodiment of the present invention.
4 is a view for explaining an image acquisition and NIR LED flashing cycle of the driver condition monitoring system according to an embodiment of the present invention.
5 is a view for explaining a face pattern detection process of the driver condition monitoring system according to an embodiment of the present invention.
6 is a view for explaining the adjustment of sharpness in the eye pattern image of the driver condition monitoring system according to an embodiment of the present invention.
7 is an operation flowchart of a driver condition monitoring system according to an embodiment of the present invention.

Advantages and features of the present invention, and methods for achieving them will be clarified with reference to embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but will be implemented in various different forms, and only the present embodiments allow the disclosure of the present invention to be complete, and the ordinary skill in the art to which the present invention pertains. It is provided to fully inform the holder of the scope of the invention, and the invention is defined by the claims. On the other hand, the terms used in this specification are for describing the embodiments and are not intended to limit the present invention. In the present specification, the singular form also includes the plural form unless otherwise specified in the phrase. As used herein, "comprises" or "comprising" means the presence of one or more other components, steps, operations and/or elements other than the components, steps, operations and/or elements mentioned, or Addition is not excluded.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. First, in adding reference numerals to the components of each drawing, the same components are assigned the same reference numerals as possible, even if they are displayed on different drawings, and in describing the present invention, specifics of related known configurations or functions are described. When the description may obscure the subject matter of the present invention, the detailed description is omitted.

A driver status monitoring (DSM) system implemented in a recent vehicle is a system that warns the driver of a driver's neglect due to drowsiness or unnecessary actions related to driving. At this time, the driver condition monitoring system checks whether the driver's forward gaze is neglected using the face image of the driver's face.

The driver condition monitoring system according to an embodiment of the present invention controls the blinking of a NIR (Near Infrared) LED (Light Emitting Diode) to display a face image reflecting sunlight (natural light) and a face image reflecting infrared light and sunlight. Acquiring, generating a vehicle image according to the image difference between the two images, and using this, checks whether the driver is looking ahead.

To this end, as shown in FIG. 3, the driver status monitoring system 100 according to an embodiment of the present invention includes an image acquisition unit 110, an image storage unit 120, an NIR control unit 130, and an image processing unit 140 And a DSM performing unit 150.

* The image acquisition unit 110 acquires the face image frame of the driver and is a camera. At this time, the image acquisition unit 110 photographs the image captured through the lens, a camera sensor for converting the captured optical signal into an electrical signal, and a signal processing unit for converting the analog image signal taken from the camera sensor into digital data To be equipped. Here, the camera sensor may be a CCD (Charge Coupled Device) sensor or a Complementary Metal Oxide Semiconductor (CMOS) image sensor, and the signal processor may be implemented as a Digital Signal Processor (DSP), but is not limited thereto.

The image acquisition unit 110 acquires a face image frame of the driver. For example, as illustrated in FIG. 4, the image acquiring unit 110 acquires an image frame in which a driver's face is photographed at every cycle in which a camera image frame input (V_sync) signal is input.

The face image frame acquired by the image acquisition unit 110 is stored in real time in the image storage unit 120. At this time, the face image frames may be stored by being assigned frame numbers in the order of acquisition. Here, the image storage unit 120 may be a non-volatile memory as a means for storing data (face image frame).

The NIR control unit 130 controls the NIR LED that emits near infrared (NIR) light to flash (lit and off) at regular intervals. Here, the NIR LED is mounted at a predetermined position on the vehicle such that the NIR light is turned on toward the driver's face.

In this case, the NIR control unit 130 may control the NIR LED to flash at different periods from the operation period of the image acquisition unit 110. For example, according to the NIR LED control of the NIR control unit 130, the image acquisition unit 110 is a face image frame (first face image frame) in a state in which NIR and sunlight are emitted, and in a state in which only sunlight is emitted. The face image frame (second face image frame) may be alternately acquired. For example, the odd-numbered (1, 3, 5, ...) image frame is a face image frame (first face image frame) in which the driver's face is photographed while both NIR and sunlight are emitted, and the even-numbered image frame (2) , 4, 6, ... th) is a face image frame (second face image frame) in which the driver's face is photographed without NIR being emitted.

Alternatively, the NIR control unit 130 may always turn on the NIR LED whenever the image acquisition unit 110 acquires the driver's face image frame. Accordingly, the image acquisition unit 110 may selectively acquire only the face image frame in the state where the NIR is emitted. In this case, regardless of the number of image frames, each face image frame is a face image frame (first face image frame) in which the driver's face is captured while both NIR and sunlight are emitted. As described above, controlling the operation cycle of the NIR LED in the NIR control unit 130 may be operated differently depending on whether it is a day time zone or a night time zone by the later day and night determination unit 160.

The image processing unit 140 calculates an image difference between the first face image frame and the second face image frame to generate a differential image frame. At this time, the image processing unit 140 generates a difference image frame when the NIR LED operates alternately between lighting and extinguishing by the NIR control unit 130 every period of acquiring a face image frame.

The image processing unit 140 calculates an image difference between a face image frame (first face image frame) in which both NIR and sunlight are emitted and a face image frame (second face image frame) in which NIR is not emitted, and , Thereby generating a difference video frame. By calculating an image difference between a face image frame in which both NIR and sunlight are reflected and a face image frame in which NIR is not reflected, the generated difference image frame may be composed of NIR image components. At this time, the image processing unit 140 may generate an image difference calculation and a difference image frame using an image processing algorithm that calculates an image difference between two images.

For example, the image processing unit 140 between the first face image frame (NIR light + sunlight emitted image frame) and the second face image frame (image light emitted only sunlight) obtained from the image acquisition unit 110 A difference image frame is generated by calculating an image difference. Similarly, the image processing unit 140 between the third face image frame (NIR lighting + sunlight emitted image frame) and the fourth face image frame (image light emitted only sunlight) obtained from the image acquisition unit 110 A difference image frame is generated by calculating an image difference.

The image processing unit 140 transmits the generated difference image frame to the DSM performing unit 150.

On the other hand, the driver state monitoring system 100 according to an embodiment of the present invention may omit the operation of generating a car image frame according to whether it is a current day time zone or a night time zone. To this end, the driver status monitoring system 100 according to an embodiment of the present invention further includes a day and night determination unit 160.

The day and night determination unit 160 determines whether the current day time zone or night time zone is using the time zone determination information.

As an example, the day and night determination unit 160 obtains current date information and time information as time zone determination information, and determines the current time zone in consideration of the acquired current date information and time information. The day and night determination unit 160 may grasp the average sunset and sunrise time according to seasons and seasons using the current date information, and may determine whether it is a daytime or nighttime time zone using the current time information.

As another example, the day and night determination unit 160 obtains the illuminance sensing value of the illuminance sensor as time zone determination information, and determines whether it is a day or night time zone using the obtained illuminance sensing value. The day and night determination unit 160 may determine whether the daylight time zone or the nighttime zone is based on whether the illuminance sensing value is greater than or equal to a predetermined value.

If it is determined by the day and night determining unit 160 during the daytime period, the NIR control unit 130 alternately rotates and turns off the NIR LED lighting at intervals at which the image frame of the face is acquired by the image acquisition unit 120, Accordingly, the image processing unit 140 may generate a difference image.

If it is determined by the day and night determination unit 160 at night time, it is assumed that there is little or no sunlight, and the NIR control unit 130 generates an NIR LED whenever the image acquisition unit 110 acquires a driver's face image frame. Controls to operate. That is, in the case of a night time period, the image acquisition unit 110 acquires a face image frame by photographing a driver's face in a state in which NIR illumination is emitted every frame.

The DSM performing unit 150 checks whether the driver is neglecting the forward gaze using the car image frame received from the image processing unit 140 or the face image frame (first face image frame) obtained from the image acquisition unit 120. . At this time, the DSM execution unit 150 performs a driver status monitoring (DSM) algorithm to check whether or not the forward gaze is neglected, and may use the DSM algorithm used in the conventional driver status monitoring system.

For example, the DSM performing unit 150 uses the car image frame acquired during the daytime to determine the driver's gaze direction and whether the eyes are closed, and uses the driver's gaze direction and whether the eyes are closed to accumulate for a predetermined period of time to use the driver Let's check whether we are negligent. At this time, the DSM performing unit 150 detects a face region and an eye region by processing the difference image frame through an image through a pattern recognition algorithm. In addition, the DSM performer 150 may further perform an image processing operation for sharply adjusting an image of the eye region detected in the car image frame, and then check whether the driver is looking ahead.

In addition, the DSM performing unit 150 receives a face image frame (a first face image frame) obtained from the image acquisition unit 110 at night time to determine whether the driver is neglected to look ahead. For example, in order to determine whether an anterior gaze is present, the DSM performing unit 150 performs image processing on the face image frame as illustrated in FIGS. 5(a), (b), and (c) through a pattern recognition algorithm, and The eye area is detected. Furthermore, the DSM performing unit 150 after image processing to sharply adjust each of the images of the eye region detected as shown in FIGS. 6 (a) and (c) in the face image frame as shown in FIGS. 6 (b) and (d). , It is possible to check whether the driver is looking ahead.

As described above, according to the present invention, a face image frame reflecting sunlight and a face image frame reflecting infrared light and sunlight are obtained, and a driver's driver's attention is neglected by using a difference image frame generated according to an image difference between the two images. By checking whether or not, it is possible to reduce light reflection by ornaments that reflect sunlight such as glasses, and accordingly, it is possible to prevent a decrease in performance of recognizing a pattern of faces and eyes in a driver's face image.

7 is an operation flowchart of a driver condition monitoring system according to an embodiment of the present invention.

The driver status monitoring system 100 acquires time zone determination information (S701). At this time, the driver status monitoring system 100 obtains the current date information and time information as time zone determination information.

The driver status monitoring system 100 determines whether the current day is a time zone using the time zone determination information (S703). For example, the driver status monitoring system 100 may grasp the average sunset and sunrise time according to seasons and seasons using the current date information, and may determine whether it is a daytime or nighttime time using the current time information.

If it is determined in step S703 that it is not a daytime period (S703-N), the driver status monitoring system 100 acquires a face image frame photographing the driver's face in the state where the NIR illumination is emitted (S705). At this time, the driver status monitoring system 100 controls the NIR LED to light whenever the driver's face is photographed. Accordingly, the driver condition monitoring system 100 may acquire a face image frame of the driver in a state in which NIR illumination is always emitted at night time.

The driver condition monitoring system 100 checks whether the driver's forward gaze is neglected using the driver's face image frame (S707). For example, the driver status monitoring system 100 checks whether the driver is looking ahead or not by using a driver status monitoring (DSM) algorithm that checks whether the driver is looking ahead. The driver condition monitoring system 100 detects the driver's face and eye area in the face image frame and determines whether the driver's gaze direction and eyes are closed. The driver status monitoring system 100 checks whether the driver is looking ahead or not by using the result of the driver's gaze direction and whether the eyes are closed, accumulated for a predetermined time.

If, as a result of the determination in step S703, it is determined that the day is a time zone (S703-Y), the driver status monitoring system 100 checks whether the order of obtaining the face image frames is odd (Odd) (S709).

As a result of checking in step S709, if the order of acquiring the face image frames is an odd number (S709-Y), the driver condition monitoring system 100 may take a face image frame (first face) that photographs the driver's face while the NIR light is emitted. Image frame) is acquired (S711), and the acquired first face image frame is stored (S713). At this time, the first face image frame is a face image frame in which NIR and sunlight are reflected together.

After storing the first face image frame, the driver state monitoring system 100 feeds back to step S709 to check whether the order of obtaining the face image frames is odd (S709).

As a result of the verification, if the order of obtaining the face image frames is even (S709-N), the driver condition monitoring system 100 may obtain a face image frame (second face image) of the driver's face while the NIR illumination is not emitted. Frame) is obtained (S715). At this time, the second face image frame is a face image frame in which only sunlight is reflected.

The driver condition monitoring system 100 calculates an image difference between the first face image frame and the second face image frame, and generates a difference image frame (S717).

For example, the driver condition monitoring system 100 may include an image between a face image frame (first face image frame) in which both NIR and sunlight are emitted and a face image frame (second face image frame) in which NIR is not emitted. The difference is calculated, and a difference image frame is generated accordingly. At this time, the image processing unit 140 may generate an image difference calculation and a difference image frame using an image processing algorithm that calculates an image difference between two images. As described above, as the image difference between the face image frame in which both NIR and sunlight is reflected and the face image frame in which NIR is not reflected is calculated, the generated difference image frame may be composed of NIR image components.

The driver condition monitoring system 100 checks whether the driver is looking ahead by using the car image frame (S719).

At this time, the driver condition monitoring system 100 checks whether the driver's forward gaze is neglected using a car image frame and a DSM algorithm acquired during the daytime period. For example, the driver condition monitoring system 100 detects the driver's face and eye area in the car image frame, determines the driver's gaze direction and whether the eyes are closed, and accumulates results for a predetermined time (the driver's gaze direction and whether the eyes are closed) ) To check whether the driver is looking ahead.

As described above, according to the present invention, a face image frame reflecting sunlight and a face image frame reflecting infrared light and sunlight are obtained, and a driver's driver's attention is neglected by using a difference image frame generated according to an image difference between the two images. By checking whether or not, it is possible to reduce light reflection by ornaments that reflect sunlight such as glasses, and accordingly, it is possible to prevent a decrease in performance of recognizing a pattern of faces and eyes in a driver's face image.

The configuration of the present invention has been described in detail through preferred embodiments of the present invention. However, those skilled in the art to which the present invention pertains are disclosed herein without changing the technical spirit or essential features of the present invention. It will be understood that the lesson may be implemented in other specific forms. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. The protection scope of the present invention is indicated by the claims, which will be described later, rather than by the detailed description, and it should be construed that all modifications or variations derived from the claims and equivalent concepts are included in the scope of the present invention.

100: driver condition monitoring system
110: image acquisition unit 120: image storage unit
130: NIR control unit 140: image processing unit
150: DSM performance unit 160: day and night judgment unit

Claims (10)

A day and night judgment unit to determine whether it is currently day or night;
A NIR control unit for lighting a NIR (Near Infrared) light toward the driver;
The first face image frame in which the driver's face is photographed while the NIR light is on and the second face image frame in which the driver's face is photographed when the NIR light is off are obtained alternately, or the first An image acquiring unit selectively acquiring only face image frames;
An image processor generating a difference image frame by calculating an image difference between the first face image frame and the second face image frame; And
A driver status monitoring (DSM) performing unit that checks whether the driver's forward gaze is neglected using the car image frame or the first face image frame based on a result of determining whether it is currently day or night . The method according to claim 1,
Day and night determination unit driver status monitoring system, characterized in that it is determined whether the current day or night using the light intensity sensing value of the illumination sensor. The method according to claim 1,
The DSM performing unit monitors the driver's condition, characterized by determining whether the driver's gaze direction and eyes are closed through the car image frame or the first face image frame and the DMS algorithm to determine whether the driver's gaze is neglected. system. The method according to claim 1,
If it is determined by the day and night judgment section at night,
The NIR control unit lights up the NIR illumination every cycle to acquire the face image of the driver while the NIR illumination is turned on in the image acquisition unit,
The DSM performer, the driver status monitoring system, characterized in that for checking whether the driver's forward gaze is neglected by using the first face image frame acquired at each cycle. The method according to claim 1,
If it is judged by the day and night judgment section weekly,
The NIR controller turns on and off the NIR illumination so that the image acquisition unit alternately acquires the first face image frame and the second face image frame of the driver,
The DSM performing unit, the driver status monitoring system, characterized in that for checking whether the driver's forward attention is neglected using the car image frame. Acquiring a first face image frame in which a driver's face is photographed while a NIR (Near Infrared) light is on and a second face image frame in which the driver's face is photographed when the NIR light is off, or Selectively obtaining only one face image frame;
If the first and second face image frames are obtained, calculating a difference in image between the first face image frame and the second face image frame to generate a difference image frame; And
And using the car image frame or the first face image frame to determine whether the driver is neglected to look ahead. The method according to claim 6,
The checking step,
The vehicle image frame or the first face image frame and the DMS algorithm to determine whether the driver's gaze direction and eyes are closed, and checking the driver's forward gaze. The method according to claim 6,
Before the step of obtaining, determining whether it is currently day or night using the illuminance sensing value of the illuminance sensor;
In the acquiring step, an operation method of a driver condition monitoring system characterized by checking whether the driver is looking ahead or not by using the car image frame or the first face image frame based on a result of determining whether it is currently day or night. . The method according to claim 8,
When it is judged as a week as a result of the judgment,
After the determining step, the step of turning on and off the NIR illumination so that the driver's first face image frame and the second face image frame are obtained alternately;
The checking step is a method of operating a driver condition monitoring system, characterized in that it checks whether the driver is looking ahead or not by using the car image frame. The method according to claim 8,
If it is determined at night as a result of the judgment,
After the determining step, lighting the NIR light every cycle to obtain a face image of the driver while the NIR light is on; Further comprising,
The checking step is a method of operating a driver condition monitoring system, characterized in that it checks whether or not the driver is looking ahead by using the first face image frame obtained at each cycle.

Images