KR102371591B1 - Apparatus and method for determining condition of driver - Google Patents

Abstract

The present invention relates to an apparatus and method for determining a driver's condition. The device according to the present invention includes an eye detector that extracts an eye image from a driver's face image taken by a camera and detects an eye center point from the eye image, and a reflection point formed within a predetermined range from the eye image based on the eye center point. a reflection point detection unit to detect, a reflection point position calculation unit calculating a relative position of a reflection point with respect to the eye center point in the eye image by comparing the positions of the eye center point and the reflection point, and a relative of the calculated reflection point in the eye image and a gaze position determining unit that determines the driver's gaze position based on the position and determines the driver's inattention state according to the driver's gaze position.

Description

Apparatus and method for determining driver status

The present invention relates to an apparatus and method for determining a driver's condition.

A driver status monitoring (DSM) system uses a camera to capture a face image of a driver in order to determine the driver's condition in the vehicle, and uses this to determine the driver's condition.

However, since the driver's face image is taken through the camera, LED irradiation is inevitable in order to cope with light environments such as day/night.

In this way, when the LED is irradiated to the driver's face to take a face image, a clear face image can be obtained. Degradation of the performance of the Driver Condition Monitoring (DSM) system may occur.

In order to reduce the reflection of the LED on the glasses, a technology is used to minimize the LED reflection by mounting polarizing filters with opposite directions on the light emitting part and the light receiving part of the LED, respectively, but the cost increases due to the additional installation of the polarizing filter and , there was a problem that the effect was insignificant in the reflection of sunlight incident in all directions.

An object of the present invention is to determine the driver's careless state by calculating the position of the reflection point of the LED formed on the driver's glasses, and to display the driver's careless state determination result on an HMI (Human-machine Interface) device to improve the performance of the driver's condition monitoring system An object of the present invention is to provide an apparatus and method for determining a driver's condition to be improved.

The technical problems of the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art from the following description.

An apparatus according to an embodiment of the present invention for achieving the above object includes an eye detector extracting an eye image from a face image of a driver photographed by a camera and detecting an eye center point from the eye image, and from the eye image A reflection point detection unit for detecting a reflection point formed within a predetermined range with respect to the eye center point, a reflection point position calculation unit for calculating the relative position of the reflection point with respect to the eye center point in the eye image by comparing the positions of the eye center point and the reflection point; and a gaze position determiner configured to determine the driver's gaze position based on the relative position of the calculated reflection point in the eye image, and to determine the driver's inattention state according to the driver's gaze position.

On the other hand, the method according to an embodiment of the present invention for achieving the above object, extracting an eye image from a face image of a driver photographed by a camera and detecting an eye center point from the eye image, from the eye image detecting a reflection point formed within a predetermined range with respect to the eye center point; calculating a relative position of the reflection point with respect to the eye center point in the eye image by comparing the positions of the eye center point and the reflection point; and the eye and determining the driver's gaze position based on the relative position of the calculated reflection point in the image, and determining the driver's careless state according to the driver's gaze position.

According to the present invention, the driver's careless state is determined by calculating the position of the reflection point of the LED formed on the driver's glasses, and the driver's careless state determination result is displayed on the HMI (Human-machine Interface) device to improve the performance of the driver's condition monitoring system There is an effect that can make it happen.

1 is a diagram illustrating a configuration of an apparatus for determining a driver's state according to an embodiment of the present invention.
2A and 2B are diagrams illustrating an exemplary embodiment referenced to describe an operation of extracting a reflection point of an apparatus for determining a driver's state according to an embodiment of the present invention.
3 and 4 are diagrams illustrating an exemplary embodiment referenced to describe a reflection point position determination operation of the driver state determination apparatus according to an embodiment of the present invention.
5A to 5G are diagrams illustrating an exemplary embodiment referenced for explaining a gaze position determination operation of the driver state determination apparatus according to an embodiment of the present invention.
6 is a diagram illustrating an example referenced to explain an operation of outputting an alarm of the driver's condition determination apparatus according to an embodiment of the present invention.
7 is a diagram illustrating an operation flow of a method for determining a driver's condition according to an embodiment of the present invention.
8 is a diagram illustrating a computing system to which an apparatus according to an embodiment of the present invention is applied.

Hereinafter, some embodiments of the present invention will be described in detail with reference to exemplary drawings. In adding reference numerals to the components of each drawing, it should be noted that the same components are given the same reference numerals as much as possible even though they are indicated on different drawings. In addition, in describing the embodiment of the present invention, if it is determined that a detailed description of a related known configuration or function interferes with the understanding of the embodiment of the present invention, the detailed description thereof will be omitted.

In describing the components of the embodiment of the present invention, terms such as first, second, A, B, (a), (b), etc. may be used. These terms are only for distinguishing the elements from other elements, and the essence, order, or order of the elements are not limited by the terms. In addition, unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs. Terms such as those defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning in the context of the related art, and should not be interpreted in an ideal or excessively formal meaning unless explicitly defined in the present application. does not

1 is a diagram illustrating a configuration of an apparatus for determining a driver's state according to an embodiment of the present invention.

The driver state determination apparatus 100 according to the present invention may be implemented inside a vehicle. In this case, the driver state determination apparatus 100 may be integrally formed with the vehicle's internal control units, or may be implemented as a separate device and connected to the vehicle's control units by a separate connection means. Here, the driver state determination apparatus 100 may operate in connection with a driver state monitoring system, a human-machine interface (HMI) device, and/or a vehicle display device.

Accordingly, referring to FIG. 1 , the driver state determination apparatus 100 includes a control unit 110 , an interface unit 120 , a camera 130 , a light emitting unit 135 , a storage unit 140 , an eye detection unit 150 , It may include a reflection point detection unit 160 , a reflection point position calculation unit 170 , and a gaze position determination unit 180 . Here, the controller 110 may process a signal transmitted between each part of the driver state determination apparatus 100 .

The interface unit 120 may include an input unit for receiving a control command from a user and an output unit for outputting an operation state and result of the driver state determination apparatus 100 .

Here, the input means may correspond to a key button, and may correspond to a mouse, a joystick, a jog shuttle, a stylus pen, or the like. In addition, the input means may correspond to a soft key implemented on the display.

The output means may include a display, and may include an audio output means such as a speaker. In this case, when a touch sensor such as a touch film, a touch sheet, or a touch pad is provided in the display, the display operates as a touch screen, and the input means and the output means are integrated.

In this case, the display includes a liquid crystal display (LCD), a thin film transistor liquid crystal display (TFT LCD), an organic light-emitting diode (OLED), and a flexible display (Flexible Display). , a field emission display (FED), and a three-dimensional display (3D display) may include at least one.

Also, the display may include a Head Up Display (HUD).

The camera 130 is a device for capturing an image in the vehicle, and captures an image of the driver's face. Here, the camera 130 is a monocular camera, and it is assumed that the camera 130 is disposed in the lower front direction of the driver, for example, the steering wheel or its vicinity.

The camera 130 captures an image of the driver's face when the driver is riding or while driving, and transmits it to the controller 110 . The controller 110 stores the driver's face image transmitted from the camera 130 in the storage unit 140 , and stores the stored face image in the eye detection unit 150 , the reflection point detection unit 160 , the reflection point position calculation unit 170 , etc. can be provided as

The light emitting unit 135 operates in conjunction with the camera 130 when the camera 130 operates, and emits light in the direction of the driver's face when a face image is captured through the camera 130 . In this case, the light emitting unit 135 may include an infrared LED (IR_LED), but is not limited thereto.

The light emitting unit 135 may be implemented as a device separate from the camera 130 , but may also be formed integrally with the camera 130 .

The storage unit 140 may store data and/or algorithms required for the driver state determination apparatus 100 to operate.

The storage unit 140 may store the driver's face image photographed by the camera 130, and condition information and algorithms set for eye detection, reflection point detection, reflection point position calculation and trial calculation position determination, etc. may be stored. . Also, the storage unit 140 may store a control command for calling the driver's attention according to a result of determining the driver's condition.

Here, the storage unit 140 includes a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), a programmable read-only memory (PROM), and an electrically erasable programmable read (EEPROM). -Only Memory) may include a storage medium.

The eye detector 150 may extract an eye image from the face image captured by the camera 130 , and detect the eye from the extracted eye image. Here, the eye detector 150 may detect the center point of the driver's eye. If the driver wears glasses, the eye detector 150 may extract an eye image including the glasses from the face image captured by the camera 130 .

In this case, the reflection point detector 160 detects the first eye center point for the left eye and the second eye center point for the right eye of the eye image, respectively.

When the detection of the first eye center point and the second eye center point from the eye image is completed, the eye detector 150 may transmit the eye center point detection result to the controller 110 and the reflection point position calculator 170 .

The reflection point detector 160 detects a reflection point formed on the driver's glasses from an image of the driver's eyes including the glasses when the driver wears glasses. An embodiment of detecting a reflection point from an eye image will refer to FIGS. 2A and 2B .

The driver's eye image including the glasses may be displayed as shown in FIG. 2A . The reflection point detector 160 binarizes the eye image shown in FIG. 2A based on the brightness level. The result image obtained by binarizing the eye image shown in FIG. 2A based on the brightness level may be represented as shown in FIG. 2B . Accordingly, the reflection point detection unit 160 detects the light source pattern 211 formed around the driver's eye center position in the binarized image shown in FIG. 2B , and detects the reflection point (or reflection center point) from the detected light source pattern 211 . can

In this case, the reflection point detector 160 detects the first reflection point formed on the left glasses and the second reflection point formed on the right glasses of the eye image, respectively.

Here, it is assumed that the eye image used for detecting the reflection point is the same image as the eye image used for detecting the center point of the driver's eye by the eye detector 150 .

When the detection of the first reflection point and the second reflection point from the eye image is completed, the reflection point detection unit 160 may transmit the reflection point detection result to the controller 110 and the reflection point position calculator 170 .

The reflection point position calculation unit 170 may calculate the reflection point position based on the eye image by comparing the positions of the eye center point and the reflection point detected by the eye detection unit 150 and the reflection point detection unit 160 . Refer to FIG. 3 for an embodiment of calculating the position of the reflection point based on the eye image.

As shown in FIG. 3 , the reflection point position calculation unit 170 includes a first eye center point 311 detected by the eye detection unit 150 in the eye image, and a first reflection point detected by the reflection point detection unit 160 ( 321), and calculates the relative position (distance and direction) of the first reflection point 321 with respect to the first eye center point 311 in the eye image. In addition, the reflection point position calculation unit 170 compares the positions of the second eye center point 315 detected by the eye detection unit 150 and the second reflection point 325 detected by the reflection point detection unit 160 in the eye image. and calculates the relative position (distance and direction) of the second reflection point 325 with respect to the first eye center point 315 in the eye image.

The reflection point position calculation unit 170 provides the calculated relative position information of the first reflection point 321 and the second reflection point 325 to the gaze position determination unit 180 .

The gaze position determiner 180 extracts a spectacle region from an eye image including glasses previously used for detecting the eye center point and the reflection point and divides the region into a plurality of regions. For example, the gaze position determiner 180 may divide the spectacles region of the eye image into a plurality of regions having the same area. For an embodiment of this, refer to FIG. 4 .

As shown in FIG. 4 , the gaze position determiner 180 may divide the left and right glasses regions 411 and 421 in the eye image to have an N×N structure in terms of width and length. Here, it is assumed that N is an odd number. For example, the gaze position determiner 180 may divide the left and right spectacle regions 411 and 421 in the eye image into nine first to ninth regions each having a 3×3 structure. In this case, the gaze position determining unit 180 defines the middle region among the 3×3 separated regions as the first region, and defines eight regions positioned at the edges with respect to the first region as the second region to the ninth region. area can be defined. Of course, this is only one embodiment, and it is natural that determining the order of each area can be variously changed according to the embodiment.

Also, the gaze position determiner 180 may define the driver's gaze direction in response to the first to ninth regions. For example, the first area is in the front direction of the camera, the second area is in the right direction of the camera, the third area is in the upper right direction of the camera, the fourth area is in the front upper direction of the camera, the fifth area is in the upper left direction of the camera, and the sixth area is The camera left direction, the seventh area may be defined as a camera bottom left direction, the eighth area may be defined as a front bottom direction of the camera, and the ninth area may be defined as a camera right lower direction direction.

Examples of gaze directions corresponding to the first to ninth regions will be described with reference to FIGS. 5A to 5G .

5A illustrates an image of a driver who wears glasses and looks at the front of the camera. When the driver looks at the front of the camera as shown in FIG. 5A , a reflection point by the light emitting unit 135 may be formed in a first area among the first to ninth areas of the eye image including glasses.

5B illustrates an image of a driver who wears glasses and looks at the right side of the camera. When the driver looks at the right side of the camera as shown in FIG. 5B , a reflection point by the light emitting unit 135 may be formed in a second area among the first to ninth areas of the eye image including glasses.

5C shows an image of a driver wearing glasses and looking at the top of the front of the camera. As shown in FIG. 5C , when the driver gazes at the top of the front of the camera, a reflection point by the light emitting unit 135 may be formed in a fourth area among the first to ninth areas of the eye image including glasses.

5D illustrates an image of a driver who wears glasses and looks at the left side of the camera. When the driver looks at the left side of the camera as shown in FIG. 5D , a reflection point by the light emitting unit 135 may be formed in a sixth area among the first to ninth areas of the eye image including glasses.

5E illustrates an image of a driver wearing glasses and looking at the lower left corner of the camera. As shown in FIG. 5E , when the driver gazes at the lower left corner of the camera, a reflection point by the light emitting unit 135 may be formed in a seventh area among the first to ninth areas of the eye image including glasses.

5F shows an image of a driver wearing glasses and looking at the lower front of the camera. As shown in FIG. 5F , when the driver looks at the lower front of the camera, a reflection point by the light emitting unit 135 may be formed in an eighth region among the first to ninth regions of the eye image including glasses.

5G illustrates an image of a driver wearing glasses and looking at the lower right corner of the camera. When the driver looks at the lower right corner of the camera as shown in FIG. 5G , a reflection point by the light emitting unit 135 may be formed in a ninth area among the first to ninth areas of the eye image including glasses.

As such, the gaze position determiner 180 may determine in which region the reflection point is located among the first to ninth regions according to the relative position of the reflection point with respect to the eye position. In this case, the gaze position determiner 180 may determine the driver's gaze position with respect to the HUD based on the gaze direction defined in the region where the reflection point is located among the first to ninth regions.

Here, the gaze position determiner 180 may pre-define an inattentive region among the first to ninth regions. For example, the first region, the second region, and the sixth region to the ninth region among the first to ninth regions may be defined as careless regions. Accordingly, when the area in which the reflection point is located among the first to ninth areas corresponds to the careless area, the gaze position determiner 180 may determine that the driver is in a careless state and output an alert. Here, an embodiment of outputting an alert for the driver's careless state will be described with reference to FIG. 6 .

Although not shown in FIG. 1 , the driver state determination apparatus 100 may further include a communication unit (not shown).

The communication unit may include a communication module that supports a communication interface with electronic devices and/or control units provided in the vehicle. As an example, the communication module may be communicatively connected with a display device provided in the vehicle to transmit a result of determining the driver's state and a corresponding control command to the display device.

Here, the communication module may include a module supporting vehicle network communication such as CAN (Controller Area Network) communication, LIN (Local Interconnect Network) communication, and Flex-Ray communication. In addition, the communication module may include a module for wireless Internet access or a module for short range communication.

The operation flow of the device according to the present invention configured as described above will be described in more detail as follows.

7 is a diagram illustrating an operation flow of a method for determining a driver's condition according to an embodiment of the present invention. 7 illustrates an operation of determining a driver's state based on an eye image including glasses when the driver wears glasses.

Referring to FIG. 7 , the driver state determination apparatus 100 according to the present invention detects an eye center point from an eye image extracted from a face image captured by the camera 130 ( S110 ).

In addition, the driver state determination apparatus 100 binarizes the eye image based on the brightness level (S120), and in the binarized image of the process 'S120', a reflection point formed on the driver's glasses from the light source pattern formed around the driver's eye center position is detected (S130).

When the eye center point and the reflection point are detected, the driver state determination apparatus 100 compares the positions of the detected eye center point and the reflection point to calculate the relative position (distance and direction) of the reflection point with respect to the eye center point in the eye image ( S140).

The driver state determination apparatus 100 determines the driver's gaze position based on the relative position of the reflection point calculated in the process 'S140'. Here, the driver state determination apparatus 100 may divide the glasses region of the eye image into nine regions having the same area, and define a gaze direction corresponding to each of the divided regions.

The driver state determination apparatus 100 determines a region in which the reflection point is located among the nine divided regions (S150), and determines the gaze direction defined for the region as the driver's gaze direction, for example, the gaze position of the HUD. It can be done (S160).

Thereafter, the driver state determination apparatus 100 checks whether the driver's gaze is an inattentive area, and if the driver's gaze is in the careless area, recognizes that the driver is in an inattentive state. Accordingly, when it is confirmed that the driver is in an inattentive state (S170), the driver state determination apparatus 100 draws the driver's attention through an alarm notification (S180).

The device 100 according to the present embodiment operating as described above may be implemented as an independent hardware device, and at least one or more processors are included in other hardware devices such as a microprocessor or general-purpose computer system. can be driven by

8 is a diagram illustrating a computing system to which an apparatus according to an embodiment of the present invention is applied.

Referring to FIG. 8 , the computing system 1000 includes at least one processor 1100 , a memory 1300 , a user interface input device 1400 , a user interface output device 1500 , and storage connected through a bus 1200 . 1600 , and a network interface 1700 .

The processor 1100 may be a central processing unit (CPU) or a semiconductor device that processes instructions stored in the memory 1300 and/or the storage 1600 . The memory 1300 and the storage 1600 may include various types of volatile or nonvolatile storage media. For example, the memory 1300 may include read only memory (ROM) and random access memory (RAM).

Accordingly, the steps of a method or algorithm described in connection with the embodiments disclosed herein may be directly implemented in hardware, a software module executed by the processor 1100 , or a combination of the two. A software module resides in a storage medium (ie, memory 1300 and/or storage 1600 ) such as RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM. You may. An exemplary storage medium is coupled to the processor 1100 , the processor 1100 capable of reading information from, and writing information to, the storage medium. Alternatively, the storage medium may be integrated with the processor 1100 . The processor and storage medium may reside within an application specific integrated circuit (ASIC). The ASIC may reside within the user terminal. Alternatively, the processor and storage medium may reside as separate components within the user terminal.

The above description is merely illustrative of the technical spirit of the present invention, and various modifications and variations will be possible without departing from the essential characteristics of the present invention by those skilled in the art to which the present invention pertains.

Accordingly, the embodiments disclosed in the present invention are not intended to limit the technical spirit of the present invention, but to explain, and the scope of the technical spirit of the present invention is not limited by these embodiments. The protection scope of the present invention should be construed by the following claims, and all technical ideas within the scope equivalent thereto should be construed as being included in the scope of the present invention.

100: driver state determination device 110: control unit
120: interface unit 130: camera
135: light emitting unit 140: storage unit
150: eye detection unit 160: reflection point detection unit
170: reflection point position calculation unit 180: gaze position determination unit

Claims (19)

an eye detection unit that extracts an eye image from a driver's face image captured by a camera and detects an eye center point from the eye image;
a reflection point detection unit configured to detect a reflection point formed within a predetermined range based on the eye center point from the eye image;
a reflection point position calculation unit calculating a relative position of a reflection point with respect to the eye center point in the eye image by comparing the positions of the eye center point and the reflection point; and
a gaze position determination unit that determines a driver's gaze position based on the relative position of the calculated reflection point in the eye image, and determines the driver's inattention state according to the driver's gaze position;
including,
The gaze position determination unit,
In the eye image, each region within a predetermined range based on the driver's left eye and right eye is divided to have an NХN structure in which N is an odd number, and a driver's gaze direction corresponding to each of the divided regions is defined. Driver condition judgment device. The method according to claim 1,
The eye image is
It includes an area of glasses worn by the driver,
The reflection point is
Driver state determination device, characterized in that formed in the glasses area. 3. The method according to claim 2,
The reflection point detection unit,
The apparatus for determining a driver's condition, characterized in that in the binarized image obtained by binarizing the eye image based on the brightness level, a light source pattern formed within a predetermined range with respect to the eye center point is detected, and a reflection point is detected from the detected light source pattern. delete delete The method according to claim 1,
The gaze position determination unit,
and dividing each region within a predetermined range based on the driver's left and right eyes in the eye image into nine regions having a 3×3 structure. The method according to claim 1,
The gaze position determination unit,
Driver state determination apparatus, characterized in that a gaze direction corresponding to a center region among the divided regions is defined as a front direction of the camera, and gaze directions corresponding to regions located at an edge with respect to the center region are respectively defined. . The method according to claim 1,
The gaze position determination unit,
A region in which the reflection point is located among the divided regions is identified based on the relative position of the calculated reflection point in the eye image, and the driver's gaze position is determined based on a gaze direction defined corresponding to the identified region Driver state determination device, characterized in that. 9. The method of claim 8,
The gaze position determination unit,
The driver state determination apparatus of claim 1, wherein an inattention region is defined based on a gaze direction defined in each of the divided regions, and when the region in which the reflection point is located corresponds to the inattention region, it is determined that the driver is in an inattentive state. The method according to claim 1,
The gaze position determination unit,
Driver state determination device, characterized in that outputting an alarm when it is confirmed that the driver is in a careless state. extracting an eye image from a driver's face image captured by a camera and detecting an eye center point from the eye image;
detecting a reflection point formed within a predetermined range based on the eye center point from the eye image;
dividing each region within a predetermined range based on the driver's left and right eyes in the eye image to have an NХN structure in which N is an odd number;
By comparing the positions of the eye center point and the reflection point, the relative position of the reflection point with respect to the eye center point is calculated in the eye image, and the driver's gaze direction is determined based on the relative position corresponding to each of the divided regions. defining; and
determining the driver's gaze position based on the relative position of the calculated reflection point in the eye image, and determining the driver's careless state according to the driver's gaze position;
Driver status determination method comprising a. 12. The method of claim 11,
The step of detecting the reflection point,
performing binarization of the eye image based on a brightness level;
detecting a light source pattern formed within a predetermined range based on the eye center point in the binarized image; and
and detecting a reflection point from the detected light source pattern. delete delete 12. The method of claim 11,
The dividing step is
In the eye image, each area within a predetermined range based on the driver's left eye and right eye is divided into nine areas having a 3×3 structure. 12. The method of claim 11,
The step of defining the driver's gaze direction includes:
defining a gaze direction corresponding to a middle area among the divided areas as a front direction of the camera; and
and defining gaze directions corresponding to regions located at edges with respect to the center region, respectively. 12. The method of claim 11,
The step of determining the driver's careless state includes:
identifying an area in which the reflection point is located among the divided areas based on the calculated relative positions of the reflection points in the eye image; and
The method of claim 1, further comprising: determining a gaze position of the driver based on a gaze direction defined in response to the identified area. 12. The method of claim 11,
Prior to the step of determining the driver's careless state,
Further comprising the step of defining an inattentive region based on the gaze direction defined in each of the divided regions,
The step of determining the driver's careless state includes:
The driver state determination method, characterized in that, when the area where the reflection point is located corresponds to the careless area, it is determined that the driver is in the careless state. 12. The method of claim 11,
and outputting an alarm when it is confirmed that the driver is in a careless state.

Images