KR20170135573A - Apparatus for detecting of driver gaze direction - Google Patents

Abstract

According to an embodiment of the present invention, provided is a device for detecting the gaze direction of a driver, comprising: a pattern generation unit installed inside a vehicle to radiate pattern light of a first frequency band on a face of a driver; a first camera unit capturing the face of the driver where the pattern light is projected to generate first image data; a second camera unit capturing a pupil area of the driver to generate second image data; and a control unit detecting the gaze direction of the driver by using the first image data, detecting the pupil direction of the driver by using the second image data, and computing the gaze direction of the driver by combining the pupil direction and the face direction of the driver.

Description

[0001] APPARATUS FOR DETECTING OF DRIVER GAS DIRECTION [0002]

One embodiment of the present invention relates to a driver's gaze direction detecting device, and more particularly to a driver's gaze direction detecting device mounted on a vehicle.

Recently, rapid development of IT technology is affecting the automobile industry, and various IT technologies are being applied to vehicles.

In addition, various safety assistant functions for assuring safety in vehicle operation are mounted on the vehicle.

According to this tendency, in order to determine the direction in which the driver views the vehicle, a method of detecting the eyes of the driver by detecting the eyes of the driver and detecting the direction of the eyes by the camera is being researched and developed. The method of detecting the pupil and determining the gaze of the driver may result in a lower probability of pupil detection or an error when wearing glasses, sunglasses, or a mask. In addition, when the face of the driver is not facing the front, a part of the pupil is obscured and a wrong result is output.

An object of the present invention is to provide a driver's gaze direction detecting apparatus capable of accurately detecting a driver's gaze direction irrespective of an object worn on the driver's face.

It is another object of the present invention to provide a driver's gaze direction detecting apparatus capable of accurately detecting the gaze direction of a driver without error in accordance with the direction, rotation, and movement of the driver's face.

Another object of the present invention is to provide a driver's sight line direction detecting apparatus which can detect a danger level according to a direction of a driver's eyes and output a warning signal or automatically brake the vehicle.

According to an embodiment of the present invention, there is provided a pattern generating apparatus comprising: a pattern generating unit installed in a vehicle and irradiating pattern light of a first wavelength band on a face of a driver; A first camera unit for photographing a face of the driver on which the pattern light is projected to generate first image data; A second camera unit for photographing a pupil area of the driver and generating second image data; And detecting the direction of the driver's face by using the first image data, detecting the direction of the driver's eye using the second image data, and calculating the gaze direction of the driver by combining the face direction and the eye direction of the driver A driver's visual-line-direction detecting device including a control section for controlling the driver's visual-line direction.

The first camera unit may further include a band-pass filter for passing the first wavelength band.

And a lighting unit for irradiating the pupil region of the driver with light of a second wavelength band different from the first wavelength band.

The second camera unit may further include a band-pass filter for passing the second wavelength band.

The first wavelength band may be an infrared region.

The pattern generator may emit pattern light of a horizontal line, a vertical line, or a lattice pattern.

The controller senses a pattern of a first wavelength band in the first image data, and detects a face direction of the driver by analyzing a degree of deformation of the pattern.

The control unit may detect the pupil direction of the driver by analyzing the pupil position in the second image data.

At least one of the pattern generating unit and the first camera unit may track the driver's face.

At least one of the illumination unit and the second camera unit may track the driver's pupil region.

And a warning unit for outputting a warning signal when the viewing direction is out of the preset forward viewing range under the control of the control unit.

And a warning unit for outputting a warning signal when the viewing direction exceeds a predetermined forward viewing range exceeding the threshold time under the control of the control unit.

The control unit may output a vehicle braking signal when the viewing direction exceeds the predetermined forward viewing range exceeding the threshold time.

The control unit may output a vehicle braking signal when an object is detected within a predetermined distance from the front of the vehicle and the visual direction is out of a predetermined forward viewing range.

The driver's gaze direction detecting apparatus according to the present invention can accurately detect the driver's gaze direction regardless of the object worn on the driver's face.

In addition, it is possible to accurately detect the driver's gaze direction without any error depending on the direction, rotation, and movement of the driver's face.

Further, it is possible to output a warning signal or to automatically brake the vehicle by sensing the degree of danger according to the direction of the driver ' s eyes.

1 is a configuration diagram of a driver's gaze direction detecting apparatus according to an embodiment of the present invention,
2 to 5 are conceptual diagrams for explaining the operation of the driver's gaze direction detecting apparatus according to the embodiment of the present invention,
6 to 7 are operational flowcharts of a driver's gaze direction detecting apparatus according to an embodiment of the present invention.

The present invention is capable of various modifications and various embodiments, and specific embodiments are illustrated and described in the drawings. It should be understood, however, that the invention is not intended to be limited to the particular embodiments, but includes all modifications, equivalents, and alternatives falling within the spirit and scope of the invention.

The terms including ordinal, such as second, first, etc., may be used to describe various elements, but the elements are not limited to these terms. The terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the second component may be referred to as a first component, and similarly, the first component may also be referred to as a second component. And / or < / RTI > includes any combination of a plurality of related listed items or any of a plurality of related listed items.

It is to be understood that when an element is referred to as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, . On the other hand, when an element is referred to as being "directly connected" or "directly connected" to another element, it should be understood that there are no other elements in between.

The terminology used in this application is used only to describe a specific embodiment and is not intended to limit the invention. The singular expressions include plural expressions unless the context clearly dictates otherwise. In the present application, the terms "comprises" or "having" and the like are used to specify that there is a feature, a number, a step, an operation, an element, a component or a combination thereof described in the specification, But do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or combinations thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in commonly used dictionaries are to be interpreted as having a meaning consistent with the contextual meaning of the related art and are to be interpreted as either ideal or overly formal in the sense of the present application Do not.

FIG. 1 is a configuration diagram of a driver's gaze direction detecting apparatus according to an embodiment of the present invention, and FIG. 2 is a conceptual diagram illustrating an operation of a driver's gaze direction detecting apparatus according to an embodiment of the present invention.

1 and 2, a driver's gaze direction detecting apparatus 100 according to an embodiment of the present invention includes a pattern generating unit 10, a first camera unit 20, an illumination unit 30, a second camera unit 40, a control unit 50, and a warning unit 60. The driver's gaze direction detecting apparatus 100 according to the embodiment of the present invention can be mounted inside the vehicle and interlocked with the vehicle control system. For example, the driver's gaze direction detecting apparatus 100 according to an embodiment of the present invention can read and use vehicle sensor data, driving information, driver's seat setting information, and the like from a vehicle control system, and outputs a command for controlling the vehicle The vehicle can be controlled.

The pattern generating unit 10 is installed in the vehicle and is capable of irradiating the face of the driver 1 with the pattern light of the first wavelength band. The pattern generating unit 10 can be miniaturized as a module type light source such as a laser diode, an LED, or the like, and can be mounted on a vehicle instrument panel or a center fascia.

The pattern generating section 10 can irradiate pattern light of a horizontal line, a vertical line, or a lattice pattern. However, the present invention is not limited to this, and the pattern light may be configured to include at least one of a solid line, a dotted line, and a mark, and may be a concept including a pattern of various shapes. The pattern light can be composed of a two-dimensional pattern, and marks, solid lines, and dotted lines are arranged at intervals that can detect distortion or deformation of the pattern in a region similar to a face size of a person.

The pattern generating unit 10 can irradiate the light of the invisible light wave band to the face of the driver 1 and can irradiate the light of the infrared wavelength band to the face of the driver 1, for example. The pattern generating unit 10 may determine the height of the driver's seat based on the driver's seat setting information received from the vehicle control system and irradiate the pattern light in a direction predicted to the driver's face.

The pattern generating unit 10 may operate under the control of the control unit 50. The control unit 50 controls the pattern generating unit 10 using the seating state of the driver 1, Can be determined. The pattern generating unit 10 may further include a tilting device. The pattern generating unit 10 may track the face of the driver 1 under the control of the controller 50 and irradiate the pattern light.

Next, the first camera unit 20 can photograph the face of the driver 1 on which the pattern light is projected to generate the first image data. The first camera unit 20 may include a band-pass filter for passing the first wavelength band. The lens 21 of the first camera unit photographs the face of the driver 1 while generating a first image data while directing the same direction as the direction of the light source 11 of the pattern generating unit. At this time, the first camera unit 20 can generate image data for the pattern light projected on the face of the driver 1 by blocking light other than the first wavelength band. That is, the first image data generated by the first camera unit 20 may mean a pattern light image or an image projected on the face of the driver. Dimensional pattern light irradiated by the pattern generating section 10 is deformed along the contour of the driver's face to have a three-dimensional shape.

The first camera unit 20 may operate under the control of the controller 50. The controller 50 may control the operation of the first camera unit 20 based on whether the driver 1 is seated, It is possible to determine whether the operation unit 20 operates. The first camera unit 20 may further include a tilting device. The first camera unit 20 may track and photograph the face of the driver under the control of the controller 50.

Next, the illumination unit 30 can irradiate the pupil region of the driver 1 with the light of the second wavelength band different from the first wavelength band. The lighting unit 30 can be installed in the vehicle and can irradiate the light of the second wavelength band to the face of the driver 1. [ The lighting unit 30 can be miniaturized as a module type light source such as a laser diode, an LED, or the like, and can be mounted on a vehicle instrument panel or a center fascia.

The light in the second wavelength band may include visible light, and the illumination unit 30 may irradiate the light of the wavelength band that minimizes eye fatigue to the pupil area of the driver 1. [ In addition, the illumination unit 30 may emit light of different brightness depending on the ambient illuminance, and may not operate when the pupil area can be easily detected in the second image data due to natural light.

The illuminating unit 30 can determine the height of the driver's seat based on the driver's seat setting information received from the vehicle control system and irradiate light in a direction predicted by the pupil region of the driver 1. [ The illumination unit 30 may further include a tilting device and may track the pupil area of the driver 1 under the control of the control unit 50 and irradiate light.

The lighting unit 30 can operate according to the control of the control unit 50. The control unit 50 determines whether the lighting unit 30 is operated or not based on whether the driver 1 is seated, Can be determined.

Next, the second camera unit 40 can photograph the pupil area of the driver 1 and generate second image data. The second camera unit 40 may include a band-pass filter for passing light of a second wavelength band. The lens 41 of the second camera unit photographs the pupil area of the driver while generating the second image data while directing the same direction as the direction of the light source 31 of the illumination unit. At this time, the second camera unit 40 can cut out light other than the second wavelength band, so that the pupil area of the driver 1 can be clearly photographed without being influenced by the pattern light irradiated by the pattern generating unit 10. [

The second camera unit 40 may operate under the control of the control unit 50. The control unit 50 may control the operation of the second camera unit 40 based on whether the driver 1 is seated, (40). The second camera unit 40 may further include a tilting device. The second camera unit 40 may track the pupil area of the driver 1 under the control of the controller 50.

The control unit 50 detects the face direction of the driver 1 using the first image data and detects the eye direction of the driver 1 using the second image data and detects the face direction of the driver 1 and the eye direction of the driver 1, Direction of the driver 1 can be calculated by combining the directions.

3 to 4 are conceptual diagrams for explaining the operation of the driver's gaze direction detecting apparatus according to the embodiment of the present invention.

Referring to FIG. 3, the controller may detect the pattern of the first wavelength band in the first image data 12, and may detect the face direction of the driver 1 by analyzing the degree of deformation of the pattern. When a pattern light having a two-dimensional planar lattice structure is projected on the face of the driver 1, the pattern light has a deformed or distorted three-dimensional shape due to the face structure of the driver, eyes, nose, mouth, The control unit analyzes the degree of distortion or distortion of the grating lines constituting the pattern light to grasp the position of the driver's eyes, nose, and mouth, thereby detecting the face direction of the driver 1. [ The degree of deformation or distortion of the pattern light differs depending on the direction of the face of the driver 1 so that the controller can detect the direction in which the face of the driver 1 faces by analyzing the degree of distortion or distortion of the first image data 12 have.

Referring to FIG. 4, the control unit can detect the pupil direction of the driver 1 by analyzing the position of the pupil 3 in the second image data 22. The control unit can detect the center and radius of the pupil 3 using the geometric characteristics of the circle. For example, after extracting the circular contour candidate points of the pupil through the mathematical morphological operation in the second image data, The center and radius of the pupil (3) can be detected by obtaining two line segments using the point and taking the intersection point between the vertical bisectors of these line segments as the center of the circle. The control unit can detect the position of the pupil 3 even when a part of the pupil region is covered by detecting the radius and the center of the pupil 3.

Alternatively, the controller may analyze the position of the pupil 3 through the contrast in the second image data 22. That is, the control unit can detect the direction of the driver's eye by analyzing the iris region and the pupil region and analyzing the region where contrast contrast is high to the position of the pupil (3).

Referring to FIG. 5, the controller calculates the gaze direction of the driver by combining the face direction and the pupil direction. When the right direction is defined as the positive direction and the left direction is defined as the negative direction centering on the reference line bisecting the nose of the driver, the driver's gaze direction is the sum of the angle in the facial direction and the angle in the pupil direction .

The control unit can detect the driver's gaze direction according to the following equation (1).

[Equation 1]

That is, the driver's gaze direction is designed to accurately detect the direction the driver is looking at, considering the movement of the pupil in the direction in which the driver's face is oriented. For example, if the driver's face is facing the front, the eyes are looking to the right, the driver's face is facing to the right, and the pupil is facing to the left, the detection of the driver's face direction or pupil direction alone Can not be detected. Therefore, in the embodiment of the present invention, the direction of the driver's eyes can be accurately detected by combining the face direction of the driver and the direction of the pupil.

Referring again to FIG. 1, the warning unit 60 may output a warning signal when the visual direction is out of the preset forward viewing range. The forward viewing range may mean a critical range in which a front view can be ensured at the time of traveling and the warning unit 60 may display a visual or audible warning signal when the driver's sight direction is out of the forward viewing range under the control of the controller 50. [ Can be output to the outside.

Or the warning unit 60 may output a warning signal when the visual direction exceeds the predetermined forward viewing range exceeding the threshold time. The controller 50 controls the warning unit 50 to output a visual or auditory warning signal to the outside when the driver's visual direction is out of the forward viewing range. In this case, the critical time may vary depending on the running state, and may be set differently depending on, for example, the running speed.

Alternatively, the control unit may output the vehicle braking signal when the visual direction exceeds the predetermined forward viewing range exceeding the threshold time. The vehicle braking signal causes the vehicle braking device to operate through the vehicle control system. At this time, the control unit may set the threshold time for outputting the warning signal and the threshold time for outputting the vehicle braking signal differently.

Or the control unit may output a vehicle braking signal when an object is detected within a predetermined distance from the front of the vehicle and the visual direction is out of the predetermined forward viewing range. The control unit may receive object detection information from the vehicle control system.

6 to 7 are operational flowcharts of a driver's gaze direction detecting apparatus according to an embodiment of the present invention.

Referring to FIG. 6, the control unit continuously determines whether the viewing direction is within the forward viewing range (S601).

The control unit outputs a primary warning signal when the visual direction is out of the forward viewing range. The primary warning signal may be, for example, an audible signal (S602).

Next, the control unit outputs a secondary warning signal when the visual direction is out of the forward viewing range for the first threshold time. The secondary warning signal may be an auditory signal whose magnitude is larger than the primary warning signal, and a visual signal may be output together (S603 to 604).

Next, the control unit outputs a vehicle braking signal when the visual direction is out of the forward viewing range for the second threshold time. The second threshold time is set longer than the first threshold time, and can be automatically changed according to the vehicle running information (S605 to 606).

That is, when the driver's line of sight is out of the forward-looking range during driving, the first warning signal and the second warning signal are sequentially outputted. When the time for judging that there is an accident risk is reached, the vehicle braking signal is automatically outputted So that the vehicle can be stopped.

Referring to FIG. 7, the control unit outputs a warning signal when the visual direction is out of the forward viewing range. The warning signal may be, for example, an audible signal (S701 to S702).

Next, in a state where an object is detected within a predetermined distance from the front of the vehicle, the control unit may output the vehicle braking signal if the visual direction is continuously out of the predetermined forward viewing range (S703 to 704).

That is, the warning signal is firstly output when the driver's sight direction deviates from the forward viewing range during driving of the vehicle, and when an object is detected in front of this state, the vehicle can be stopped by automatically outputting the vehicle braking signal have.

As used in this embodiment, the term " portion " refers to a hardware component such as software or an FPGA (field-programmable gate array) or ASIC, and 'part' performs certain roles. However, 'part' is not meant to be limited to software or hardware. &Quot; to " may be configured to reside on an addressable storage medium and may be configured to play one or more processors. Thus, by way of example, 'parts' may refer to components such as software components, object-oriented software components, class components and task components, and processes, functions, , Subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays, and variables. The functions provided in the components and components may be further combined with a smaller number of components and components or further components and components. In addition, the components and components may be implemented to play back one or more CPUs in a device or a secure multimedia card.

10: pattern generator
20: First camera section
30:
40:
50:
60: Warning section

Claims (14)

A pattern generating unit installed in the vehicle and irradiating a pattern light of a first wavelength band to a face of a driver;
A first camera unit for photographing a face of the driver on which the pattern light is projected to generate first image data;
A second camera unit for photographing a pupil area of the driver and generating second image data; And
A face direction of the driver is detected using the first image data, a direction of a pupil of the driver is detected using the second image data, and a direction of a driver's gaze is calculated by combining the face direction and the pupil direction of the driver A driver's visual line direction detecting device comprising a control unit. The method according to claim 1,
Wherein the first camera unit further comprises a band-pass filter for passing the first wavelength band. The method according to claim 1,
And a lighting unit for irradiating light of a second wavelength band different from the first wavelength band to the pupil region of the driver. The method of claim 3,
Wherein the second camera unit further comprises a band-pass filter for passing the second wavelength band. The method according to claim 1,
Wherein the first wavelength band is an infrared region. The method according to claim 1,
Wherein the pattern generating unit irradiates pattern light of a horizontal line, a vertical line, or a lattice pattern. The method according to claim 1,
Wherein the controller detects a pattern of a first wavelength band in the first image data and detects a face direction of the driver by analyzing a degree of deformation of the pattern. The method according to claim 1,
Wherein the control unit analyzes the pupil position in the second image data to detect the pupil direction of the driver. The method according to claim 1,
Wherein at least one of the pattern generating unit and the first camera unit tracks the face of the driver. The method of claim 3,
And at least one of the illumination unit and the second camera unit tracks the driver's eyeball region. The method according to claim 1,
Further comprising a warning unit for outputting a warning signal when the viewing direction is out of a predetermined forward viewing range under the control of the control unit. The method according to claim 1,
And a warning unit for outputting a warning signal when the gaze direction exceeds a predetermined forward viewing range exceeding a threshold time under the control of the control unit. The method according to claim 1,
Wherein the controller outputs a vehicle braking signal when the viewing direction exceeds the predetermined forward viewing range exceeding the threshold time. The method according to claim 1,
Wherein the control unit outputs a vehicle braking signal when an object is detected within a predetermined distance from the front of the vehicle and the visual direction is out of a predetermined forward viewing range.

Images