KR101610521B1 - Apparatus and method for tracking gaze - Google Patents

Abstract

The present invention relates to an apparatus and a method for tracking gaze. According to the present invention, the apparatus comprises: a camera detecting gaze of a vehicle driver; a movement detection unit detecting vehicle movement during a predetermined time frame, and a change in a camera pose based on an absolute coordinate system in accordance with the vehicle movement; a vector cone calculation unit calculating a gaze vector in each pose of the camera detected during the time frame, and calculating a cone with respect to the calculated gaze vector; and a control unit tracking a position wherein a cone area of the gaze vector in each pose of the camera is overlapped to determine a gaze coordinate of a driver.

Description

[0001] APPARATUS AND METHOD FOR TRACKING [0002]

TECHNICAL FIELD The present invention relates to a gaze tracking apparatus and method, and more particularly, to a technique for extracting gaze coordinates using a change in a gaze vector depending on a vehicle behavior and a camera pose.

A common eye tracking technique is to determine what the driver is gazing at as a gaze direction vector.

For example, the gaze tracking technique includes a method of estimating a gaze point by calculating a point at which an object located in a gaze vector intersects. However, this method needs to know the exact coordinates of the object based on the eye tracking camera. In addition, it is difficult to judge which method to gaze when there are many objects in the direction of the eye. For example, it is difficult to determine whether a user is looking at a windshield or viewing an object beyond a windshield.

As another example, the gaze tracking technique is a method of estimating the gaze point by calculating the intersection point of the eye vectors of both eyes. However, since this method has a narrow gap between eyes, accurate results can be extracted only when the subject is close to the subject.

An object of the present invention is to provide a gaze tracking apparatus and method capable of extracting highly reliable gaze coordinates by using movement of vibration and running of a vehicle and history of a gaze vector.

According to an aspect of the present invention, there is provided a line-of-sight tracking apparatus for detecting a change of a camera pose based on an absolute coordinate system according to a vehicle motion and a vehicle motion for a predetermined time frame, A line of sight vector calculation unit for calculating a line of sight vector for each pose of the camera detected during the time frame and calculating a cone for the calculated line of sight vector, And a control unit for tracking the position where the cone region of the vehicle is overlapped and determining the coordinates as the driver's gaze coordinates.

The gaze tracking apparatus according to the present invention may further include a time frame size determining unit for determining a time frame size based on at least one of a vehicle motion and a face motion of the driver.

The time frame size determination unit may initialize the time frame size to a reference value if the vehicle motion and the face motion of the driver are not detected.

The time frame size is characterized by being inversely proportional to the vehicle speed, the steering angle and the vehicle vibration value.

And the motion detection section detects movement of the vehicle in the lateral direction, the longitudinal direction and the vertical direction for a predetermined time frame.

Wherein the motion detection unit detects a pose of the camera in each positional coordinate changed based on the motion of the vehicle detected during a predetermined time frame.

Wherein the motion detecting unit transforms the reference position coordinates of the camera using a position conversion matrix corresponding to the motion of the vehicle detected for a predetermined time frame.

According to another aspect of the present invention, there is provided a line-of-sight tracking method comprising: detecting a vehicle motion for a predetermined time frame and a pose change for a line-of-sight camera based on an absolute coordinate system according to the vehicle motion; Calculating a line-of-sight vector for each pose of the line-of-sight camera, calculating a cone for the calculated line-of-sight vector, and calculating a cone of the line-by- And determining the coordinates to be the driver's gaze coordinates.

According to the present invention, it is possible to improve the reliability by reducing the gaze point determination error according to the coordinate change by extracting the gaze coordinates using the movement of the vehicle by vibration and running and the history of the gaze vector, There is an advantage that the gazing point can be extracted without limiting the extraction distance.

Further, according to the present invention, an expensive sensor for judging the gazing point is unnecessary, thereby reducing the cost.

1 is a block diagram showing the configuration of a gaze tracking apparatus according to the present invention.
FIG. 2 and FIG. 3 are views for explaining a camera pose detecting operation according to the vehicle behavior of the eye tracking apparatus according to the present invention.
FIGS. 4 and 5 are diagrams for explaining the coordinate extraction operation of the gaze tracking apparatus according to the present invention.
FIGS. 6 and 7 are flowcharts illustrating an operation flow of the eye tracking method according to the present invention.

It is noted that the technical terms used in the present invention are used only to describe specific embodiments and are not intended to limit the present invention. In addition, the technical terms used in the present invention should be construed in a sense generally understood by a person having ordinary skill in the art to which the present invention belongs, unless otherwise defined in the present invention, and an overly comprehensive It should not be construed as meaning or overly reduced. In addition, when a technical term used in the present invention is an erroneous technical term that does not accurately express the concept of the present invention, it should be understood that technical terms that can be understood by a person skilled in the art can be properly understood. In addition, the general terms used in the present invention should be interpreted according to a predefined or context, and should not be construed as being excessively reduced.

Furthermore, the singular expressions used in the present invention include plural expressions unless the context clearly dictates otherwise. The term "comprising" or "comprising" or the like in the present invention should not be construed as necessarily including the various elements or steps described in the invention, Or may include additional components or steps.

Furthermore, terms including ordinals such as first, second, etc. used in the present invention can be used to describe elements, but the elements should not be limited by terms. Terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings, wherein like reference numerals refer to like or similar elements throughout the several views, and redundant description thereof will be omitted.

In the following description, well-known functions or constructions are not described in detail since they would obscure the invention in unnecessary detail. It is to be noted that the accompanying drawings are only for the purpose of facilitating understanding of the present invention and should not be construed as limiting the scope of the present invention.

1 is a block diagram showing the configuration of a gaze tracking apparatus according to the present invention.

1, a gaze tracking apparatus according to the present invention includes a camera 110, an input unit 120, an output unit 130, a communication unit 140, a storage unit 150, a motion detection unit 160, A decision unit 170, a line vector cone calculation unit 180, and a control unit 190. Here, the control unit 190 may control signals transmitted between the respective units of the gaze tracking apparatus.

The camera 110 is disposed around the driver's seat in the vehicle and photographs the driver's face and / or eye image. At this time, the camera 110 can photograph the motion of the driver's face, iris, pupil, and the like. An image photographed by the camera 110 may be provided to the motion detector 160 and / or the line-of-sight vector cone calculator 180 by the controller 190.

Here, the camera 110 may be an infrared camera or a color camera. In addition, the camera 110 may be a single lens camera or a stereo camera. As such, it is a matter of course that the camera 110 can be applied to any one as long as it is applicable to track the driver's gaze.

The input unit 120 is a means for receiving a control command from a user, and may be a key button implemented outside the gaze tracking device, or may be a soft key implemented on the screen of the gaze tracking device. The input unit 120 may be an input unit such as a mouse, a joystick, a jog shuttle, or a stylus pen.

The output unit 130 may include a display for displaying an operation state of the gaze tracking apparatus, an operation result, and the like. Here, the display may be used as an input device in addition to an output device when a sensor for sensing a touch operation is provided. That is, when a touch sensor such as a touch film, a touch sheet, or a touch pad is provided on the display, the display operates as a touch screen, and the input unit 120 and the output unit 130 may be integrated.

The display unit may include a liquid crystal display (LCD), a thin film transistor liquid crystal display (TFT LCD), an organic light-emitting diode (OLED), a flexible display, , A field emission display (FED), and a 3D display (3D display).

The communication unit 140 may include a communication module that supports communication interfaces with electrical components and / or control units provided in the vehicle. As an example, the communication unit 140 can transmit the gaze tracking result to the control unit that controls the vehicle according to the sight line of the driver. The communication unit 140 may include a communication module that supports vehicle network communication such as CAN (Controller Area Network) communication, LIN (Local Interconnect Network) communication, and Flex-Ray communication.

Also, the communication unit 140 can perform a wired / wireless communication function with an arbitrary external terminal. At this time, the communication unit 140 may include a module for wireless Internet access or a module for short range communication. Here, the wireless Internet technology includes a wireless LAN (WLAN), a wireless broadband (Wibro), a Wi-Fi, a World Interoperability for Microwave Access (Wimax), a High Speed Downlink Packet Access ), Etc., and the short range communication technology may include Bluetooth, ZigBee, UWB (Ultra Wideband), RFID (Radio Frequency Identification), Infrared Data Association (IrDA) have. The wired communication technology may include USB (Universal Serial Bus) communication and the like.

The storage unit 150 may store data and programs necessary for the gaze tracking apparatus to operate. For example, the storage unit 150 may store a set value for a gaze tracking operation and a gaze tracking result of the gaze tracking apparatus, and may store a position transformation matrix for transforming a reference position coordinate of the camera 110 have.

In addition, the storage unit 150 may store an algorithm necessary for performing a gaze tracking operation. In this case, the storage unit 150 may store an algorithm for detecting the movement of the vehicle and the face of the driver, and may include an algorithm for determining the time frame size according to the vehicle movement and the driver's face motion, And an algorithm for calculating a line vector cone corresponding to the line vector vector can be stored.

Here, the storage unit 150 may be a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (for example, SD or XD memory A random access memory (SRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an electrically erasable programmable read-only memory (EEPROM) Erasable Programmable Read-Only Memory).

The motion detection unit 160 detects the vehicle motion for a predetermined time frame. As an example, the motion detector 160 may detect motion in the lateral, longitudinal and vertical directions of the vehicle for a predetermined time frame.

In addition, the motion detection unit 160 detects a pose of the camera 110 at each position coordinate changed based on the motion of the vehicle detected for a predetermined time frame. Here, the motion detector 160 may convert the reference position coordinates of the camera 110 using a position transformation matrix corresponding to the detected motion of the vehicle for a predetermined time frame.

The motion detecting unit 160 can detect the face motion of the driver from the image photographed by the camera 110, and can detect the face direction of the driver.

The time frame size determination unit 170 determines a time frame size based on at least one of the vehicle motion detected by the motion detection unit 160 and the face motion of the driver. In other words, the time frame size determination unit 170 determines the time frame size based on the speed at which the vehicle moves, the steering angle, the vibration value, and the like. At this time, the time frame size is inversely proportional to the vehicle speed, the steering angle, and the vehicle vibration value. Similarly, the time frame size determining unit 170 may determine the time frame size based on the face movement speed, the rotation angle, and the like of the driver.

If the time frame size is updated according to the vehicle movement and the face movement of the driver, the motion detector 160 may detect a change in the camera pose according to the vehicle motion and the vehicle motion during the updated time frame size.

On the other hand, the time frame size determining unit 170 initializes the time frame size to the reference value if the motion detecting unit 160 does not detect the vehicle motion and the face motion of the driver. As an example, the reference value for the time frame size may be one. In this case, when the time frame size is initialized to 1, the motion detector 160 detects the camera pose based on the vehicle position for one frame.

The line-of-sight vector cone calculation unit 180 calculates a line-of-sight vector for each pose of the camera 110 detected for a predetermined time frame or a time frame updated by the time-frame size determination unit 170.

At this time, the line-of-sight vector cone calculation unit 180 extracts the position of the pupil from the face and / or eye image of the driver photographed for each pose of the camera 110, and calculates the line-of-sight vector based on the extracted eye position. A technology for calculating eye-gaze vectors by tracking pupil positions of an iris, a pupil, etc. from a face image and / or an eye image photographed through the camera 110 uses a generally known technique, and a detailed description thereof will be omitted do.

In addition, the line-of-sight vector cone calculation unit 180 calculates the line-of-sight vector cone region by reflecting the error range of the line-of-sight vector and the error range of the camera pose calculated from the images taken at the respective pose of the camera 110, (cone).

The control unit 190 disposes the line vector cone for each camera pose generated by the line vector cone calculation unit 180 on the basis of the position where the vehicle and the camera 110 have moved, Calculate the areas where the cones overlap each other. The control unit 190 tracks the position of the area where the line-of-sight vector cone is superimposed and determines it as the coordinates of the driver's gaze.

FIG. 2 and FIG. 3 are views for explaining a camera pose detecting operation according to the vehicle behavior of the eye tracking apparatus according to the present invention.

2 shows a visual vector cone in the case where there is no motion of the vehicle and the driver's face. As shown in FIG. 2, the gaze tracking device initializes the time frame size when the vehicle is in a non-moving state, that is, in a stopped state.

In this case, the gaze tracking device generates a gaze vector cone based on the vehicle position and camera pose for one time frame.

On the other hand, FIG. 3 shows a line-of-sight vector when the vehicle is moving. As shown in Fig. 3, in the state where the vehicle is moving, the gaze tracking device can be determined based on the vehicle speed, the steering angle, and the vehicle vibration value. As an example, if the speed of the vehicle increases, the time creel size may be determined to be a smaller size.

Here, the line of sight vector calculated based on one frame is shown in Fig. 3 (a). At this time, the movement of the vehicle is calculated during the determined time frame size, and the gaze vector is calculated on the basis of the position-moved camera pose after moving the camera as much as the vehicle is moved as shown in FIG.

Therefore, the gaze tracking device can combine the gaze vectors calculated corresponding to the camera pose in the position-converted coordinates according to the movement of the vehicle for a predetermined time frame size when the vehicle is moving, and calculate the overlapped position as the gaze gaze coordinates have.

Although it is shown in FIG. 3 as a line vector shape, it is obvious that the line vector can be expressed in the form of a line vector cone considering the error of the camera pose and the error of the line vector as shown in FIG.

FIGS. 4 and 5 are diagrams for explaining the coordinate extraction operation of the gaze tracking apparatus according to the present invention.

FIG. 4 shows the motion of the vehicle and the line of sight vector cone thereof during a predetermined time frame size.

First, as shown in (a) of FIG. 4, a line vector corresponding to a camera pose at a vehicle position with respect to a start frame can be represented as C1 by considering an error of V1 and V1. As shown in FIG. 4B, the line vector corresponding to the camera pose at the position of the vehicle on the basis of the next frame can be expressed as C2, which is generated by considering the error of V2 and V2. Also, as shown in FIG. 4C, the line of sight vector corresponding to the camera pose at the position of the vehicle on the basis of the next frame can be represented by C3 such that the line vector cone generated considering the error of V3 and V3 is taken.

Here, as shown in FIGS. 4A, 4B, and 4C, the gaze tracking device generates the gaze tracking device for each camera pose while changing the reference position of the camera according to the movement of the vehicle during three time frames 5 are arranged as shown in FIG. 5. At this time, the area Q in which the line-of-sight vector cones C1, C2, C3 arranged in FIG. 5 are overlapped is detected as the coordinates of the driver's gaze.

The operation of the eye-gaze tracking apparatus according to the present invention will now be described in detail.

FIGS. 6 and 7 are flowcharts illustrating an operation flow of the eye tracking method according to the present invention.

Referring to FIG. 6, the gaze tracking device measures the vehicle behavior in advance for a predetermined time frame (S110), and calculates a pose of the sighted camera moved according to the vehicle movement (S120). At this time, the gaze tracking apparatus calculates a gaze vector corresponding to the camera pose calculated in the process of 'S120', and calculates a gaze vector cone considering the calculated gaze vector error (S130).

Steps S110 to S130 are repeated for a predetermined time frame.

If the predetermined time frame size is reached (S140), the gaze tracking device calculates an intersection area of the gaze vector cone calculated for each pose of the gaze camera through the processes of S110 to S130 for a predetermined time frame (S150). At this time, the gaze tracking device extracts the intersection area calculated in the step 'S150' as the coordinates of the driver's gaze (S160).

The coordinates information of the driver extracted in the process of 'S160' may be provided to a unit, a server, a system or the like which performs control, management, etc. using the sight line information of the driver inside and / or outside the vehicle.

Here, the gaze tracking device may determine and / or update the size of the time frame according to the vehicle motion and / or the face motion of the driver.

In other words, the gaze tracking device can determine the time frame size based on the vehicle motion, for example, the vehicle speed, the steering angle, the vibration value, etc., when the vehicle motion occurs (S210) (S220).

In addition, if the driver's facial movement occurs (S230), the gaze tracking apparatus can determine the time frame size according to the facial motion (S240). Here, when at least one of the vehicle movement and the driver's face movement occurs, the gaze tracking device can determine the time frame size based on the movement.

On the other hand, if the gaze tracking device does not generate both the vehicle motion and the driver's face motion, the time frame size can be initialized to the reference value (S250). As an example, the gaze tracking device may initialize the time frame size to one.

If the time frame size is determined through the process of 'S210' to 'S250', the process after 'S110' of FIG. 6 may be performed by applying the determined time frame size.

The above processes may be implemented directly in hardware, software modules executed by a processor, or a combination of the two. The software modules may reside in storage media, such as memory and / or storage, such as RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disks, removable disks, CD-ROMs. An exemplary storage medium is coupled to the processor, which is capable of reading information from, and writing information to, the storage medium. Alternatively, the storage medium may be integral with the processor. The processor and the storage medium may reside within an application specific integrated circuit (ASIC). The ASIC may reside within the user terminal. Alternatively, the processor and the storage medium may reside as discrete components in a user terminal.

As described above, the present invention has been described with reference to particular embodiments, such as specific elements, and specific embodiments and drawings. However, it should be understood that the present invention is not limited to the above- Those skilled in the art will appreciate that various modifications, additions and substitutions are possible, without departing from the essential characteristics of the invention. Therefore, the spirit of the present invention should not be construed as being limited to the embodiments described, and all technical ideas which are equivalent to or equivalent to the claims of the present invention are included in the scope of the present invention .

110: camera 120: input unit
130: output unit 140: communication unit
150: storage unit 160: motion detection unit
170: time frame size determination unit 180: eye line vector cone calculation unit
190:

Claims (13)

A camera for detecting a driver's vision;
A time frame size determining unit for determining a time frame size based on at least one of a vehicle motion and a driver's face motion;
A motion detector for detecting a vehicle motion for a predetermined time frame by the time frame size determining unit and a change of an absolute coordinate system based camera pose according to the vehicle motion;
A line of sight vector cone calculation unit for calculating a line of sight vector for each pose of the camera detected during the time frame and calculating a cone for the calculated line of sight vector; And
A control unit for tracking the position where the cone region of the eye vector of each pose of the camera overlaps to determine the coordinates of the driver's gaze
Wherein the gaze tracking device comprises: delete The method according to claim 1,
The time frame size determination unit may determine,
Wherein if the vehicle motion and the face motion of the driver are not detected, the time frame size is initialized to a reference value. The method according to claim 1,
Wherein the time frame size is inversely proportional to a vehicle speed, a steering angle, and a vehicle vibration value. The method according to claim 1,
Wherein the motion detection unit comprises:
And detects movement of the vehicle in the lateral direction, longitudinal direction and vertical direction for a predetermined time frame. The method according to claim 1,
Wherein the motion detection unit comprises:
And detects a pose of the camera in each positional coordinate changed based on the movement of the vehicle detected during a predetermined time frame. The method of claim 6,
Wherein the motion detection unit comprises:
Wherein the reference position coordinates of the camera are converted using a position conversion matrix corresponding to the motion of the vehicle detected during a predetermined time frame. Determining a time frame size based on at least one of a vehicle motion and a driver's face motion;
Detecting a change in pose with respect to the gaze camera based on the absolute coordinate system in accordance with the vehicle motion and the vehicle motion for a predetermined time frame;
Calculating a gaze vector for each pose of the gaze camera detected during the time frame;
Calculating a cone for the calculated eye vector; And
A step of tracking the position where the cone region of the eye vector of each pose of the camera is superimposed to determine the coordinate of the eye of the driver
/ RTI > delete The method of claim 8,
Wherein determining the time frame size comprises:
And if the vehicle motion and the face motion of the driver are not detected, initializing the time frame size to a reference value. The method of claim 8,
Wherein the detecting comprises:
And detecting movement of the vehicle in the lateral direction, the longitudinal direction and the vertical direction for a predetermined time frame. The method of claim 8,
Wherein the detecting comprises:
And detecting a pose of the camera at each position coordinate changed based on the movement of the vehicle detected during a predetermined time frame. The method of claim 12,
Wherein the detecting comprises:
Further comprising transforming the reference position coordinates of the camera using a position transformation matrix corresponding to the motion of the vehicle detected during a predetermined time frame.

Images