KR20140132906A - Device and method for mobile tooltip using eye detecting - Google Patents

Abstract

The present invention relates to a mobile tooltip method by tracking pupils and a device thereof. The mobile tooltip method takes images of a user through the camera of a mobile device, identifies the pupils of the user, recognizes the viewing direction of the user, and executes a mouse-over function and a tooltip function at a position on a screen where the viewing direction of the user points. According to the present invention, the mobile device including a mobile tooltip function by tracking pupils includes: a display unit which displays information related to data and content on a screen; a pupil image acquisition unit which acquires the image of pupils by taking images of a user who are looking at the screen; a user distance calculation unit which calculates the distance to the pupils of the user; an eye position calculation unit which sets a reference point by comparing the center of the acquired pupil image and the center of the display screen, sets the ratio between the pupil image and the display image according to the distance to the pupils of the user, and calculates the eye position on the display screen as top, bottom, left, or right according to the movement of the pupil image from the set position based on the set screen ratio; and a control unit which, if the eye position on the display screen does not change for a certain period of time, controls execution of the mouse over function or tooltip function on the eye position on the display screen.

Description

[0001] The present invention relates to a mobile tool tip method and apparatus using eye tracking,

The present invention relates to a technology for executing a tool tip function in a touch input type mobile device. More specifically, the present invention relates to a technique of capturing a user's face through a camera of a mobile device to classify a user's eyes, The present invention relates to a mobile tool tip method and apparatus using a pupil tracking that enables a mouse over function and a tool tip function to be performed on a position of a screen where a user's gaze stays.

Due to the development of technology, display devices such as desktops, laptops, PDAs, and smart displays are not in need of personal use.

However, in terms of the input device, technologies such as a mouse, a keyboard, or a touch screen have been used for a long time in the past. Technology using speech recognition is still used only in some special areas due to technical limitations and is not a commonly used technology.

2. Description of the Related Art [0002] In recent mobile terminals, a touch panel is provided as a command input means of a user for size reduction and ease of use. Such a touch panel recognizes the user's touch operation such as touching and dragging, and processes the user input.

However, in the case of a mobile terminal, it is impossible to implement a mouse over function due to the motion recognition characteristics of the touch panel, and a tool tip function can not be implemented.

On the other hand, the mobile terminal displays information such as images, texts and the like in accordance with the size of the screen,

For this purpose, the user must manually select the necessary part of the screen of the mobile terminal by manually operating the input means such as a keypad, a keyboard, a mouse, and a joystick.

Thus, the conventional terminal is inconvenient to use because the user has to manually operate the input means several times in order to select a desired portion from the displayed information. Particularly, in the case of a mobile terminal, it is difficult to manipulate the input means by hand because the user must operate the input means while holding the terminal by hand, thereby increasing the inconvenience of the user.

In addition, since the mobile terminal is often used while moving, it is difficult to input when the user is unable to use both hands.

Korean Patent Registration No. 847,142 (filed on Jul. 11, 2008, entitled "Pretreatment Method for Face Recognition, Method and Apparatus for Face Recognition Using It)

An object of the present invention to solve the above-mentioned problems is to provide a mobile device that captures a face of a user through a camera of a mobile device, identifies the user's eyes, recognizes the user's gaze, mouse over function and a tool tip function using a pupil tracking method.

According to an aspect of the present invention, there is provided a display apparatus including: a display unit for displaying data, information, and contents on a screen; A pupil image acquiring unit for acquiring a pupil image by photographing a face of a user watching the display screen; A user distance calculating unit for calculating a distance to the pupil of the user; Setting a center of the acquired pupil image and a center of the display screen as a reference, setting an aspect ratio of the pupil image and the display image according to a distance to the pupil of the user, An eye position calculating unit for calculating an eye position of a display screen that is viewed upward, downward, leftward, and rightward according to a change in motion of the pupil image around the set reference; And a controller for controlling a mouse over function or a tool tip function to be performed on an eye position of the display screen when the eye position of the display screen does not change even after a predetermined time passes, / RTI >

In addition, the pupil image obtaining unit may include a camera for photographing the user's face, and the pupil image may be obtained by dividing the pupil portion of the face image captured through the camera.

When the pupil image is moved in a predetermined direction in one of the up, down, left, and right directions with respect to the reference, the eye position calculating unit calculates the eye position The eye position of the display screen can be calculated in units of pixels by using a ratio of pixels shifted by an integral multiple in one direction corresponding to the moving direction of the pupil image.

According to another aspect of the present invention, there is provided a mobile tool tip method using a pupil tracking of a mobile device including a display unit, a pupil image obtaining unit, an eye position calculating unit, and a control unit, Displaying on the screen contents related to the display unit data, information, and contents; (b) the pupil image acquiring unit photographs a face of a user watching the display screen to acquire a pupil image; (c) calculating the distance of the user's distance to the user's pupil; (d) The eye position calculation unit sets the center of the acquired pupil image and the center of the screen of the display unit as a reference, sets the aspect ratio of the pupil image and the display image according to the distance to the pupil of the user step; (e) calculating an eye position of a display screen that is viewed in upward, downward, leftward, and rightward directions according to a change in movement of the pupil image based on the set reference based on the set image ratio; And (f) performing a mouse over function or a tool tip function on the eye position of the display screen when the eye position of the display screen does not change even after a predetermined period of time has elapsed. A mobile tooltip method is provided.

In the step (b), the pupil image acquiring unit may photograph a user's face through a camera, and may divide a pupil portion of the face image photographed through the camera to acquire a pupil image.

In the step (e), when the eye-position calculating unit moves the pupil image by a certain pixel in one of the upward, downward, left, and right directions with respect to the reference, , And the eye position of the display screen may be calculated in units of pixels by using a ratio of pixels shifted by an integral multiple in one direction corresponding to the direction in which the pupil image has moved in the left and right directions.

According to the present invention, a mouse over function and a tool tip function can be implemented in a touch input type mobile device.

Further, the user can easily perform the additional function by fixing the gaze to a specific part while grasping the contents of the screen to be watched even if the user does not operate the input means while the user is holding the terminal by hand, thereby enhancing convenience for the user.

In addition, since the mobile terminal is often used on the move, even if the user's hands are difficult to use, the user can easily input the input without using both hands through mouse-over or tooltip function as long as the user concentrates on the screen.

1 is a block diagram illustrating a functional block of a mobile device according to an embodiment of the present invention.
2 is a flowchart illustrating a mobile tool tip method using eye tracking of a mobile device according to an exemplary embodiment of the present invention.
FIG. 3 is a diagram illustrating an example in which the center of a pupil image according to an embodiment of the present invention is associated with a center of a display screen and is set as a reference.
4 is a diagram illustrating eye positions of a display screen according to a change of a user's pupil image according to an embodiment of the present invention.
FIG. 5 is a diagram illustrating an example in which a tool tip function is executed at an eye position in a display screen viewed by a user according to an embodiment of the present invention when the eye position does not change after a predetermined time.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Reference will now be made in detail to the preferred embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to the like elements throughout. Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram illustrating a functional block of a mobile device according to an embodiment of the present invention.

Referring to FIG. 1, a mobile tool 100 having a mobile tool tip function using a pupil tracking according to an exemplary embodiment of the present invention includes a display unit 110, a pupil image acquisition unit 120, an eye position calculation unit 130, A control unit 140, a storage unit 150, an input unit 160, and a user distance calculation unit 170.

The display unit 110 displays contents related to data, information, and contents on the screen.

That is, the display unit 110 displays information expressed visually in accordance with the control of the control unit 140. [ The information may include not only moving images, text, photographs, and web pages, but also menus informing the functions that the terminal apparatus can perform. In addition, the display unit 110 can enlarge and display a specific portion of the displayed information under the control of the control unit 140. [ As the display unit 110, various types of display means such as a liquid crystal display (LCD) and a touch screen may be used.

The pupil image acquisition unit 120 acquires a pupil image by photographing a face of the user watching the display screen. For this purpose, the pupil image acquisition unit 120 may include a camera for photographing a face of a user, and a pupil image may be obtained by dividing the pupil portion of the face image captured through the camera.

The eye position calculation unit 130 calculates the eye position of the display screen that the pupils watch based on the acquired pupil images.

That is, the eye position calculation unit 130 tracks the gaze trajectory of the user using the user image photographed by the camera, and when the gaze concentration for the terminal operation is recognized through tracking of the gaze trajectory, Position.

For this purpose, the eye position calculation unit 130 sets the center of the pupil image to correspond to the center of the display screen and set the reference point as a reference point. The eye position of the screen can be calculated.

In this case, when the eye position calculating unit 130 moves the eye pixel to a certain pixel in one of the up, down, left, and right directions with respect to the reference point, the eye position calculating unit 130 calculates, And sets the aspect ratios of the pupil image and the display image according to the distance to the device 100 and determines the direction in which the pupil image moves in the up, The eye position of the display screen can be calculated in units of pixels by using a ratio of pixels shifted by an integer multiple such as 2 times, 3 times, or the like in the corresponding direction.

Accordingly, the eye position calculation unit 130 may be implemented with a configuration including a gaze tracking module and a position calculation module.

The line-of-sight tracking module detects line-of-sight convergence for terminal operation through tracking of the line-of-sight trajectory. The gaze concentration for the terminal operation means a state in which the user's gaze is fixed at a predetermined position over a reference time, and the reference time is set so as to distinguish the gaze movement of the ordinary user. However, since the time required for the user to fix the line of sight on the screen for operation of the terminal becomes longer in proportion to the reference time, the reference time is set within a time range in which the user's inconvenience can be minimized.

When the sight line concentration for terminal operation is detected, the position calculation module calculates the line-of-sight concentration position on the screen on which the information is displayed by the display unit 110. [ The line-of-sight concentrated position is achieved by mapping the coordinate system of the pupil image and the coordinate system on the screen displayed on the display unit.

The controller 140 controls the mouse over function or the tool tip function to be performed on the eye position of the display screen when the eye position of the display screen does not change after a predetermined time.

In the embodiment of the present invention, the mouse over is to raise the mouse cursor, to put it without clicking, and the tool tip works with the mouse cursor so that when the user hits the cursor without clicking the item, a small box appears on the item, .

The storage unit 150 stores data or information to be output on the display screen, contents or the like, or stores the acquired pupil image, data input through the input unit, and the like.

The input unit 160 inputs data or information selected or manipulated by the user.

In an embodiment of the present invention, the display unit 110 and the input unit 160 may be implemented as a single device capable of simultaneously performing an input function and an output function in the form of a touch screen, and a separate device .

The user distance calculating unit 170 calculates the distance from the user's eyes to the mobile device 100. For example, the user distance calculating unit 170 may calculate the distance using a radio wave or a wireless signal that is not harmful to a person .

Meanwhile, in the embodiment of the present invention, the controller 140 can use a color filter for pupil recognition. When a user's face is photographed through a camera in the pupil image acquisition unit 120, the illumination is simultaneously turned on in a lighting device having two lights having different wavelengths, and is illuminated on the pupil of the user. Illuminations having different wavelengths can be illuminated with a wavelength of 700 nm band and illuminated with a wavelength of 800 nm band. Therefore, a pupil image obtained by simultaneously lighting lights having different wavelengths acquires a pupil sum image for a wavelength of 700 nm band and a wavelength of 800 nm band.

Next, the control unit 140 separates the acquired pupil image by wavelength band using a color filter. At this time, the color filter is formed by alternately alternately forming cells of R, G, and B, respectively. R, G, and B have different reactivities. Here, the reactivity of R, G and B means sensitivity to light, and the sensitivities of R, G and B are different for each wavelength band. That is, R exhibits the best reactivity at 600 nm, G exhibits the highest reactivity at 550 nm, and B exhibits the highest reactivity at 450 nm.

Next, the controller 140 calculates the reactivity of the separated pupil images for each wavelength band through the following equation (1). The degree of the reaction between the illumination having the wavelength of 700 nm band and the pupil image acquired by the illumination having the wavelength of 800 nm band is detected differently.

Here, V (R), V (G), and V (B) represent the R, G, and B filter values obtained in each cell CELL, Rf 700 represents the reflectivity of the 700- 800 denotes a band 800㎚ one trillion people reflectivity of the _{object, I 700 (R), I} 700 (G), I 700 (B) shows the sensitivity of one trillion people 700㎚ units each R, G, B filters, I ₈₀₀ (R), I ₈₀₀ (G), and I ₈₀₀ (B) represent the sensitivities of the respective R, G, and B filters in the 800 nm illumination band.

Rf (700) and Rf (800), which are the reflectivities of the object, are obtained using Equation (1). In other words, it is to obtain the degree of reactivity for pupil, iris and whiteness in the pupil image.

In the pupil image acquired by the illumination of the wavelength range of 700 nm, the degree of the response to the illumination in the whiteness region and the pupil region of the pupil image becomes different, and the boundary becomes clear. At this time, the iris region and the pupil region have similar degree of reaction, and the boundary is blurred.

Therefore, in a pupil image obtained by illumination with a wavelength band of 700 nm, it is used to search for an iris region when performing iris recognition.

In the pupil image obtained by illumination at the wavelength band of 800 nm, the degree of response to illumination of the whiteness region and iris region becomes similar, and the boundary is blurred. At this time, the iris region and the pupil region have different degrees of reaction, so that the boundary becomes clear.

Therefore, the pupil image obtained by illumination at the wavelength band of 800 nm is used to search for a pupil when performing iris recognition.

2 is a flowchart illustrating a mobile tool tip method using eye tracking of a mobile device according to an exemplary embodiment of the present invention.

Referring to FIG. 2, in the mobile device 100 according to the embodiment of the present invention, the display unit 110 displays contents related to data, information, and content on the screen (S210).

That is, one of data, information, and contents is executed according to the selection or manipulation of the user, and the result of the execution is displayed on the screen. For example, the mobile device 100 may visit a specific site through an Internet connection and provide a web screen or a content screen provided by the site.

At this time, the user views the screen on which one of data, information, and contents is displayed to grasp the contents thereof.

Then, the pupil image acquisition unit 120 acquires a pupil image by photographing a user's face looking at the display screen by operating the camera (S220).

That is, the pupil image acquisition unit 120 may capture a face of a user through a camera and classify a pupil portion of the captured face image to obtain a pupil image.

Here, face recognition technology is one of the key technologies of multimedia database search, and is widely used in various applications such as summary of video, identification of face, image search of human computer interface (HCI), security, surveillance system using face information.

Face recognition is a process in which recognition results vary sensitively according to changes in the internal environment such as identity, age, race, facial expression, ornaments, and external environmental changes such as pose, external illumination, and image process There are characteristics. In particular, it is very important to use an algorithm that is robust against this external illumination change, since external illumination change is a very frequent obstacle factor in face recognition.

In this regard, principal algorithms related to face recognition include PCA (Principal Component Analysis), K-PCA (Kernel-PCA), ICA (Independent Component Analysis), SVM (Support Vector Machine) (Nural-Network: Neural Network), HMM (Hidden Markov Model: Hidden Markov Model).

In addition, the method of distinguishing the pupil portion from the photographed face image can store the minutiae information of the user's pupil or iris, and can distinguish the pupil portion from the user's facial image based on the minutia information of the previously stored pupil.

Next, the user distance calculating unit 170 calculates a distance from the mobile device to the user's eyes (S230). That is, since the sizes of the pupil image and the screen image of the display unit are different from each other, the aspect ratio of the display image according to the size of the pupil image must be set.

Next, the eye position calculation unit 130 sets the center of the acquired pupil image and the center of the display unit as a reference in association with each other, and sets the aspect ratio of the pupil image and the display image according to the distance to the pupil of the user S240).

Then, the eye position calculating unit 130 calculates the eye position of the display screen that is viewed upward, downward, leftward, and rightward according to the movement of the pupil image based on the set reference based on the set aspect ratio (S250).

Generally, when a person looks at a specific point on the screen, the eye is adjusted so that the image of the specific point is located at the center of the retina. That is, the center of the eyeball, more specifically, the center of the pupil (pupil) and a specific point are located on a straight line. Therefore, in order to track the eye, it is necessary to detect the user's eyes, especially the pupil (pupil) region, then to detect the center point of the pupil (pupil) region, and to detect, based on the relationship between the coordinate system of the pupil image and the coordinate system of the screen, The position of the center point of the image is converted into the coordinates of the main point on the screen.

That is, as shown in FIG. 3, the eye position calculating unit 130 sets the center of the pupil image and the center of the display screen in correspondence with each other as a reference, sets the upper, lower, It is possible to calculate the eye position of the display screen viewed in the left and right directions. FIG. 3 is a diagram illustrating an example in which the center of a pupil image according to an embodiment of the present invention is associated with a center of a display screen and is set as a reference. That is, although the sizes of the pupil image and the display screen are different from each other, since the center of the pupil image is matched with the center of the display screen and the reference point is set in synchronization with the center of the display screen, It is determined that the position of the eye viewed by the user on the display screen corresponds to the same direction, so that the position of the user's eyes can be grasped.

For example, the eye position calculating unit 130 may calculate the eye position of the user who watches the display screen when the pupil image photographed through the camera moves to the left with respect to the reference as shown in FIG. 4, To a position that has been moved within a certain distance to the left. 4 is a diagram illustrating eye positions of a display screen according to a change of a user's pupil image according to an embodiment of the present invention.

In addition, as shown in FIG. 4, when the pupil image moves to the right with respect to the reference, the user's eye position can be grasped as a position moved from the center of the display screen to the right side at regular intervals.

In addition, as shown in FIG. 4, when the pupil image moves upward relative to the reference, the user's eye position may be grasped as a position shifted by a predetermined distance from the center of the display screen upward, The user's eye position can be grasped as a position shifted by a predetermined distance from the center of the display screen to the lower side.

In this case, if the eye position calculation unit 130 moves a certain pixel in one of the up, down, left, and right directions with respect to the reference, the eye position calculation unit 130 calculates the eye position The eye position of the display screen can be calculated in units of pixels by using a ratio of pixels shifted by an integral multiple such as 2 times or 3 times in the direction corresponding to the moving direction of the image.

That is, since the size of the pupil image differs from that of the display screen, the eye position of the display screen according to the motion of the pupil image is calculated at a predetermined ratio in units of pixels. For example, if it is determined that the pupil image has shifted 5 pixels in the upward direction with respect to the reference, the eye position of the display screen is shifted upward by 3 times or 5 times, that is, 15 pixels or 25 pixels with respect to the center reference Can be calculated.

In the case where the eye position of the display screen does not change after a certain period of time (S260-Yes) as the eye position remains at a specific position while the user is watching the contents on the display screen, The mouse over function or the tool tip function is executed at the eye position of the display screen having no fluctuation as shown in FIG. FIG. 5 is a diagram illustrating an example in which a tool tip function is executed at an eye position in a display screen viewed by a user according to an embodiment of the present invention when the eye position does not change after a predetermined time.

That is, when the user concentrates the gaze in a fixed direction more than the reference time when the gaze concentrates for the operation of the terminal, the control unit 140 displays the gaze on the screen of the display unit 110, And calculates the concentrated position.

4, by mapping the position of the pupil (pupil) to the coordinates of the screen from the coordinate system of the pupil image captured by the camera and the geometric relationship of the coordinate system of the screen of the display unit 110 . Accordingly, the controller 140 executes a mouse over function or a tool tip function based on the calculated line-of-sight concentration position.

As described above, according to the present invention, the user's face is photographed through the camera of the mobile device to distinguish the user's eyes and recognize the user's gaze, and a mouse over function And a mobile tool tip method and apparatus using pupil tracking that enables tool tip functions to be performed.

It will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present invention as defined by the following claims and their equivalents. Only. The scope of the present invention is defined by the appended claims rather than the detailed description and all changes or modifications derived from the meaning and scope of the claims and their equivalents are to be construed as being included within the scope of the present invention do.

The present invention relates to a mobile tool tip method and apparatus using pupil tracking that enables a mouse over function and a tool tip function to be performed on a position of a screen where a user's gaze is perceived by recognizing the user's gaze on the screen of the mobile device .

100: mobile device 110: display unit
120: pupil image obtaining unit 130: eye position calculating unit
140: control unit 150:
160:

Claims (6)

A display unit for displaying contents related to data, information, and contents on a screen;
A pupil image acquiring unit for acquiring a pupil image by photographing a face of a user watching the display screen;
A user distance calculating unit for calculating a distance to the pupil of the user;
Setting a center of the acquired pupil image and a center of the display screen as a reference, setting an aspect ratio of the pupil image and the display image according to a distance to the pupil of the user, An eye position calculating unit for calculating an eye position of a display screen that is viewed upward, downward, leftward, and rightward according to a change in motion of the pupil image around the set reference; And
A controller for controlling a mouse over function or a tool tip function to be performed on an eye position of the display screen when the eye position of the display screen does not change after a predetermined time;
A mobile tooltip with mobile tooltip tracking using pupil tracking. The method according to claim 1,
Wherein the pupil image obtaining unit includes a camera for photographing the face of the user, and a pupil image is obtained by dividing the pupil portion of the face image photographed through the camera, Mobile device. The method according to claim 1,
The eye position calculating unit may calculate the eye position of the pupil image in the upper, lower, left, and right directions with respect to the center of the display screen, when the pupil image is moved by a certain pixel in one of the upward, Wherein the eye position of the display screen is calculated in units of pixels by using a ratio of pixels shifted by an integral multiple in one direction corresponding to the moving direction of the mobile unit. A mobile tool tip method using a pupil tracking of a mobile device including a display unit, a pupil image acquisition unit, an eye position calculation unit, a user distance calculation unit, and a control unit,
(a) displaying, on the screen, contents related to data, information and contents of the display unit;
(b) the pupil image acquiring unit photographs a face of a user watching the display screen to acquire a pupil image;
(c) calculating the distance of the user's distance to the user's pupil;
(d) The eye position calculation unit sets the center of the acquired pupil image and the center of the screen of the display unit as a reference, sets the aspect ratio of the pupil image and the display image according to the distance to the pupil of the user step;
(e) calculating an eye position of a display screen that is viewed in upward, downward, leftward, and rightward directions according to a change in movement of the pupil image based on the set reference based on the set image ratio; And
(f) executing a mouse over function or a tool tip function at an eye position of the display screen when the eye position of the display screen does not change after a predetermined time;
A mobile tool tip method using a pupil tracking of a mobile device. The method of claim 4,
Wherein the pupil image acquiring unit photographs a face of a user through a camera and classifies a pupil portion of the face image photographed through the camera to obtain a pupil image of the mobile device Mobile tooltip method using. The method of claim 4,
In the step (e), when the eye position calculation unit moves the pupil image by a certain pixel in one of the up, down, left, and right directions with respect to the reference, , And the eye position of the display screen is calculated in units of pixels by using a ratio of pixels shifted by an integer multiple in one direction corresponding to the moving direction of the pupil image in the right direction. .

Images