KR101671839B1 - Input device using eye-tracking - Google Patents

Abstract

A line-of-sight tracking input device is disclosed. The line-of-sight-tracking input device according to an embodiment of the present invention includes a display unit for displaying an image keyboard having a plurality of keys arranged corresponding to inputable characters or symbols through image output means; A gaze tracking unit for calculating gaze coordinates according to a user's gaze on the image keyboard; And a sensing unit for sensing a character or a symbol to be input by the user according to the calculated gazing coordinates.

Description

[0001] INPUT DEVICE USING EYE-TRACKING [0002]

Embodiments of the present invention relate to input device technology using a user's gaze.

Eye Tracking is a technology to track the position of the eye by sensing the motion of the user's eyeball. Image analysis, contact lens method, and sensor attachment method can be used. The image analysis method detects the movement of the pupil through analysis of the real-time camera image and calculates the direction of the eye based on the fixed position reflected on the cornea. The contact lens method uses the reflected light of the built-in contact lens of the mirror or the magnetic field of the contact lens built in the coil, which is less convenient and has higher accuracy. The sensor attachment method detects the movement of the eye using the change of the electric field according to the movement of the eye by attaching the sensor around the eye, and it is possible to detect the movement of the eye even when the eye is closed (sleeping, etc.).

In recent years, the devices to which the eye tracking technology is applied are widening, and technology for precise eye detection is continuously being developed. Accordingly, there has been an attempt to apply the gaze tracking technology to typing of characters. However, there has been a limit to the accuracy and speed of character typing by conventional eye tracking.

Korean Patent Publication No. 10-2013-0049915 (Feb.

Embodiments of the present invention are intended to provide an input means for inputting characters, symbols, etc. through eye-tracking of a user.

According to an exemplary embodiment of the present invention, there is provided a display apparatus comprising: a display unit displaying an image keyboard through an image output unit, the image keyboard having a plurality of keys arranged corresponding to inputable characters or symbols; A gaze tracking unit for calculating gaze coordinates according to a user's gaze on the image keyboard; And a sensing unit for sensing a character or a symbol to be input by the user according to the calculated gazing coordinates.

The display unit may arrange the plurality of keys in the image keyboard so that the size of the key corresponding to the character or symbol increases as the frequency of use of each character or symbol increases.

The display unit may arrange the plurality of keys in the image keyboard such that the key corresponding to the character or symbol is arranged closer to the center of the screen of the image output means as the frequency of use of each character or symbol is higher .

The image keyboard may include a key for repeatedly inputting a character or a symbol input immediately before.

The gaze tracking unit may repeatedly calculate the gaze coordinates of the user according to a predetermined period.

Wherein the sensing unit constitutes a line of sight movement of the user by using the gaze coordinate, and when there is a bend exceeding a predetermined angle in the line of sight movement, a key corresponding to the coordinates at which the bend occurs, Can be determined by the letter or symbol to be input.

Wherein the gaze tracking unit detects a change in the pupil size of the user on the image keyboard, and the sensing unit is configured to input a key corresponding to the coordinates at which the pupil size change over the predetermined degree is detected, It can be judged by a character or a symbol.

The line-by-line type input device may further include a word recommendation unit for estimating a word to be input by the user from the character or the symbol detected by the sensing unit and displaying the estimated word as a recommendation word through the image output unit .

The word recommendation unit may compare the character or symbol sensed by the sensing unit with a pre-stored word list, and display one or more words selected according to the similarity ranking according to the comparison as the recommendation word.

Wherein the word recommendation unit estimates a word to be input by the user in consideration of a character or a symbol sensed by the sensing unit and a character or a symbol disposed at a position adjacent to the character or symbol sensed by the sensing unit, can do.

Wherein the line-of-sight-tracking input device stores a word list including a plurality of words and a standard line-of-sight movement path for each word, and calculates a distance between a line of sight of the user calculated from the gaze coordinates calculated by the gaze- And a word recommendation unit for comparing standard sight movement paths and displaying one or more words as recommendation words on the screen according to the path similarity according to the comparison.

According to the embodiments of the present invention, a character or a symbol can be input by sensing the user's gaze and using the user's gaze, so that typing can be performed even in an environment where physical keyboard use is difficult.

1 is a block diagram illustrating a configuration of a line-of-sight-tracking input device according to an embodiment of the present invention;
FIG. 2 is a graph showing the frequency of use of each alphabet in the English document
FIG. 3 is a diagram illustrating an example of a part of a virtual keyboard according to an embodiment of the present invention.
4 is an exemplary diagram illustrating a virtual keyboard according to another embodiment of the present invention.
5 is a flowchart illustrating a calibration process in the eye-tracking unit according to an exemplary embodiment of the present invention.
6 is an exemplary diagram for explaining a key input sensing method in a sensing unit according to an embodiment of the present invention;

Hereinafter, specific embodiments of the present invention will be described with reference to the drawings. The following detailed description is provided to provide a comprehensive understanding of the methods, apparatus, and / or systems described herein. However, this is merely an example and the present invention is not limited thereto.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the following description, well-known functions or constructions are not described in detail since they would obscure the invention in unnecessary detail. The following terms are defined in consideration of the functions of the present invention, and may be changed according to the intention or custom of the user, the operator, and the like. Therefore, the definition should be based on the contents throughout this specification. The terms used in the detailed description are intended only to describe embodiments of the invention and should in no way be limiting. Unless specifically stated otherwise, the singular form of a term includes plural forms of meaning. In this description, the expressions "comprising" or "comprising" are intended to indicate certain features, numbers, steps, operations, elements, parts or combinations thereof, Should not be construed to preclude the presence or possibility of other features, numbers, steps, operations, elements, portions or combinations thereof.

1 is a block diagram illustrating a configuration of a line-of-sight-tracking input device 100 according to an embodiment of the present invention. The eye-tracking type input device 100 according to an exemplary embodiment of the present invention refers to a device for tracking a movement of a user on a screen and inputting characters or symbols accordingly. 3, the line-of-sight-tracking input device 100 includes a display unit 102, a gaze tracking unit 104, a sensing unit 106, and an output unit 108, And may further comprise a word recommendation unit 110 according to an embodiment.

The display unit 102 displays an image keyboard on which a plurality of keys corresponding to inputable characters or symbols are arranged through image output means. In one embodiment, the image output means is an apparatus for displaying visually recognizable information by a user, and includes various kinds of display means such as a monitor provided in a personal computer or a laptop computer, a television, a display of a tablet or a smart phone, It means.

The image keyboard is a kind of virtual keyboard displayed on the image output means. In one embodiment, the on-screen keyboard may have a layout that is the same as or similar to a commonly used QWERTY keyboard, or a two-round / three-round keyboard. Most computer-savvy users are familiar with the keyboard layout of a typical keyboard, so if the on-screen keyboard is configured to have a layout similar to a regular keyboard, the user can use the on- .

In another embodiment, the display unit 102 may arrange the plurality of keys in the picture keyboard so that the size of the key corresponding to the character or the symbol increases as the frequency of use of each character or symbol increases . The alphabets or phonemes that make up a particular language differ in their frequency of use. FIG. 2 is a graph showing frequency of use of each of the alphabets obtained by analyzing documents in English. As shown, the alphabets such as E and T are relatively frequently used in the language, while the alphabets such as X, Q, and Z are used in a very low frequency. Accordingly, the virtual keyboard can be configured to increase the size of the key corresponding to the corresponding character or symbol as the character or symbol having a high frequency of use reflects such statistics.

3 is a diagram illustrating a portion of a virtual keyboard according to an embodiment of the present invention. As shown, in the case of a virtual keyboard according to an embodiment of the present invention, a corresponding key is relatively large in the case of an alphabet having a high frequency of use in a language such as E, T, A, and Q, Z, W, Y, etc., the corresponding key can be configured to be relatively small. In the case of a conventional keyboard configured to type by pressing a button with a finger, it is not necessary to increase the size of a specific key according to the frequency of use because the input is performed by pressing the key through the user's fingertip. However, in the case of the virtual keyboard according to the embodiments of the present invention, when the size is large, it is easy for the user to be conspicuous and the range of the user's sight can be increased. Thus, the efficiency of typing can be increased .

Further, in another embodiment, the display unit 102 displays the key or the symbol corresponding to the character or symbol in a position near the center of the screen of the image output means so that the higher the frequency of use of each character or symbol, A plurality of keys can be arranged. An example of this is shown in FIG.

4 is a diagram illustrating an example of a virtual keyboard according to another embodiment of the present invention. In the case of the virtual keyboard according to the present embodiment, a sentence currently being typed is displayed at the center of the screen and an alphabetical key surrounds the periphery. As shown in the figure, in the case of an alphabet having a relatively high frequency of use such as N, I, A, etc., a virtual keyboard is configured such that relatively low frequency alphabets such as W and Q are positioned at the edge of the screen can do. In this case, the effect of reducing the distance of line of sight for inputting a specific sentence can be obtained. This is because frequently used words are concentrated in the central part.

In addition, the on-screen keyboard may be configured to include a separate key for repeatedly inputting a character or symbol inputted immediately before. The line-of-sight-type input device 100 according to an embodiment of the present invention senses movement of a line of sight for each key and performs typing. Therefore, the process of repeatedly inputting the same letter may be relatively difficult compared to other input means. Therefore, when the image keyboard is provided with a separate key for key repetition, and when the sight line is located in the key, the character or symbol immediately before is repeated so that the repetition of the character can be easily recognized.

While the above embodiments have been described with reference to the English alphabet, the same applies to other languages. That is, it is needless to say that the size or position of each key can be configured differently according to the frequency of each phoneme. In addition, although the frequency of each key constituting each language can be initially used from the general literature or the like, it is also possible to adjust the frequency by reflecting the accumulation of the input data of the user. For example, the display unit 102 may dynamically change the layout of the on-screen keyboard to reflect this if a particular user frequently uses a particular alphabet or vice versa.

Next, the gaze tracking unit 104 calculates the gaze coordinates according to the gaze of the user on the image keyboard. Specifically, the gaze tracking unit 104 may repeatedly calculate the gaze coordinates of the user according to a predetermined period. For example, the gaze tracking unit 104 can measure the user's gaze coordinates at intervals of several to several tens of times per second, and configure the path of the user's gaze on the image keyboard when connecting the gaze coordinates .

In an embodiment of the present invention, the gaze tracking unit 104 may be configured to track the user's gaze in various ways and obtain gaze coordinates therefrom. Examples of techniques for tracking the user's gaze include video analysis, contact lens, and sensor attachment. The video analysis method detects the movement of the pupil through analysis of the real-time camera image and calculates the direction of the eye based on the fixed position reflected on the cornea. The contact lens method uses the reflected light of the built-in contact lens of the mirror or the magnetic field of the contact lens built in the coil, which is less convenient and has higher accuracy. The sensor attachment method uses the electric field according to the movement of the eyes by attaching the sensor around the eyes, and it is possible to detect the motion of the eyes even when the eyes are closed (sleeping, etc.). It should be noted, however, that the embodiments of the present invention are not limited to specific line-of-sight tracking methods or algorithms.

In addition, the gaze tracking unit 104 may perform calibration for the user before performing gaze tracking to correct an error according to the characteristics of the user's eyes.

5 is a flowchart illustrating a calibration process in the eye-tracking unit 104 according to an embodiment of the present invention.

In step 502, the gaze tracking unit 104 acquires a user's eye image through a means such as a camera.

In step 504, the gaze tracking unit 104 detects the center of the pupil and the reflection point from the obtained eye image. The center of the pupil and the reflection point becomes a default value for measuring the position of the user's gaze thereafter.

In step 506, the gaze-tracking unit 104 outputs a plurality of minutiae points on the screen to allow the user to gaze, and calculates the difference between the outputted minutiae point and the actual gaze of the user.

In step 508, the gaze tracking unit 104 completes the calibration by mapping the calculated difference in step 506 on the screen.

Next, the sensing unit 106 senses a character or a symbol to be input by the user according to the calculated gazing coordinates. Basically, the sensing unit 106 regards the key of the image keyboard corresponding to the position of the user's gaze as a key that the user desires to input. However, since the user's gaze is not intermittent but continuous, the sensing unit 106 needs to include an identification algorithm for determining which portion of the continuous line of sight movement should be viewed as an input.

In one embodiment, the sensing unit 106 may construct a line-of-sight movement path according to the user's time from the gaze coordinates acquired by the gaze-tracking unit 104. The sensing unit 106 analyzes the shape of the visual path, determines whether or not there is a bend greater than a predetermined angle in the movement path, and if there is a key corresponding to the coordinates at which the bend occurs, It is determined that the user wants to input characters or symbols.

6 is a diagram illustrating an example of a key input sensing method in the sensing unit 106 according to an embodiment of the present invention. For example, assume that the movement of the user's gaze tracked on the virtual keyboard as shown at the top of FIG. 6 is as shown at the bottom of FIG. In this case, except for the starting point of 1 and the ending point of 6, there are four points where the bending occurs at 2, 3, 4, and 5. Accordingly, the sensing unit 106 can sense that the user has attempted to input the word " family " by sequentially connecting the keys of the on-screen keyboard corresponding to the starting point, the position of occurrence of breakage, and the end point. In this case, the sensing unit 106 determines that the point where the user's eyesight has not passed is not intended to type the key of the corresponding portion. For example, in the embodiment shown in FIG. 5, in order to move the user's line of sight from F to A, D and S are sequentially passed. However, since the user's line of sight is not bent at the positions corresponding to D and S, the sensing unit 106 ignores the key input of the corresponding portion.

In addition, in another embodiment, the sensing unit 106 may calculate the gaze movement speed of the user using the gaze coordinates, and transmit a key corresponding to the coordinates whose gaze movement speed is less than a predetermined value to the user Can be determined by the letter or symbol to be input. For example, in the case where the moving speed of the user is fast, the moving speed of the user may be between the key and the key, but may be the intention of inputting the key of the corresponding position when the moving speed is slow. Accordingly, the sensing unit 106 may be configured to sense the moving speed of the line of sight and input a key corresponding to a point at which the moving speed is less than a predetermined value.

Further, in another embodiment, the gaze tracking unit 104 may be configured to detect the user's eye flicker on the on-screen keyboard. In this case, the sensing unit 106 may determine a key corresponding to the detected coordinates of the flicker as a character or a symbol that the user desires to input.

In addition, in another embodiment, the gaze tracking unit 104 senses a pupil size change that is not a flicker of the user on the image keyboard, and the sensing unit 106 detects a change in the pupil size The corresponding key may be determined as a letter or symbol that the user desires to input. In general, it is known that a human pupil expands when pupils are interested or when they are interested. Therefore, when the user finds a desired character among keyboards of the keyboard, the user can perform typing using the effect of increasing the pupil.

In the above-described embodiment, the degree of bending, the change in the line-of-sight traveling speed, the variation in the size of the pupil, and the like can be set appropriately in consideration of the physical characteristics of the user, the size and layout of the screen of the image keyboard, That is, it is noted that embodiments of the present invention are not limited to a specific range of parameters.

Next, the output unit 108 outputs a signal corresponding to the detected character or symbol. For example, the output unit 108 may be configured to output an ASCII code or Unicode corresponding to the detected character or symbol.

Meanwhile, as described above, the line-of-sight-type input device 100 according to an exemplary embodiment of the present invention may further include a word recommendation unit 110.

In one embodiment, the word recommendation unit 110 estimates a word to be input by the user from the character or symbol detected by the sensing unit 106, and displays the estimated word as a recommendation word through the image output means .

The word recommendation unit 110 may include a word list matching the language that the user inputs. Accordingly, when the user inputs a specific character string, the word recommendation unit 110 compares the character string with the stored word list, and displays one or more selected words on the screen as the recommended words according to the similarity ranking. The user can then move his / her eyes to the word he / she is trying to type out of the recommended words, thereby completing the typing of the word without having to input all the remaining alphabets of the word.

In addition, the word recommendation unit 110 may estimate a word to be input by the user in consideration of a character or a symbol sensed by the sensing unit 106 and a character or a symbol disposed at a position adjacent to the image keyboard . For example, when the character detected by the sensing unit 106 is " a ", the word recommendation unit 110 considers q, s, z and the like adjacent to " a " on the QWERTY keyboard, You can estimate the word you want. In this case, even when the user's line-of-sight input is somewhat inaccurate, the recommendation word can be effectively provided.

In another embodiment, the word recommendation unit 110 estimates a word to be input by the user from the gaze coordinate calculated by the gaze tracking unit 104, and outputs the estimated word as a recommendation word through the image output means . Specifically, the word recommendation unit 110 compares the gaze movement path of the user, which is calculated from the gaze coordinates calculated by the gaze tracking unit 104, with the standard gaze movement path for each word, The above words can be displayed on the screen as recommendation words.

In this case, the word recommendation unit 110 may include a word list including a plurality of words and a standard line-of-sight movement path for each word. The standard line of sight movement path is a path through which a user's line of sight must move in order to input each word. The standard line-of-sight movement path may be preset for each word or may be dynamically configured . For example, when the user inputs the same word a plurality of times, the standard line of sight movement path may be an average value of the line-of-sight movement paths obtained at the respective inputs.

In the case of typing using the line-of-sight input, the speed and accuracy are relatively reduced as compared with the typing using a general keyboard. Thus, by using such a word recommendation, loss of typing speed and accuracy can be compensated. Various means are known in the art to which the present invention pertains, for example, an algorithm for calculating the similarity between an input string and a word list, and a detailed description thereof will be omitted here.

On the other hand, an embodiment of the present invention may include a program for performing the methods described herein on a computer, and a computer-readable recording medium including the program. The computer-readable recording medium may include a program command, a local data file, a local data structure, or the like, alone or in combination. The media may be those specially designed and constructed for the present invention, or may be those that are commonly used in the field of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape, optical recording media such as CD-ROMs and DVDs, and specifically configured to store and execute program instructions such as ROM, RAM, flash memory, Hardware devices. Examples of such programs may include machine language code such as those produced by a compiler, as well as high-level language code that may be executed by a computer using an interpreter or the like.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments, but, on the contrary, . Therefore, the scope of the present invention should not be limited to the above-described embodiments, but should be determined by equivalents to the appended claims, as well as the appended claims.

100: eye-tracking type input device
102:
104: eye tracking unit
106:
108:
110: word recommendation section

Claims (12)

A display unit for displaying an image keyboard having a plurality of keys arranged corresponding to inputable characters or symbols through image output means;
A gaze tracking unit for calculating gaze coordinates according to a user's gaze on the image keyboard;
A sensing unit for sensing a character or a symbol to be input by the user according to the calculated gazing coordinates; And
And a word list including a plurality of words and a standard line of sight movement path for each word is stored. The user's line of sight movement path calculated from the gaze coordinates calculated by the gaze-tracking unit is compared with the standard- And a word recommendation unit for displaying one or more words as recommendation words on the screen according to the path similarity according to the comparison.
The method according to claim 1,
Wherein the display unit arranges the plurality of keys in the image keyboard such that a size of a key corresponding to the character or the symbol increases as the frequency of use of each character or symbol increases.
The method according to claim 1,
Wherein the display unit arranges the plurality of keys in the image keyboard such that a key corresponding to the character or symbol is arranged closer to the center of the screen of the image output means as the frequency of use of each character or symbol is higher, Tracked input device.
The method according to claim 1,
Wherein the image keyboard includes a key for repeatedly inputting a character or symbol inputted immediately before.
The method according to claim 1,
Wherein the gaze tracking unit repeatedly calculates the gaze coordinates of the user according to a predetermined period.
The method according to claim 1,
Wherein the sensing unit constructs a line of sight movement of the user using the gaze coordinate,
Wherein a key corresponding to the coordinates at which the bending occurs is determined as a character or a symbol that the user desires to input if there is a bend greater than a predetermined angle in the line of sight movement path.
The method according to claim 1,
The sensing unit may calculate the gaze movement speed of the user using the gaze coordinates,
And determines a key corresponding to a coordinate at which the calculated gaze movement speed is equal to or less than a predetermined value as a character or a symbol to be input by the user.
The method according to claim 1,
Wherein the gaze-tracking unit detects an eyeball flicker of the user on the image keyboard,
Wherein the sensing unit determines a key corresponding to the detected coordinates of the blinking as a character or symbol that the user desires to input.
The method according to claim 1,
Wherein the gaze tracking unit detects a change in pupil size of the user on the image keyboard,
Wherein the sensing unit determines a key corresponding to a coordinate at which the pupil size change is detected by a predetermined degree or more as a character or a symbol that the user desires to input.
The method according to claim 1,
And a word recommendation unit for estimating a word to be input by the user from the character or the symbol detected by the sensing unit and displaying the estimated word as a recommendation word through the image output unit.
The method of claim 10,
Wherein the word recommendation unit compares a character or symbol sensed by the sensing unit with a previously stored word list and displays one or more words selected according to the similarity rank according to the comparison as the recommendation word.
The method of claim 11,
Wherein the word recommendation unit estimates a word to be input by the user in consideration of a character or a symbol sensed by the sensing unit and a character or a symbol disposed at a position adjacent to the character or symbol sensed by the sensing unit, Eye-tracking type input device.

Images