KR101685430B1 - Smart glass interface system and method using gaze gesture - Google Patents

Abstract

A smart glasses interface system and method capable of performing various operations between a line of sight and smart glasses is disclosed. To this end, the present invention provides a visual display device, comprising: a visual-line sensing unit for generating eye-blending information by detecting at least two combined visual-line directions among a plurality of visual-line directions, generating eye blinking information by sensing blinking degree of eyes; A glasses screen portion for displaying a plurality of programmed execution menus on a glasses screen; And a processor unit for generating a predetermined command signal through the gaze combination information and the blinking information to operate an arbitrary electronic device or to execute movement, selection and cancellation of the execution menu.
Therefore, the present invention is excellent in accuracy, so that various types of electronic apparatuses and execution menus can be operated without a remote control operation, and the recognition error of the line of sight can be reduced.

Description

TECHNICAL FIELD [0001] The present invention relates to a smart glasses interface system and method using a gizmic gesture,

The present invention relates to a smart glasses interface system and method, and more particularly, to a smart glasses interface system and method capable of performing various operations between a sight line and smart glasses.

2. Description of the Related Art [0002] In recent years, as information and wireless interface technologies have evolved, various types of user-friendly electronic devices have emerged. As a typical electronic device, smart glasses can be mentioned.

The smart glasses are glasses that display information desired by a user in real time. Internet connection is available, and a microphone or camera is installed to provide the user with the desired information. For example, when you look at the sky, weather forecasts show you the promise information with your friends when you look at the wall. The user can also listen to music through the glasses, stop the phone when he or she is calling, and make video calls.

However, the functions of the above-mentioned smart glasses have not been realized yet, and they have accomplished some functions of the above-mentioned smart glasses by using the technique of eye tracking.

For example, smart glasses called 'Google Glass' were able to combine speech recognition and determine the extent of eye gaze, so that information such as images could be selected and displayed in smart glasses.

In this way, since the conventional smart glasses use only the degree of staying of the eyes as objects to be recognized, various operations (functions) can not be performed and errors are frequently generated.

Korean Patent Publication No. 2011-0101374, Publication Date: September 16, 2011 Title of the invention: Ubiquitous remote controller using eye tracking glasses.

The present invention has been made in order to solve the above-mentioned problems, and it is an object of the present invention to provide an image processing apparatus and a method thereof, which can operate various types of execution menus and internal and external electronic apparatuses displayed on a glasses screen very conveniently using at least two combined lines of sight and eye blinking degree And to provide a smart glasses interface system.

The features of the present invention for achieving the objects of the present invention as described above and performing the characteristic functions of the present invention described below are as follows.

According to one aspect of the present invention, there is provided a smart eyeglass interface system using a gaze gesture, comprising: at least two combined directions of sight lines of a plurality of eye directions to generate eye pair information; A visual-line detecting sensor unit for detecting eye blinking information and generating eye flicker information; A glasses screen portion for displaying a plurality of programmed execution menus on a glasses screen; And a processor unit for generating a predetermined command signal through the gaze combination information and the blinking information to operate any electronic device or to execute movement, selection and cancellation of the execution menu do.

Here, the processor unit according to one aspect of the present invention may utilize the blinking information for selecting and canceling the moved execution menu when the gaze combination information is utilized for the movement function between the execution menus.

In addition, when the processor unit according to one aspect of the present invention utilizes the eye-sight combination information for the movement, selection, and cancellation of the execution menu, the processor blinks the blinking information to the selection and cancellation function of the sub- Can be utilized.

In addition, the processor unit according to one aspect of the present invention can display the corresponding image on the display screen of the electronic device connected to the eyeglass screen via a wired / wireless communication interface by execution according to the selection of the execution menu and sub execution menu.

According to another aspect of the present invention, there is provided a smart glasses interface method using a gaze gesture, comprising: (a) displaying a plurality of programmed execution menus on a glasses screen; (b) detecting a line of sight of the eye and a degree of flickering in the line of sight sensor; (c) generating gaze combination information and eye blinking information through the sight sensing sensor unit through the sensing; And (d) generating a predetermined command signal through the generated gaze combination information and eye blinking information to operate an arbitrary electronic device, or to execute, move, select and cancel the execution menu in the processor unit. A glasses interface method is provided.

Here, in the step (d) according to another aspect of the present invention, when the eye-sight combination information is utilized in the action menu moving function, the eye blinking information can be utilized for selecting and canceling the moved action menu have.

According to another aspect of the present invention, in the step (d), when the gaze combination information is utilized for the movement, selection and cancellation of the execution menu, the blinking information may be displayed on the submenu of the submenu It can be used for selection and cancellation functions.

According to another aspect of the present invention, in the step (d), the corresponding image can be displayed on the glasses screen and the display screen of the electronic device by execution according to the selection of the execution menu and the sub-execution menu.

As described above, according to the present invention, it is possible to operate various types of electronic devices and execution menus without having to operate the remote controller by only knowing the direction of the eyes and the degree of blinking of eyes arranged in the brain according to the manual.

For example, it is very convenient to operate all electronic devices connected to a wired / wireless network, such as executing a specific function of a smart phone to display and turning on / off a TV, selecting a channel, There is an effect that can be made.

Further, according to the present invention, compared to conventional smart glasses (for example, only recognizing the degree of staying in the eyes), the mobility of the gaze direction and the flicker of the eyes quantified as the size of the eyes enable movement and selectivity of the menu It is possible to reduce the recognition error.

FIG. 1 is an exemplary diagram illustrating a smart glasses interface system 100 according to a first embodiment of the present invention.
FIGS. 2 to 5 are diagrams showing an operation state of the execution menu 122 using the eye-combination information and / or the eye flicker information according to the first embodiment of the present invention.
Figs. 6A to 6C are views showing a directional combination of gaze combination information according to the first embodiment of the present invention. Fig.
FIG. 7 is a flowchart illustrating a smart glasses interface method (S100) according to a second embodiment of the present invention.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings, so that those skilled in the art can easily carry out the present invention. In the drawings, like reference numerals refer to the same or similar functions throughout the several views.

First Embodiment

FIG. 1 is an exemplary diagram illustrating a smart glasses interface system 100 according to a first embodiment of the present invention.

As shown in FIG. 1, the smart glasses interface system 100 according to the first embodiment of the present invention operates various electronic devices 140 and specific functions of the smart glasses using a gaze gesture The eyeglass screen unit 120, the processor unit 130, the electronic device 140, and the wired / wireless interface 150 to the user. The gesture refers to a gesture that utilizes various directional and eye conditions of the line of sight.

First, the eye-gaze sensor unit 110 according to the present invention is provided along the center of a rough border of the eyeglass screen unit 120 to be described later to track the directionality of a human eye and recognize the degree of blinking ≪ / RTI >

At any point in the eyeglass screen portion 120, the eyeglass screen portion 120 may be mounted on any portion of the eyeglass screen portion 120 without being provided along the rim of the eyeglass screen portion 120, Of course.

In order to track the directionality of the line of sight 101 as described above, the line of sight sensing unit 120 according to the present invention senses at least two combined line-of-sight directions 101 among a plurality of line-of-sight directions 101, .

The viewing direction 101 may have, for example, a total of eight viewing directions, that is, four diagonal directions existing between the up / down / left / right direction and the up / down / left / right / direction. Accordingly, it is possible to generate gaze combination information by combining the directivity of at least two of the eight gaze directions 101.

In addition, the eye-gaze sensor 110 according to the present invention senses the degree of eye flicker and generates eye flicker information. In this case, the degree of flickering of the eye means a state in which the eye continues to be closed within a limited time, a state in which the eye is closed, and a state in which the eye is repeated a predetermined number of times. This may be in line with sensing the size of the eye over a limited (set) amount of time.

Next, the eyeglass screen unit 120 according to the present invention includes not only the frame of the smart glasses but also the eyeglasses screen 121 which is the same position as the smart eyeglass lens. The eyeglass screen 121 displays execution menus 122 and displays an image based on the operation of the execution menu 122.

The execution menu 122 may have a plurality of programmed icon menus, for example, similar to an app icon displayed on a display screen of a smartphone.

Finally, the processor unit 130 according to the present invention generates a predetermined command signal through the eye combination information and / or the eye flicker information generated by the eye-gaze sensor unit 110 described above. For example, an icon movement command signal for moving from an icon at any specific position among a plurality of execution menus 122 to an icon at another position is generated through the eye combination information, and an icon moved through the eye blinking information is selected The icon selection command signal can be generated.

When the command signal is generated as described above, the processor unit 130 according to the present invention can use any of the electronic devices 140 connected to the wired / wireless interface 150 (communication network), for example, a TV, a smart phone, It can be used to operate the electronic device 140 or to move between the execution menus 122 and execute and cancel the moved execution menus 122 with respect to the execution menus 122 displayed on the glasses screen 121 .

At this time, the movement, selection and cancellation functions of the execution menu 122 according to an arbitrary command signal can be applied to a principle such as movement, selection and cancellation of a menu by a touch by a human hand in a commonly known smart phone .

For example, when the command signal is used (interlocked) with the movement function between the execution menus 122, the eye blinking information is utilized for the selection and cancel function of the moved execution menu 122 ).

In addition, when the command signal is used (combined) in the movement, selection and cancellation function of the menu 122, the eye blinking information may be displayed on the submenu of the submenu (not shown) included in the execution menu 122 And may be utilized (linked) to the selection and cancellation functions.

 The processor unit 130 according to the present invention executes the execution of the execution menu 122 and the execution of the sub execution menu to transmit the corresponding image and data to the electronic device connected to the glasses screen 121 and the wired / wireless communication interface 150 140 on the display screen.

Although it has been described above that the corresponding function is performed by using the eye combination information and the eye flicker information at the same time, it goes without saying that all of the above functions can be performed through either one of them.

Meanwhile, the operation of the electronic device 140 described above may be various examples. That is, not only the basic operation of turning on and off the electronic device 140 such as a mobile phone, a TV, etc., but also various functions such as performing a specific function implemented in each electronic device 140, It can mean that it contains a function.

Hereinafter, how the action menu 122 is actually moved, selected and canceled using the gaze combination information and / or the blinking information described above will be described.

Example of execution menu

FIGS. 2 to 5 are diagrams showing an operation state of the execution menu 122 using the eye-combination information and / or the eye flicker information according to the first embodiment of the present invention.

First, a plurality of execution menus 122 are listed on the eyeglass screen 121 shown in FIG. Assuming that the execution menu 122 of the first row and the first column shown in FIG. 2 is selected (122a), in FIG. 3, the execution menu 122 of one row and three columns (122b). ≪ / RTI >

Next, FIG. 4 shows that the execution menu 122 of the first row and the third column is selected (122c) through the information of the eyebrow pinning information, for example, eyebrow pinning information obtained by blinking the eye twice. Next, FIG. 5 shows that the execution menu 122 of the selected one-row and two-column is the canceled state 122d.
For reference, cancellation of the execution menu 122 is performed by opening eyes more than the normal state, by appropriately combining sight lines or by blinking a plurality of eyes, but is not limited thereto and can be set differently according to the convenience of the user .

In this way, although the execution menu 122 is moved, selected, and canceled using the eye-sight combination information and the eye blinking information, in the above-described state, when moving, selecting, and canceling the other execution menu 122 It is needless to say that eye direction combination information and eye blink information of different directions can be utilized together.

Example of gaze combination information

6 (a) to 6 (c) are views showing a directional combination of eye combination information according to the first embodiment of the present invention. 6A shows that the line of sight combination information 101b in the right direction and the line of sight combination information 101c in the upward direction are combined (combined). In Fig. 6B, 101d and the downward-direction gaze combination information 101e are combined (combined).

On the other hand, FIG. 6C shows that the right-eye line combination information 101f, the upward-direction eye-combination information 101g, and the diagonal line-of-sight combination information 101h from the downward direction are combined (combined).

As can be seen from this example, in the present embodiment, it can be seen that the above-described electronic device 140 can be operated through a combination of at least one line of sight direction, or a function of moving, selecting, and canceling an execution menu can be executed There will be.
For example, menu movement can be done through a combination of right and upward gaze, menu selection through right and down gaze combination, cancellation through left and down gaze combination, Eye direction, and the gaze combination for movement, selection, cancellation, and execution described above is not limited to this, but can be set differently according to the convenience of the user.
In addition, a shortcut function or the like can be set through the above-described line-up from the upward direction and from the upward direction to the downward diagonal line.

Second Embodiment

FIG. 7 is a flowchart illustrating a smart glasses interface method (S100) according to a second embodiment of the present invention.

As shown in FIG. 7, the smart glasses interface method S100 according to the second embodiment of the present invention is an interface method using a gaze gesture, and includes steps S110 to S140. 1 to 6 described in the above-described embodiments, the description will be omitted because it has been described above.

First, in step S110 according to the present invention, a plurality of programmed execution menus 122 are displayed on the glasses screen 121. [

Thereafter, in step S120 according to the present invention, the direction of the eyes and the degree of flickering of the eyes are sensed by the sight line sensor unit 110. [

Thereafter, in step S130 according to the present invention, eye gaze sensor 110 generates gaze combination information and eye flicker information through sensing in step S120. The eye combination information means information including four diagonal directions existing between the up / down / left / right direction and the up / down / left / right / direction, and the eye blinking information continuously keeps the eye in a limited time A state of winding, a state of repeating the two by a predetermined number of times, and the like.

Finally, in step S140 according to the present invention, a predetermined command signal is generated through the eye combination information and the eye flicker information generated in step S130 to operate an arbitrary electronic device 140, The execution menu 122 can be moved, selected and canceled.

The examples of the move, select and cancel functions of the execution menu 122 have been fully described in FIGS. 2 to 5, and a description thereof will be omitted. Here, the movement, selection and cancellation functions of the execution menu 122 according to an arbitrary command signal can be applied to a principle such as movement selection and cancellation of a menu by a touch by a human hand in a commonly known smart phone.

For example, when the command signal is used (interlocked) with the movement function between the execution menus 122, the eye blinking information is utilized for the selection and cancel function of the moved execution menu 122 ).

In addition, when the command signal is used (combined) in the move, select, and cancel functions between the execution menus 122, the blinking information is displayed in the submenu of the submenu (not shown) included in the execution menu 122 And may be utilized (linked) to the selection and cancellation functions.

 In step S140 according to the present invention, the execution of the execution menu 122 and the execution of the sub-execution menu allows the corresponding image and data to be transmitted to the electronic device 140 connected to the eyeglass screen 121 via the wired / wireless communication interface 150 So that it can be displayed on the display screen.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the exemplary embodiments or constructions. You can understand that you can do it. The embodiments described above are therefore to be considered in all respects as illustrative and not restrictive.

100: smart glasses interface system 110: visual line sensor unit
120: Glasses screen part 121: Glasses screen
122: Execution menu 130:
140: Electronic device 150: wired / wireless interface

Claims (8)

A smart glasses interface system for generating a command signal through a gaze gesture and eye flicker information to operate an electronic device and a device,
A line of sight detection unit for generating line-of-sight information by sensing at least two combined line-of-sight directions among a plurality of line-of-sight directions, generating eye blinking information by sensing a blinking degree of eyes;
A glasses screen portion for displaying a plurality of programmed execution menus on a glasses screen; And
And a processor unit for generating a predetermined command signal through the gaze combination information and the blinking information to operate an electronic device and a device connected through a wired / wireless communication interface, or to execute movement, selection and cancellation of the execution menu,
Wherein the processor unit is configured to set a shortcut function through a line combination. The method according to claim 1,
The processor unit,
Wherein the eye blinking information is utilized for selecting and canceling the moved execution menu when the gaze combination information is utilized for the movement function between the execution menus. The method according to claim 1,
The processor unit,
Wherein the eye blinking information is utilized for selecting and canceling a sub-execution menu included in the execution menu when the gaze combination information is utilized for the movement, selection, and cancellation of the execution menu. The method of claim 3,
The processor unit,
And displays the corresponding image on the display screen of the electronic apparatus and the apparatus connected to the eyeglass screen via a wired / wireless communication interface by execution according to the selection of the execution menu and the sub execution menu. 1. A smart glasses interface method for operating an electronic device and a device by generating a command signal through a gaze gesture and eye flicker information,
(a) displaying a plurality of programmed execution menus on a glasses screen;
(b) detecting the direction of the eye and the degree of flicker in the visual line sensor;
(c) generating gaze combination information by sensing at least two combined gaze directions among a plurality of gaze directions through the sensing, generating eye blinking information by sensing the degree of flickering of the eyes; And
(d) generating a predetermined command signal through the generated gaze combination information and eye blinking information to operate an electronic device and a device connected through a wired / wireless communication interface, or executing, selecting, and canceling the execution menu by the processor unit Wherein the processor unit is capable of setting a shortcut function through a line of sight combination. 6. The method of claim 5,
The step (d)
Wherein the eye blinking information is used for selecting and canceling the moved execution menu when the eye-sight combination information is utilized for the menu-to-menu movement function. 6. The method of claim 5,
The step (d)
Wherein the blinking information is used for selecting and canceling a sub-execution menu included in the execution menu when the gaze combination information is utilized for the movement, selection, and cancellation of the execution menu. 8. The method of claim 7,
The step (d)
And displaying the corresponding image on the spectacle screen and the display screen of the electronic apparatus and the apparatus by execution according to the selection of the execution menu and the sub-execution menu.

Images