KR101943206B1 - Method and apparatus for inputting command using illusion user interface - Google Patents

Abstract

A method and apparatus for inputting an instruction using an illusion UI are provided. An apparatus for inputting a command using a user interface (hereinafter referred to as " illusion UI ") using an image causing an optical illusion effect according to an embodiment of the present invention includes a gaze tracking unit for tracking a gaze of a user looking at a screen, A gaze information measuring unit for measuring a gaze of the user when the user's gaze gazes at a specific optical illusion UI displayed on the screen and a gaze information measuring unit for measuring the gaze of the user when the measured eye movement falls within a predetermined threshold range, And a control instruction execution unit for executing a control instruction for executing the control instruction.

Description

TECHNICAL FIELD [0001] The present invention relates to a method and apparatus for inputting a command using an ill-

The present invention relates to a technique for providing a user interface (hereinafter, referred to as 'illusion UI') using an image causing an optical illusion and for inputting a control command corresponding to the illusionary UI based on a user's eye movement .

Command input on a device such as a computer or smart phone should be conscious behavior of the user, especially with the user 's hand. For example, an action of pressing a key button of a keyboard or an operation of touching a specific area of a smartphone may be performed so that an execution command required by a user is made.

Such conventional interfacing is an interfacing that can not be used by persons with disabilities who can not freely use their hands and requires conscious actions, thereby causing fatigue depending on the situation.

Meanwhile, in recent years, studies on VR (Virtual Reality) devices have been rapidly developed. In such a VR environment, it is difficult to perform appropriate interfacing by hand touch operation and button push operation.

An interface that does not require a user's physical behavior is a brain-computer interface. The brain-computer interface uses an invasive method of measuring the activity of the brain by inserting electrodes directly into the brain and a noninvasive method of measuring brain activity using brain waves observed from outside the skull.

Such a brain-computer interface has a problem in that it is practically difficult for an ordinary person to use because an expensive device for measuring brain waves is separately required and accuracy is not guaranteed.

Recently, eye tracking has been used as an interface to track eye movements in VR environment. The human eye moves in the direction that you want to see and is used to track and interface it. However, eye tracking can be utilized only for the purpose of grasping the gaze of the user described above, and the eye tracking can not be utilized as an interface for issuing direct execution commands, It is true.

Disclosure of Invention Technical Problem [8] The present invention provides a method and apparatus for providing an optical illusion UI and inputting various control commands without conscious actions based on a user's eye movement.

According to an aspect of the present invention, there is provided an apparatus for inputting a command using a user interface (hereinafter referred to as " illusion UI ") using an image causing an optical illusion, A line of sight tracking unit for tracking a line of sight of the user; A gaze information measuring unit for measuring an eye movement of the user when the gaze of the user gazes at a specific illusion UI displayed on the screen; And a control command execution unit that executes a control command corresponding to the illusion UI when the measured eye movement is determined to be an eye movement corresponding to the recognition change.

The eye movement corresponding to the cognitive conversion includes at least one of a gaze jump instantaneously converted by the gaze fixation point, a change in the focal distance of the eye, and a change in the pupil size.

Further comprising an illusion UI state change unit for changing the first state of the illusion UI to a second state and displaying the changed state on the screen when a control command corresponding to the illusion UI is executed, The first state and the second state of the illusion UI correspond to different control commands, respectively.

The illusion UI includes a button, and whether or not the button is operated is displayed in the first state and the second state.

The button is implemented as a necker cube, and is implemented with a plurality of buttons by applying one or more of different colors and rotation angles.

The control instruction execution unit may be configured to determine that the plurality of illu- mined UIs are gazed with a time difference and if the measured eye movement with respect to the gazing of each illu- minated UI is determined to be an eye movement corresponding to a cognitive switch, To execute a control command specifying a drag or a block.

When the eye UI is a mouse cursor and the eye movement corresponding to the recognition change is detected in the gaze information measurement unit, the control command execution unit executes a control command to click the mouse cursor.

According to another aspect of the present invention, there is provided a method of inputting a command using a user interface (hereinafter referred to as " illusion UI ") using an image causing an optical illusion, comprising the steps of: (a) step; (b) measuring the eye movement of the user when the user's gaze gazes at the specific optical illusion UI displayed on the screen; And (c) executing a control command corresponding to the illusion UI when the measured eye movement is determined to be an eye movement corresponding to a cognitive conversion.

According to an embodiment of the present invention, by inputting a command to select or deselect the corresponding ill-fated button based on the eye movement of the user gazing at the illusion UI button, the control command Can be input.

In addition, it can be widely used for contents such as augmented reality and virtual reality, as well as the expression of physically disabled persons and device control.

It should be understood that the effects of the present invention are not limited to the above effects and include all effects that can be deduced from the detailed description of the present invention or the configuration of the invention described in the claims.

1 is a block diagram illustrating an apparatus for receiving a command using an illusion UI according to an exemplary embodiment of the present invention.
2 is a diagram showing an illusion UI.
3 is a flowchart illustrating a process of receiving an instruction using an illusion UI according to an exemplary embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, the present invention will be described with reference to the accompanying drawings. The present invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein.

In order to clearly illustrate the present invention, parts not related to the description are omitted, and similar parts are denoted by like reference characters throughout the specification.

Throughout the specification, when a part is referred to as being "connected" to another part, it includes not only "directly connected" but also "indirectly connected" .

Also, when an element is referred to as " comprising ", it means that it can include other elements, not excluding other elements unless specifically stated otherwise.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram illustrating an apparatus for receiving a command using an illusion UI according to an exemplary embodiment of the present invention.

(Hereinafter, referred to as 'command input device') 100 receiving an instruction using an illusion UI according to an embodiment of the present invention includes a gaze tracking unit 110, a gaze information measurement unit 120, A UI unit 140, a controller 150, and a storage unit 160. The UI unit 140 may include a UI unit 130, an UI UI unit 140, a controller 150,

The present invention replaces an existing interface with an image causing an optical illusion and uses a point of occurrence of a specific eye movement or a specific brain wave when the user looks at the image, ). When the user senses a change in the specific eye movement and the pupil size at the time of gazing at the illusionary UI, it is determined that the input is a user input for executing a control command corresponding to the optical illusion UI, and a control command corresponding to the optical illusion UI is executed The invention.

The command input device 100 according to an embodiment of the present invention can be applied to an embodiment that performs various operations through a user interface (button, etc.) provided on a screen, such as a desktop computer, a notebook computer, or a digital TV.

In addition, the command input device 100 according to an embodiment of the present invention may include an HMD (Head / Helmet Mounted Display) or HWD (Head Worn Display) for providing an AR, a VR, The present invention can be applied to an image providing apparatus to be worn on a person's head or face portion.

The present invention replaces an existing UI with an optical illusion UI, wherein the UI may include any type of UI displayed through a display, such as a play button, a stop button, a forward button, a window close button, . ≪ / RTI >

On the other hand, the mouse cursor may also be replaced by an illusion UI. In this case, the mouse cursor may be set to move according to the user's gaze.

In the present invention, the control command is executed by detecting the eye movement occurring when the UI for the specific operation is replaced by the UI.

Referring to FIG. 1, the gaze tracking unit 110 tracks the gaze of the user looking at the screen, and can grasp the point where the user's gaze is directed on the screen. Specifically, the gaze tracking unit 110 grasps a point viewed by the user and selects an optical illusion UI to be viewed by the user.

The eye tracking method of the user can utilize various known eye tracking techniques. At this time, in a case where the command input device 100 of the present invention is applied to a user who uses a desktop computer, a notebook computer, a digital TV, or the like, and in a case where it is applied to a user using an HMD or HWD, And the miniaturization of the components for the light source, etc.).

For example, a camera may be used to track a user's gaze, and a camera may capture the movement of an eyeball to determine the point of view of the user. Of course, it will be obvious to those skilled in the art that a variety of sensors other than a camera can be used to track the eyes.

A plurality of illusionary UIs may be provided on the screen that the user views, and the eye-eye tracking unit determines whether or not the user gazes at a specific illusionary UI among a plurality of illusionary UIs. The user can look at the play button of the media player, the close button of the browser of the browser, or the mouse cursor to see what kind of UI the user is looking at.

On the other hand, if the gaze information measuring unit 120 determines that the gaze of the user grasped by the gaze tracking unit 110 is gazing at the specific illusion UI displayed on the screen, the gaze information measuring unit 120 may measure the gaze information by sensing the eye movement of the user .

To this end, the gaze information measuring unit 120 may use a camera or an electro-oculogram (EOG) as an embodiment for measuring eye movements.

Herein, the 'ocular potential sensor' may be disposed around both eyes of the user, and a signal (hereinafter referred to as 'ocular potential signal') generated by electrical activity of muscles around the eye when the user moves his or her eye Respectively.

For reference, when the gaze information measuring unit 120 is applied to the HMD or the HWD, at least one of the ocular potential sensor and the camera may be installed around the eyes of the HMD or the HWD.

Hereinafter, eye movements measured by the gaze information measuring unit will be described. Eye movements that occur when gazing at the type of the illusion UI and the illusion UI are described with reference to FIG.

First, an optical image before the optical illusion UI will be briefly described. An optical illusion image is an image in which an object displayed in an image can be recognized as various types of objects, as shown in FIG. That is, the optical illusion image refers to an image in which the object appears in a different form according to the viewing point, and FIGS. 2 (a) to 2 (d) show an example of the optical illusion image. In the present invention, an optical illusion image as shown in Figs. 2 (a) to 2 (d) is displayed in place of the existing interface button.

For example, (a) is a perspective reversal optical image, showing a wolf image and a necker cube.

In the case of a wolf image, it may be seen as a wolf looking to the right or a wolf looking to the left depending on the viewpoint.

Also, in the case of a necker cube, the cube appears to protrude at 4 ~ 5 o'clock or protrude at 10 ~ 11 o'clock depending on the viewpoint.

Also, (b) is a content reversal optical illusion image, which may be seen as a duck with black eyes depending on the viewpoint, and a rabbit with a black eye (mouth of a duck is seen as a rabbit's ear).

Also, (c) is a figure-ground reversal optical image, which may be seen as a skeleton depending on the viewpoint, a glass on the table, and a man and a woman holding hands. And the other optical illusion image can be seen as the profile of two people facing each other on a black background, or as a black cup placed on a white background.

(d) is an embodiment in which an illusion UI using a necker cube is implemented as a toggle button in the optical illusion image described above, and may be implemented in a first state indicating ON and a second state indicating OFF. Hereinafter, the case of the optical illusion button as an embodiment of the illusionary UI will be described.

For reference, an optical illusion button using a necker cube can be used as a different optical illusion button by changing the color of the necker cube or changing the rotation angle in the basic state.

As described above, since the objects displayed in one image can be recognized as different objects according to the viewpoint, the user can focus more on the image to look at the image, A special eye movement occurs at a point in time when the object is recognized as another object.

For example, when looking at the necker cube shown in FIG. 2 (a) and FIG. 2 (d), initially protruding in the direction of 4 to 5 o'clock, A special eye movement occurs.

Accordingly, it is possible to distinguish between a case where the user looks at an unattended button without a purpose and a case where a user gazes carefully for selection by using a special eye movement generated when the user looks at an illusion button, It is possible to perform various operations only by grasping.

The control command execution unit 130 may execute a control command corresponding to the optical illusion button when the eye movement measured by the gaze information measurement unit 120 corresponds to a predetermined range of values.

Hereinafter, when the user gazes at the optical illusion button and the change of the eye movement and the pupil size corresponds to the value of the specific range, the user is referred to as 'conversion' for convenience of explanation. , It is determined that there is an execution command for the optical illusion UI currently being examined.

In one embodiment, when an illusion button is implemented as a toggle button, the control command execution unit 130 generates a control command corresponding to an ON (first state) and OFF (second state) A control command can be executed. Of course, the first state may be OFF and the second state may be ON.

In another embodiment, when the UI is implemented as a simple button rather than as a toggle button, the control command execution unit 130 may execute the same control command each time the user performs the recognition change for the UI.

For example, when a user looks at a specific key of a virtual keyboard, the character of the key can be displayed once on the screen.

In addition, the control command execution unit 130 can execute a control command corresponding to the drag or block setting using a plurality of illusion UIs.

In one embodiment, an illusion button is displayed at a start point and an end point of a sentence or a paragraph of a document, and an illusion button displayed at a start point of a specific paragraph is looked at and an illusion button displayed at an end point of the paragraph or another paragraph , The control command execution unit 130 may set a block in the corresponding section from the illusion button at the start point of the user to the illusion button at the end point.

On the other hand, when the control command corresponding to the first state of the eyesight button is executed in the control command execution unit 130, the eyesight UI state change unit 140 changes the first state of the eyesight button to the second state, Can be displayed.

As an embodiment, the UI UI changing unit 140 may change the corresponding optical illusion button to the OFF state and display it on the screen when the control command when the optical illusion button is selected in the ON state is executed.

When the user desires to stop the current operation by selecting the corresponding optical illusion button, the user may retake the corresponding optical illusion button displayed in the OFF state so that the control command when the illusion button is selected in the OFF state can be executed. At this time, the state of the optical illusion button can be changed to the ON state and displayed.

As another embodiment, the illus- tioned UI state changing section 140 may display in the first state and the second state whether the illusion button is selected or not selected.

For example, when a virtual keyboard is implemented using an illusion button, if the user gazes at a specific key and the corresponding key is selected, the UI UI change unit 140 changes the key from the first state (unselected) to the second state (Selectable display). When another key is selected, the first key selected can be displayed from the second state to the first state.

Of course, it is not essential that the UI state change is performed, and only the control command corresponding to the UI UI can be performed without changing the state of the UI.

In another embodiment, the mouse cursor may be displayed as an illusion UI, and a mouse click command may be executed when the user performs a changeover while looking at a mouse cursor implemented in the illusionary UI.

Hereinafter, a method of detecting the cognitive change in the test information measuring unit 120 will be described. As described above, special eye movements occur when the user recognizes the user's eyes and switches the eyesight.

Special eye movements that occur when cognitive conversion occurs include changes in gaze fixation points due to saccadic movement in which the gaze fixation point changes instantaneously, changes in the focal distance of the eye (vergence), and change in pupil size do.

When the eye-switching UI is performed while looking at the UI like the Necker cube, the eye-gazing point changes momentarily, unlike the normal eye movement. This change takes place between 20 ms and 40 ms.

The gaze information measuring unit 120 determines whether or not a saccadic movement occurs by using various sensors such as a camera or an ocular potential sensor. The gaze information measuring unit 120 determines whether a gaze fixation point is changed over a critical range within a predetermined threshold time .

In addition, the gaze information measuring unit 120 detects a change in the focal distance of the eye, which can be detected by measuring the degree of convergence of the two eyes. The display that the user is viewing is flat, so if the display is still being viewed, the change in focus distance will not occur except in exceptional cases. However, if a change of eyes occurs while looking at an illusion UI such as a Necker cube, a change in focal distance instantaneously occurs (instantaneous change in degree of convergence of two eyes) It is determined whether or not a change of the focal distance occurs.

In addition, the gaze information measuring unit 120 detects whether or not a change in the pupil size occurs. When the cognitive conversion is performed, a change in the pupil size occurs instantaneously, and the change in size is detected by using a camera or the like.

When at least one of a gaze jump, a focal length change, and a pupil size change is detected by the gaze information measurement unit 120, the control command execution unit 130 determines that the recognition change has been made and executes the UI currently viewed by the user .

The control unit 150 controls the components of the command input device 100 such as the gaze tracking unit 110, the gaze information measurement unit 120, the control instruction execution unit 130, 140 may perform the above-described operation, and the storage unit 160 may also be controlled.

Meanwhile, the storage unit 160 may store various data required or derived in the course of controlling according to the algorithm and the algorithm for allowing the controller 150 to control the components of the command input device 100.

3 is a flowchart illustrating a process of receiving an instruction using an illusion UI according to an exemplary embodiment of the present invention.

3 can be performed by the command input device 100, and a user interface including an illusion button is displayed on a screen viewed by the user.

First, the command input device 100 tracks the line of sight of the user looking at the screen (S301).

After step S301, the command input device 100 confirms whether the user's gaze looks at a specific illusion UI among a plurality of illusion UIs displayed on the screen (S302).

As a result of the determination in step S302, the command input device 100 detects a user's eye movement when the specific illusion UI is examined (S303).

After step S303, the command input device 100 determines whether or not the sensed eye movement is a movement performed when the recognition change is made. If the determination is affirmative, the command input device 100 determines that recognition change has been made to the illusion UI (S304). As described above, if at least one of the change of the gaze fixation point, the change of the focal length, and the change of the pupil size due to the visual gaze occurs, it is determined that the change has been made.

After the control command in S304 is executed, the command input device 100 changes the state of the illusive UI from the first state to the second state and displays it on the screen (S305).

The above-described technical features may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium.

The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination.

The program instructions recorded on the medium may be those specially designed and constructed for the embodiments or may be available to those skilled in the art of computer software.

Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; Magneto-optical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like.

Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like.

The hardware device may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

It will be understood by those skilled in the art that the foregoing description of the present invention is for illustrative purposes only and that those of ordinary skill in the art can readily understand that various changes and modifications may be made without departing from the spirit or essential characteristics of the present invention. will be.

It is therefore to be understood that the above-described embodiments are illustrative in all aspects and not restrictive.

For example, each component described as a single entity may be distributed and implemented, and components described as being distributed may also be implemented in a combined form.

The scope of the present invention is defined by the appended claims, and all changes or modifications derived from the meaning and scope of the claims and their equivalents should be construed as being included within the scope of the present invention.

100: a device (command input device) for receiving a command by using an ill-
110: eye tracking unit
120: Examination information measuring section
130: Control command execution unit
140: UI state changing section
150:

Claims (15)

An apparatus for inputting a command using a user interface (hereinafter referred to as " illusion UI ") using an image causing an illusion,
A line of sight tracking unit for tracking a line of sight of a user viewing the screen;
A gaze information measuring unit for measuring an eye movement of the user when the gaze of the user gazes at a specific illusion UI displayed on the screen; And
When the measured eye movement is determined to be an eye movement corresponding to the cognitive switch, a control command execution unit
And an input unit for inputting an instruction using the optical illusion UI.
The method according to claim 1,
Wherein the eye movement corresponding to the cognitive conversion includes at least one of a gaze jump instantaneously converted by the gaze fixation point, a change of the focal distance of the eye, and a change of the pupil size.
The method according to claim 1,
When the control command corresponding to the illusion UI is executed, changes the first state of the illusionary UI to the second state and displays it on the screen,
Further comprising:
Wherein the first state and the second state of the illusion UI correspond to different control commands, respectively.
The method of claim 3,
Wherein the disguise UI includes a button and whether or not the button is operated is displayed in the first state and the second state.
5. The method of claim 4,
Wherein the button is implemented as a necker cube and is implemented as a plurality of buttons by applying at least one of different colors and rotation angles.
The method according to claim 1,
The control command execution unit
When a plurality of illusion UIs are examined with a time difference and it is determined that the measured eye movement with respect to the gazing of each illusion UI is eye movement corresponding to the recognition change, a drag or a block is added to the area formed by the plurality of illusion UI And the control command to be input is executed.
The method according to claim 1,
Wherein the control command execution unit executes a control command to click the mouse cursor when the illusion UI is a mouse cursor and the eye movement corresponding to the recognition change is detected in the gazing information measurement unit Input device.
A method of inputting a command using a user interface (hereinafter referred to as " illusion UI ") using an image causing an illusion,
(a) tracking the user's gaze on the screen
(b) measuring the eye movement of the user when the user's gaze gazes at the specific optical illusion UI displayed on the screen; And
(c) executing a control command corresponding to the illusion UI when it is determined that the measured eye movement is an eye movement corresponding to a cognitive conversion
And inputting a command for inputting a command using the optical illusion UI.
9. The method of claim 8,
Wherein the eye movement corresponding to the cognitive conversion includes at least one of a gaze jump instantaneously converted by the gaze fixation point, a change of the focal distance of the eye, and a change of the pupil size.
9. The method of claim 8,
(d) changing a first state of the illusion UI to a second state and displaying it on the screen when a control command corresponding to the illusion UI is executed
Further comprising:
Wherein the first state and the second state of the illusion UI correspond to different control commands, respectively.
11. The method of claim 10,
Wherein the illusion UI comprises a button and whether or not the button is operated is displayed in the first state and the second state.
12. The method of claim 11,
Wherein the button is implemented as a necker cube and is implemented as a plurality of buttons by applying one or more of different colors and rotation angles.
9. The method of claim 8,
A plurality of illusion UIs are examined with a time difference, and if the measured eye movement for each eye UI is in a range of a predetermined threshold value, a drag or a block is assigned to the area formed by the plurality of illusion UIs A step of executing a control command
And inputting a command for inputting the command using the optical illusion UI.
9. The method of claim 8,
Wherein the step (c) executes a control command to click the mouse cursor when the eyesight UI is a mouse cursor and eye movement corresponding to a cognitive switch is detected in the step (b) Command input method.
A computer program stored in a recording medium comprising a series of instructions for performing the method according to any one of claims 8 to 13.

Images