KR101272811B1 - System and method for interface - Google Patents

Abstract

An interface system and method are disclosed. The interface system includes a display device equipped with an infrared light, a camera for acquiring an eye image of an eye irradiated with infrared light, an electroencephalogram device for acquiring brain waves through an EEG sensor and calculating brain wave information, and an eye image. And a terminal for tracking the position, performing a command set in an area corresponding to the gaze position on the output screen of the display device, and controlling the output screen by performing a command corresponding to the EEG information.

Description

Interface system and method {System and method for interface}

The present invention relates to an interface in a terminal having a display device.

BACKGROUND OF THE INVENTION Background of the Invention There is a patent document on evaluating consumer experiences by judging neuro-response measurements for consumers exposed to products, services, offers and stimuli. Examples of neuro-response measurements include brain waves, eye tracking, etc. It is disclosed.

KR 2010-0047865 A 2010. 5. 10

The present invention is to control the output screen of the terminal by using the gaze position information and brain wave information of the user in place of an input device such as a keyboard or a mouse.

According to an aspect of the present invention, an interface system using eye tracking and brain waves is disclosed.

An interface system according to an embodiment of the present invention is a display device equipped with an infrared light, a camera for acquiring an eye image of the eye irradiated by the infrared light, and an EEG analysis for obtaining EEG information by obtaining an EEG through an EEG sensor The line of sight is tracked using the device and the eye image, the command is set in an area corresponding to the line of sight on the output screen of the display device, and the command is performed to control the output screen. It includes a terminal to be.

The terminal may include a first receiver configured to receive the eye image, a pupil part detected from the eye image, a determiner configured to determine a pupil center in the pupil area, and the pupil center based on a preset conversion function. A calculation unit configured to calculate the gaze position by mapping to a display plane of a display device, a second receiver configured to receive the EEG information including attention, and a movement set in the gaze position region corresponding to the gaze position on the output screen And a control unit for moving the output screen in a direction and controlling a moving speed of the output screen according to the concentration level.

Here, the calculating unit calculates the number of black pixels in the pupil area, and the controller detects eye blink according to the change of the number of black pixels, and executes a command corresponding to the eye blink.

Here, when the eye blink is detected, the controller moves or stops the output screen.

Here, when the eye blink is detected, the output screen is enlarged or stopped.

Here, the EEG analysis device is formed in the form of a headset (Headset) that can be worn on the head, and is equipped with an EEG sensor for receiving EEG through the left frontal lobe of the user.

Here, the camera is attached to any one of the wearable device of glasses, safety glasses, helmet type device.

According to another aspect of the present invention, an interface method in an interface system having a display device is disclosed.

According to an embodiment of the present invention, an interface method includes outputting a screen to the display device, tracking a gaze position using an eye image acquired through a camera, and moving the screen in a movement direction set in an area corresponding to the gaze position. Moving the screen; calculating brain wave information using brain waves acquired through an brain wave sensor; and controlling a moving speed of the screen according to the brain wave information.

Here, the brain wave information includes a concentration degree, and the controlling of the movement speed of the screen according to the brain wave information is controlling the movement speed of the screen according to the concentration degree.

The moving of the screen may include detecting an eye blink using the eye image and moving the screen in a moving direction set in an area corresponding to the gaze position when the eye blink is detected. .

In the detecting of eye blink using the eye image, calculating the number of black pixels of the pupil area in the eye image and determining that the eyes are closed when the number of black pixels of the pupil area decreases below a preset threshold. And when the number of black pixels exceeds a threshold within a preset time, determining that the eyes are left again and detecting the blinking of the eyes.

Here, after the step of controlling the moving speed of the screen, if blinking of the eye is detected, stopping the movement of the screen, the gaze position corresponds to the area where the stop command is set, if the blinking of the eye is detected, And enlarging the screen, and stopping the screen enlargement when the eye blink is detected during the enlargement of the screen.

The tracking of the gaze position may include receiving the eye image, detecting a pupil area from the eye image, determining a pupil center in the pupil area, and using the preset conversion function. Calculating the gaze position by mapping to a display plane of the display device.

Here, the screen output to the display device is a map screen or a three-dimensional virtual space screen.

The present invention can control the output screen of the terminal by using the gaze position information and brain wave information of the user in place of an input device such as a keyboard or a mouse.

In addition, the present invention can be utilized as an interface in an electronic device having a two-dimensional or three-dimensional display device, an electronic device interface to a severely disabled person or a brain disease patient through an interface using gaze position information and brain wave information.

1 is a configuration diagram schematically illustrating an interface system.
2 is a block diagram schematically illustrating a configuration of a terminal.
3 illustrates a method of determining a pupil center.
4 is a diagram illustrating a geometric transformation method for performing user calibration.
5 illustrates the calculation of the number of black pixels for the pupil area.
6 is a graph showing EEG information.
7 is a flowchart illustrating an interface method performed in an interface system.
8 and 9 are diagrams showing examples of applying an interface method.

The present invention may be variously modified and have various embodiments, and specific embodiments will be illustrated in the drawings and described in detail with reference to the accompanying drawings. It should be understood, however, that the invention is not intended to be limited to the particular embodiments, but includes all modifications, equivalents, and alternatives falling within the spirit and scope of the invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, the present invention will be described in detail with reference to the accompanying drawings. In addition, numerals (e.g., first, second, etc.) used in the description of the present invention are merely an identifier for distinguishing one component from another.

Also, in this specification, when an element is referred to as being "connected" or "connected" with another element, the element may be directly connected or directly connected to the other element, It should be understood that, unless an opposite description is present, it may be connected or connected via another element in the middle.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the following description of the present invention, the same reference numerals will be used for the same means regardless of the reference numerals in order to facilitate the overall understanding.

1 is a configuration diagram schematically illustrating an interface system.

Referring to FIG. 1, the interface system includes a terminal 100, a display device 200 including an infrared light 210, an EEG analysis device 300 including an EEG sensor 310, and a camera 400. .

The EEG analyzing apparatus 300 obtains EEG using the EEG sensor 310, and analyzes EEG to calculate EEG information. Here, the EEG information may include attention information and stability information. For example, when the user wearing the EEG analyzing apparatus 300 consciously concentrates, concentration may increase.

For example, the EEG analyzing apparatus 300 may be formed in the form of a headset that can be worn on the head, and the EEG sensor 310 for receiving EEG through the left frontal lobe of the user is shown in FIG. 1. It may be provided as. Here, the brain wave sensor 310 may be applied to one dry active sensor.

The display apparatus 200 is connected to the terminal 100 and outputs image information received from the terminal 100. For example, the terminal 100 may be a desktop computer main body, and the display apparatus 200 may be a monitor connected to the desktop computer main body.

In addition, as shown in FIG. 1, the display apparatus 200 may be equipped with infrared light 210 at each of four corners. Here, the infrared light 210 is composed of any one of an infrared light emitting diode (LED), a halogen, a xenon lamp, and an incandescent lamp to perform infrared light illumination on the user's eyes.

The camera 400 captures an image of an eye of a user. Here, the eye image includes illumination reflected light illuminated from the infrared light 210. That is, the infrared light 210 mounted on the display apparatus 200 irradiates infrared rays to the user's eyes, so that the camera 400 may capture an eye image in which the boundary between the pupil and the iris is divided.

For example, the camera 400 may be attached to a wearable device such as glasses, safety glasses, a helmet type device, and may acquire an eye image of the user while the user wears the wearable device. In addition, the camera 400 includes an infrared transmission filter, a lens, and an image sensor, and an infrared transmission filter is attached to the front of the lens or the front of the image sensor to acquire an eye image of the eye irradiated with infrared rays. The image sensor may be configured as a charge coupled device (CCD) or a complementary metal-oxide semiconductor (CMOS).

The terminal 100 receives the EEG information from the EEG analysis device 300, receives an eye image from the camera 400, and controls a screen output to the display apparatus 200 using the received EEG information and the eye image. do. This will be described later in detail with reference to FIGS. 2 to 6.

2 is a block diagram schematically illustrating a configuration of a terminal.

Referring to FIG. 2, the terminal 100 includes a first receiver 10, a detector 20, a calculator 30, a second receiver 40, and a controller 50.

The first receiver 10 receives an eye image of the user from the camera 400. Here, the camera 400 acquires an eye image including the illumination reflection light 34 generated by the infrared light 210 illuminating infrared light to the eye.

The determination unit 20 detects the pupil area from the received eye image and determines the pupil center in the detected pupil area.

3 is a diagram illustrating a method of determining a pupil center. Hereinafter, a method of determining the pupil center by the determination unit 20 will be described in detail with reference to FIG. 3.

Referring to FIG. 3, as shown in (a), the determination unit 20 may determine a circular edge detection template composed of an inner circle 32 and an outer circle 31 according to a circular detection algorithm. Move in the eye image. At this time, the determination unit 20 calculates a difference between the gray level sums of the inner circle 32 and the outer circle 31 of the circular detection template. Then, the determination unit 20 detects the pupil area as in (b) by detecting the area where the difference in gray level sum is greatest.

However, the shape of the pupil included in the eye image has an irregular shape to simplify in the form of a circle, and may appear as an ellipse rather than a circle according to the gaze position and the camera shooting angle, and thus the pupil detected by the circular detection algorithm. If the pupil center is determined using an area, the determined pupil center may be inaccurate. In addition, when the illumination reflection light 34 is located at the boundary of the pupil, the pupil center cannot be accurately determined.

Therefore, the determination unit 20 performs local binarization on the basis of the detected pupil area as in (c).

In the binarization method, a pixel value of an image is set to 0 or 255 in order to obtain information (eg, position, size, shape, etc.) of a desired object in the image. Binarization is divided into global binarization and local binarization. The global binarization method is a method of performing binarization on a pixel of an entire image using one threshold value, and the local binarization method divides an image into several areas and performs binarization using different threshold values for each area. It is a way.

That is, the determination unit 20 forms a region (square region) 33 in a preset range from the center of the detected pupil region to perform local binarization. Since the blind area is classified into two types, a pupil area (object) and a non-pupil area (background), the determination unit 20 may make the pupil area black and the non-pupil area white.

Thereafter, the determination unit 20 performs a morphology operation to fill the empty area generated by the illumination reflection light 34 in the pupil area. (d) shows the result of the morphology operation.

Next, the determination unit 20 calculates the center of gravity (+) of the black region (ie, the pupil region generated through local binarization and morphology calculation), as shown in (e). Thus, the determination unit 20 finally determines the center of gravity of the black region as the pupil center of the pupil region. Here, the determination unit 20 may calculate the pupil center coordinates of the determined pupil center.

The calculator 30 may calculate the gaze position information of the user with respect to the display apparatus 200 by mapping the determined pupil center to the display plane of the display apparatus 200 using a predetermined conversion function. Here, the preset conversion function may be calculated through a user calibration process performed at system initialization. The transform function may be one of transform functions such as linear interpolation, geometric transform, or cross ratio.

For example, FIG. 4 is a diagram illustrating a geometric transformation method for performing user calibration. 4, the calculating section 30 defines a transition area (S ₁₎ of the pupil by a user calibration process staring at the center of the screen and moving area of the cavity (S ₁₎ and the display plane (S ₂₎ The relationship between can be represented using geometric transformations.

That is, the determination unit 20 receives an eye image when the user gazes at the center of the display apparatus 200 to determine the center of the pupil, and determines the pupil center by the four infrared lights 210 mounted at four corners. The display plane S ₂ is identified through the four illumination reflections. Thus, the calculator 30 may calculate the coordinates of the display plane S ₂ corresponding to the pupil center coordinates by using the geometric transformation function.

First, the calculator 30 defines the movement area S ₁ of the user pupil by using the pupil center coordinates when the user gazes at the center of the display apparatus 200. Here, the coordinates of the movement area S ₁ of the user pupil are the coordinates of the four illumination reflected light, and the upper left coordinates (C _x1 , C _y1 ), the upper right coordinates (C _x2 , C _y2 ), and the lower left coordinates (C _x4 , C _y4 ), and lower right coordinates (C _x3 , C _y3 ). Subsequently, the calculator 30 uses the coordinates of the movement area S ₁ of the user pupil and the coordinates of the display plane S ₂ corresponding to the coordinates of the movement area S ₁ of each pupil, that is, the geometric transformation function. , Upper left coordinates (M _x1 , M _y1 ), upper right coordinates (M _x2 , M _y2 ), lower left coordinates (M _x4 , M _y4 ), and lower right coordinates (M _x3 , M _y3 ) are calculated. Here, the coordinates of the display plane S _{2 represent} the coordinates of the display apparatus 200 that the user gazes at.

Equation 1 below represents a geometric transformation function. Referring to Equation 1 below, when the user stares at an arbitrary position of the display apparatus 200, a process of calculating the position of the display plane S ₂ corresponding to the center of the user's pupil will be described in detail.

[Equation 1]

Referring to Equation _1, S 1 will showing the coordinates of the moving region (S ₁₎ of the pupil in a matrix, S ₂ will display the coordinates of the display plane (S ₂₎ to the matrix. T is a matrix of geometric transformation functions. The calculation unit 30 calculates the matrix of the coordinates of the movement region S ₁ of the pupil and the matrix of the geometric transformation function to calculate the coordinates of the display plane S ₂ corresponding to the coordinates of the movement region S ₁ of each pupil. Calculate. The calculator 30 calculates a gaze position on the display plane S ₂ of the user using Equation 2 below.

&Quot; (2) "

Referring to Equation 2, G is a matrix representing the actual gaze position of the user, W is a matrix representing the position of the pupil extracted at the moment the user gazes. T is a matrix of geometric transformation functions. That is, when the user gazes at an arbitrary position of the display apparatus 200 using Equation 1, the calculator 30 calculates a display plane position corresponding to the center of the user pupil, and uses Equation 2 to calculate the position of the user. The actual gaze position (ie, the position of the line of interest of the user) is calculated.

In addition, the calculator 30 calculates the number of black pixels in the pupil area. That is, the calculator 30 may calculate the number of black pixels for the pupil area calculated by performing the morphology calculation in FIG. 3. For example, FIG. 5 is a diagram illustrating the calculation of the number of black pixels for the pupil area. Referring to FIG. 5, the area of the pupil area that is exposed according to the degree of occlusion of the eyelid is reduced, and thus, the black pixel of the pupil area that is calculated decreases.

The second receiver 40 receives the EEG information from the EEG analysis device 300. The received EEG information is transmitted to the controller 50.

For example, FIG. 6 is a graph showing EEG information. That is, FIG. 6 is a graph showing the magnitudes of Attention and Meditation measured over time with respect to the case where the user focuses the gaze and the case where the user does not concentrate. As shown in FIG. 6, when the user concentrates, the increase in concentration is relatively greater than the increase in stability.

The controller 50 controls the screen of the display apparatus 200 by using the gaze position information of the user transmitted from the calculator 30 and the EEG information transmitted through the second receiver 40.

That is, the controller 50 may recognize the gaze position of the user on the screen of the display apparatus 200 using the gaze position information, and may perform a command corresponding to the gaze position. For example, the screen of the display apparatus 200 may be divided into a plurality of areas in which the moving direction is set, and the controller 50 may move the screen in the moving direction of the area corresponding to the gaze position.

In addition, the controller 50 may perform a command according to the EEG information while performing a command corresponding to the gaze position. For example, the controller 50 may increase the screen moving speed if the concentration is increased while moving the screen in the moving direction of the area corresponding to the gaze position, and conversely, if the concentration is decreased, the screen moving speed may be decreased. Can be.

In addition, the controller 50 detects blinking by using the number of black pixels in the pupil area calculated by the calculator 30 and performs a corresponding command. That is, the controller 50 may determine that the eyes are closed when the calculated number of black pixels in the pupil area is less than or equal to a preset threshold, and the number of black pixels is within a predetermined time (for example, within 1 second). If exceeds, it can be determined that the eyes have re-opened to detect eye blinking.

For example, the controller 50 may recognize a gaze position, and if a blink of an eye is detected, the controller 50 may recognize it as a movement command and move the screen in a moving direction of an area corresponding to the gaze position.

In addition, the controller 50 may stop the screen movement by recognizing it as a stop command when eye blink is detected during the screen movement.

In addition, when the eye gaze position corresponds to a still area of the screen (for example, the center of the screen), and the blinking of the eye is detected, the controller 50 may recognize the zoom command to enlarge the screen and blink again during the screen enlargement. If this is detected, it can be recognized as an enlargement stop command and stop the enlargement of the screen.

7 is a flowchart illustrating an interface method performed in an interface system, and FIGS. 8 and 9 illustrate examples of applying an interface method. Hereinafter, an interface method will be described with reference to FIG. 7, but will be described with reference to FIGS. 8 and 9.

In operation S710, the interface system outputs a screen of the display apparatus 200. For example, referring to FIG. 8, the interface system may output a map on the screen. In this case, the screen may be divided into a plurality of areas in which the moving direction is preset. That is, the screen may be divided into five areas of the top, bottom, left part, right part, and center part, and each area may be set to move up, down, left, right, and stop commands.

For example, referring to FIG. 9, the interface system may output a 3D virtual city implemented in a virtual reality markup language (VRML) on a screen. In this case, the screen may be divided into a plurality of areas in which the moving direction is preset. That is, the screen may be divided into five areas of the top, bottom, left part, right part, and center part, and each area may be set to commands for moving forward, backward, moving left, moving right and stopping.

In operation S720, the interface system detects a pupil area of the user to determine the pupil center, calculates a display plane position corresponding to the pupil center, and tracks the gaze position of the user. At this time, the interface system calculates the number of black pixels in the pupil area to detect the occurrence of eye blink.

In operation S730, when the blinking of the eye is detected, the interface system recognizes this as a movement command and moves the screen in the movement direction set in the area corresponding to the gaze position on the screen.

For example, referring to FIG. 8, when the gaze position is included in the right area of the screen, the interface system moves the cursor to the position of the screen corresponding to the gaze position, and when the eye blink is detected, the interface system recognizes the movement command. To move the map screen to the right based on the line of sight.

For example, referring to FIG. 9, when the gaze position is included in the left region of the screen, the interface system may move the 3D virtual city screen to the left.

In step S740, the interface system adjusts the moving speed of the screen according to the brain waves. That is, the interface system may increase the moving speed of the screen when the concentration increases, and conversely, when the concentration decreases, the moving speed of the screen may decrease. For example, referring to FIG. 8, while the interface system moves the map screen to the right, if the concentration increases, the interface system may increase the right moving speed accordingly. In addition, referring to FIG. 9, while moving the 3D virtual city screen to the left, the interface system may increase the left moving speed accordingly if the concentration is increased.

In operation S750, when the blinking of the eye is detected, the interface system stops moving the screen. That is, if the blinking of the eye is detected during the screen movement, the interface system may recognize the stop command and stop the screen movement.

In operation S760, when the gaze position is included in the center area of the screen and the blinking of the eye is detected, the interface system recognizes this as a screen magnification command and enlarges the screen. Subsequently, if a blinking eye is detected during the screen magnification, the interface system may recognize this as a zoom stop command and stop the screen magnification.

On the other hand, the interface method according to an embodiment of the present invention can be implemented in the form of program instructions that can be executed through various electronic means for processing information can be recorded in the storage medium. The storage medium may include program instructions, data files, data structures, etc. alone or in combination.

Program instructions to be recorded on the storage medium may be those specially designed and constructed for the present invention or may be available to those skilled in the art of software. Examples of storage media include magnetic media such as hard disks, floppy disks and magnetic tape, optical media such as CD-ROMs, DVDs, and magnetic-optical media such as floppy disks. hardware devices specifically configured to store and execute program instructions such as magneto-optical media and ROM, RAM, flash memory, and the like. In addition, the above-described medium may be a transmission medium such as an optical or metal wire, a waveguide, or the like including a carrier wave for transmitting a signal specifying a program command, a data structure, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as devices for processing information electronically using an interpreter or the like, for example, a high-level language code that can be executed by a computer.

The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

It will be apparent to those skilled in the art that various modifications and variations can be made in the present invention without departing from the spirit or scope of the invention as defined in the appended claims. It will be understood that the invention may be varied and varied without departing from the scope of the invention.

100: terminal
200: display device
210: infrared light
300: EEG analysis device
310: brain wave sensor
400: camera

Claims (14)

Display units with infrared illumination;
A camera for acquiring an eye image of an eye irradiated with infrared light by the infrared light;
An EEG analyzer for acquiring EEG through an EEG sensor and calculating EEG information; And
A terminal for tracking the gaze position using the eye image, executing a command set in an area corresponding to the gaze position on an output screen of the display device, and controlling the output screen by performing a command corresponding to the brain wave information Including,
The terminal,
A first receiver configured to receive the eye image;
A determination unit for detecting a pupil area from the eye image and determining a pupil center in the pupil area;
A calculator configured to map the pupil center to a display plane of the display apparatus using a preset conversion function to calculate the gaze position;
A second receiver configured to receive the EEG information including an attention level; And
And a controller configured to move the output screen in a movement direction set in a gaze position area corresponding to the gaze position on the output screen, and to control a moving speed of the output screen according to the concentration level.
delete The method of claim 1,
The calculating unit calculates the number of black pixels in the pupil area,
The controller detects eye blink according to the change of the number of black pixels, and performs a command corresponding to the eye blink.
The method of claim 3,
The control unit
And when the eye blink is detected, moving or stopping the output screen.
The method of claim 3,
The control unit
And when the eye blinks, the output screen is enlarged or stopped.
The method of claim 1,
The EEG analysis apparatus is formed in the form of a headset (Headset) that can be worn on the head, the interface system, characterized in that for mounting the EEG sensor for receiving the EEG through the left frontal lobe of the user.
The method of claim 1,
The camera is an interface system, characterized in that attached to the wearable device of any one of glasses, goggles, helmet type device.
In the interface method in the interface system having a display device,
Outputting a screen to the display device;
Tracking the gaze position using an eye image acquired through a camera;
Moving the screen in a movement direction set in an area corresponding to the gaze position;
Calculating EEG information by using EEG acquired through EEG sensor; And
Controlling the movement speed of the picture according to the EEG information,
The EEG information includes a concentration level,
Controlling the moving speed of the screen according to the EEG information
Interface method characterized in that for controlling the moving speed of the screen according to the concentration.
delete 9. The method of claim 8,
The step of moving the screen
Detecting eye blink using the eye image; And
And when the eye blink is detected, moving the screen in a movement direction set in an area corresponding to the gaze position.
The method of claim 10,
Detecting eye blink using the eye image
Calculating the number of black pixels of the pupil area in the eye image;
Determining that the eyes are closed when the number of black pixels in the pupil area decreases below a preset threshold; And
If the number of black pixels exceeds a threshold within a preset time, determining that the eyes are left again and detecting the blinking of the eyes.
9. The method of claim 8,
After controlling the moving speed of the screen,
When the eye blink is detected, stopping the movement of the screen;
Enlarging the screen when the gaze position corresponds to a region where a stop command is set and the eye blink is detected; And
If the eye blink is detected while the screen is enlarged, stopping the screen magnification.
9. The method of claim 8,
The track that tracks the line of sight is
Receiving the eye image;
Detecting a pupil area from the eye image and determining a pupil center in the pupil area; And
And calculating the gaze position by mapping the pupil center to a display plane of the display device by using a preset conversion function.
9. The method of claim 8,
And the screen output to the display device is a map screen or a 3D virtual space screen.

Images