KR102083372B1 - System for alternative computer access using eye-tracking and voice recognition and operating method having the same - Google Patents

Abstract

The present invention relates to a computer alternative access system using eye tracking and speech recognition and a method of operating the same.
The present invention includes a user terminal for moving the mouse cursor on the screen by tracking the movement of the eyeball of the user, and an external input device for transmitting a signal input from the user to the user terminal, the user terminal, the user terminal A camera unit for photographing the pupil image of the user, an eye recognition unit for acquiring gaze coordinates by recognizing the eye through the edge detection from the pupil image, and obtaining gaze point coordinates from the recognized eye The gaze point processor that performs point filtering and the mouse cursor move according to the result of the gaze point filtering, and when the distance between the current gaze point coordinates and the previous gaze point coordinates is greater than a predetermined distance, the mouse cursor is moved. It includes a mouse cursor moving unit for moving.

Description

SYSTEM FOR ALTERNATIVE COMPUTER ACCESS USING EYE-TRACKING AND VOICE RECOGNITION AND OPERATING METHOD HAVING THE SAME}

The present invention relates to a computer alternative access system using eye tracking and voice recognition, and more particularly, a computer alternative approach for a physically handicapped person who moves a mouse cursor through eye tracking and transmits an input signal using voice recognition and an external device. It is about the system.

Computer-accessible assistive devices are assistive devices that help people with disabilities use computers more easily. Recently, with the development of information and communication technology, the need for participation of socially disabled people is increasing.

On the other hand, most of these devices provide access to the computer using the foot, head, etc. Due to the characteristics of the physically handicapped, it is not easy to control the mouse using the foot, head, etc., and there are limitations in speed and accuracy. In order to improve such a problem, eye tracking technology for approaching a mouse by recognizing blinking eyes or looking at a point like a certain time has been proposed, but there is a problem that the recognition rate is low and it is inconvenient to use. Prior arts include Korean Patent No. 10-1501165 (Eye Mouse Using Eye Tracking), but it merely discloses contents of gradation conversion of color image data acquired from an infrared camera and eye recognition through edge detection.

The problem to be solved by the present invention is to provide a computer alternative access system utilizing eye tracking and speech recognition implemented to solve the above problems to make it easier for people with disabilities to use a computer.

The present invention for achieving the above object includes a user terminal for moving the mouse cursor on the screen by tracking the movement of the eyeball of the user, and an external input device for transmitting a signal input from the user to the user terminal, The user terminal is disposed in front of the user terminal, the camera unit for capturing the eye image of the user, the eye recognition unit for acquiring gaze coordinates by recognizing the eye through the detection of the outline from the eye image, and from the recognized eye A gaze point processing unit which acquires gaze point coordinates and performs gaze point filtering, and moves the mouse cursor according to the gaze point filtering result, and compares the present gaze point coordinates with previous gaze point coordinates to determine a predetermined distance. It includes a mouse cursor moving unit for moving the mouse cursor when the difference is abnormal.

The present invention through the above problem solving means moves the mouse cursor through the eye tracking, and transfer the input signal by using the voice recognition and external devices can be easily disabled people using the computer.

In addition, gaze-point filtering is performed to prevent mouse shaking, and when an external input button is selected through a hooking technology, left-clicking of the mouse operates to operate quickly and easily.

1 is a block diagram of a computer alternative access system using eye tracking and voice recognition according to an embodiment of the present invention.
2 is an operation method of a computer alternative access system using eye tracking and voice recognition according to an embodiment of the present invention.
3 to 7 are examples of the operation of the computer alternative access system using eye tracking and voice recognition according to an embodiment of the present invention.
8 to 9 are comparative examples of a computer alternative access system using eye tracking and voice recognition according to an embodiment of the present invention and the prior art.

Specific structural or functional descriptions of the embodiments according to the inventive concept disclosed herein are provided only for the purpose of describing the embodiments according to the inventive concept. It may be embodied in various forms and is not limited to the embodiments described herein.

Embodiments according to the inventive concept may be variously modified and have various forms, so embodiments are illustrated in the drawings and described in detail herein. However, this is not intended to limit the embodiments in accordance with the concept of the present invention to the specific forms disclosed, it includes all changes, equivalents, or substitutes included in the spirit and scope of the present invention.

The terminology used herein is for the purpose of describing particular example embodiments only and is not intended to be limiting of the invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this specification, terms such as "comprise" or "have" are intended to indicate that there is a feature, number, step, action, component, part, or combination thereof described herein, but one or more other features. It is to be understood that the present invention does not exclude the possibility of the presence or the addition of numbers, steps, operations, components, parts, or combinations thereof.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram of a computer alternative access system using eye tracking and voice recognition according to an embodiment of the present invention.

Referring to FIG. 1, the computer alternative access system 10 includes a user terminal 100 and an external input device 200.

The user terminal 100 includes a camera unit 110, an eyeball recognition unit 120, a gaze point processing unit 130, a mouse cursor moving unit 140, a control unit 150, a display unit 160, a text input unit 170, and a voice. The input unit 180 and the communication unit 190 are included. The user terminal 100 in the specification of the present invention is a computing device, such as a smart phone, tablet PC can run a processor determined by the operating system software and various application software, a mobile computing device that is easy to carry and can be easily used on the go However, the present invention is not limited thereto, and a desktop PC and a laptop PC can also be applied.

The camera unit 110 is disposed in front of the user terminal 100 to photograph the pupil image of the user. The camera unit 110 may track the position by detecting the movement of the pupil, and uses at least one of a video analysis method, a contact lens method, and a sensor attachment method. The video analysis method detects pupil movement through analysis of a real-time camera image and calculates the direction of the eye based on a fixed position reflected by the cornea. The contact lens method uses reflected light of a mirror-embedded contact lens or a magnetic field of a coil-embedded contact lens. The sensor attachment method uses an electric field according to the movement of the eyes by attaching the sensor around the eyes, and the movement of the eyes can be detected even when the eyes are closed.

The eye recognition unit 120 may recognize the eyeball of the user through the detection of the outline from the pupil image photographed by the camera unit 110, thereby obtaining the coordinates gaze at the eyeball.

The gaze point processor 130 obtains gaze point coordinates from the recognized eye and performs gaze point filtering. The gaze point processor 130 may minimize the shaking of the mouse cursor due to the gaze of the gaze point through the gaze point filtering process. The gaze-point filtering compares the current gaze point and the previous gaze point and moves the mouse cursor to the current gaze point only when the distance difference is 50 pixels or more, and does not move the mouse cursor if it is 50 pixels or less.

The mouse cursor moving unit 140 obtains the coordinates of the gaze point from the eye recognition unit 120 and then moves the mouse cursor after filtering. When the position of the mouse cursor is different from the position of the gaze point, the user may match the position of the mouse cursor and the gaze point through gaze point correction. For example, if the position of the mouse coordinate is located to the right of the position of the gaze point, the user presses a predetermined specific key. At this time, you can hook the message of a specific key and move the position of the mouse cursor to the left by 5 pixels.

The controller 150 controls the processing of the process related to the execution of the software, and controls the operation of each component of the commercial terminal 100. The controller 150 may control processing of processes related to execution using a web browser through the world wide web as well as software.

The display unit 160 may display the movement of the mouse cursor on the software GUI and provide various contents.

The text input unit 170 receives an input event executed by the input means. The text input unit 170 may be implemented as an image keyboard and may input text of a consonant and vowel combination based on the coordinates received from the eye recognition unit 120. In addition, the text combination may be generated by the input signal received from the external input device 200.

The voice input unit 180 may convert the text to text based on the voice of the user received through voice recognition. Send flac type voice file to API Server through POST method. After that, voice is delivered as Json type data through GET method. Finally, extract only necessary text from Json format data. Use the Window API to record sound through the microphone. Then use the NAudio library and the CUETool library to format wav files and convert them to flac files. Finally, in order to parse the Json-type texts delivered through API Server, Newtonsoft library can be used to implement speech recognition.

The communication unit 190 is a communication module that plays a role of exchanging data with the external input device 200. The input signal may be received from the communication unit of the external input device 200.

The external input device 200 may be implemented as a pedal / foot switch to facilitate a physically handicapped person, and performs a mouse click function. An input signal may be transmitted to the user terminal 100 through wired or wireless communication. That is, the mouse click is generated by pressing the external input device. This operation is accomplished through a hooking technique. Hooking is a technique for intercepting a message that is generated by an event. For example, if you hook a message about pressing the 'a' key, instead of typing 'a', you'll trigger another Action. In this way, when the external button is pressed, the corresponding message is hooked to generate the left mouse button Down event. In addition, when the external button is released, the mouse is operated by hooking the corresponding message to generate the left mouse button Up event.

2 is an operation method of a computer alternative access system using eye tracking and voice recognition according to an embodiment of the present invention.

Referring to FIG. 2, in the alternative computer access system 10, the camera unit 110 of the user terminal 100 first photographs an eyeball image of a user and recognizes an eyeball (S210). The eye recognition unit 120 of the user terminal 100 extracts a gaze point at which the eye gazes (S220), and the gaze point processing unit filters the gaze point (S230). The mouse cursor moving unit 140 moves the mouse cursor or inputs text based on the gaze point coordinates. When the user receives an input signal from the external input device 200, an operation such as a mouse click is executed.

3 to 7 are examples of operations of a computer alternative access system using eye tracking and voice recognition according to an embodiment of the present invention.

3 is a first execution screen of software provided in a display unit of the user terminal 100 of the present invention. When the software is started for the first time, the main toolbar 320 having a main screen 310 and a mouse, keyboard, and the Internet button is loaded by default. When the Internet is executed through the Internet button, login, search, etc. 'Internet Toolbar' is launched.

4 shows a screen when a mouse button is selected. Another window 310b appears as shown in the right picture when the mouse is clicked in the main toolbar. Pressing the external button once performs a left click by default, and any other click can be done by clicking the functions in the right picture. The right click, double click and drag functions 320b are executed only once when clicked and the next clicked execution returns to the default left click.

5 shows a screen when the Internet button is selected. When the Internet button in the main toolbar 320d is pressed, the Internet window is automatically turned on as shown below, and the Internet toolbar 320c is turned on on the left side separately.

6 shows a screen when a keyboard button is selected. When the keyboard button 340a in the main toolbar is clicked, the keyboard is turned on in the full screen and is composed of Korean, English, and special characters. There are two ways to use the keyboard. The first is to stare at each letter you want to enter and press the external button to enter letters. The second method is to use the voice recognition of the keyboard. Pressing the microphone button turns red, speaking the text into the microphone and clicking the button again changes the original color to green. The text input by the voice is displayed on the screen.

7 illustrates a Gaze Keeper when a keyboard button is selected. When a character is input through the image keyboard, the user should check whether the current string is well written while pressing the button through his eyes. Therefore, the user has to alternate between the input unit 353a and the output unit 351aa. In order to minimize the inconvenience, the input unit also outputs the characters being input in word units.

8 to 9 are comparative examples of a computer alternative access system using eye tracking and voice recognition according to an embodiment of the present invention and the prior art.

For comparison with the prior art, subjects are divided into four types as follows. Subject 1 without physical constraints and using a computer via a normal keyboard and mouse, hands and arm movements are free but limited in using fingers, and Subject 2 using a computer via a touch screen monitor, and hand and arm use. It is divided into subject 3 who uses computer through mouse stick, and subject 4 who uses computer through general keyboard and mouse because there is restriction of hand, arm, eye and head fixation.

8 (a) is the test result of the test subject 1, FIG. 8 (b) is the test result of the test subject 2, FIG. 9 (a) is the test result of the test subject 3, Figure 9 (b) is the test subject 4 Is the result of experiment.

Experiment 1 is the portal site login completion time. In case of Subject 1 and Subject 2, the portal site login speed is 4.42 (sec) and 2.91 (sec) faster than the proposed system. In Test Subject 3, the portal site login speed through the proposed system is 13.22 seconds faster than the alternative device. In Test Subject 4, the portal site login speed through the alternate device was 31.29 seconds faster than the proposed system.

Experiment 2 is the search speed of portal site, and in case of test subject 1, the search speed of portal site through the conventional computer approach is 6.9 (sec) faster than the proposed system. In case of Subject 2, the search speed of the portal site through the proposed computer approach and the proposed system is similar, with a difference of 0.24 seconds. In case of Subject 3, the search speed of Naver portal site through the proposed system is 8.99 (seconds) faster than the alternative device. In case of test subject 4, the pronunciation of speech was so great that the speech recognition did not work properly.

Experiment 3 is the video site search speed. In case of test subject 1, the video site search speed is similar to that of the conventional computer approach and the proposed system, and the difference is 0.86 (seconds). In case of Subject 2, the video site search speed through the proposed system is 3.94 seconds faster than the alternative device. In Test Subject 3, the video site search speed through the proposed system is 15.23 seconds faster than the alternative device. For test subject 4, the foot

Since the speech was so confused that speech recognition did not work properly, we could not experiment the video site search through the proposed system.

In case of test subject 1 without physical constraints, three experiments were performed using the conventional computer approach and the proposed system, respectively. In all experiments, the conventional computer approach took less time than the proposed system. Based on these results, Subject 1 can determine that it is appropriate to use a computer in the conventional computer approach.

Hand and arm movements were free, but subject 2, who had limitations in using fingers, performed three experiments with the conventional computer approach and the proposed system, respectively. In Experiment 1, the conventional computer approach is faster than the proposed system, but in Experiments 2 and 3, the proposed system is similar or faster than the conventional computer approach. This seems to be because the search word input speed through the voice recognition of the system proposed in Experiments 2 and 3 is faster than the search word input speed through the conventional computer approach. Accordingly, in case of Test Subject 2, the mouse operation speed is faster than that of the conventional computer approach, but the proposed system is faster than the search word input speed.

Subject 3, who had limitations in using the hand and arm, performed three experiments with the conventional computer access method and the proposed system, respectively, and showed the result as shown in FIG. 9 (a). In all experiments, the proposed system took less time than the conventional computer approach. Based on these results, Subject 3 can determine that it is appropriate to use a computer as a proposed system.

Subject 4, whose hands, arms, eyes, and heads were constrained, performed three experiments with the conventional computer approach and the proposed system, respectively. In Experiment 1, the proposed system takes more time than the conventional computer approach. In Experiments 2 and 3, the proposed system cannot be measured. In light of this, it is judged that computer use through the system proposed to Subject 4 is not suitable. Based on the above experimental results, it can be seen that the proposed system is a computer alternative access system suitable for the subject 3.

Although the present invention has been described with reference to the embodiments shown in the drawings, this is merely exemplary, and those skilled in the art will understand that various modifications and equivalent other embodiments are possible therefrom. Therefore, the true technical protection scope of the present invention will be defined by the technical spirit of the appended claims.

10; Computer alternative access system 100; User terminal
110; A camera unit 120; Eye Recognition Department
130; Gaze point processing unit 140; Mouse cursor moving part
150; A controller 160; Display
170; Text input unit 180; Voice input section
190; Communication unit 200; Input device

Claims (4)

In computer alternative access system using eye tracking and voice recognition,
A user terminal for tracking the movement of the eyeball of the user and moving the mouse cursor on the screen, and an external input device for transmitting a signal input from the user to the user terminal,
The user terminal,
A camera unit disposed at the front of the user terminal to photograph an eye image of the user;
An eye recognition unit configured to acquire eye gaze coordinates by recognizing the eye by detecting an outline from the eye image;
A gaze point processing unit which obtains gaze point coordinates from the recognized eye and performs gaze point filtering; And
And a mouse cursor moving unit for moving a mouse cursor according to a result of performing gaze point filtering, and moving the mouse cursor when a predetermined distance difference is greater than a current gaze point coordinate and a previous gaze point coordinate.
The mouse cursor moving unit receives a preset specific key from the user when the position of the mouse cursor is located to the right of the gaze point coordinates, hooks a message of the specific key, and moves the position of the mouse cursor by 5 pixels to the left,
The external input device is implemented as a pedal / foot switch to facilitate a physically handicapped person, and when the external input device is pressed, a corresponding message is hooked to generate a left mouse button Down event, and when the external input device is released, the corresponding message is released. A computer alternative access system that utilizes eye tracking and voice recognition that mouse clicks are triggered by generating a left mouse button UP event by hooking the mouse. The method of claim 1,
The gaze-point filtering compares the current gaze point and the previous gaze point and moves the mouse cursor to the current gaze point only when the distance difference is 50 pixels or more, and when it is 50 pixels or less, eye tracking and voice that do not move the mouse cursor. Computer Alternative Approach System Using Recognition. delete In the operation method of computer alternative access system using eye tracking and voice recognition,
(a) a camera unit photographing an eyeball image of a user to recognize eyeballs;
(b) acquiring gaze point coordinates by recognizing the eye through the detection of an outline from the pupil image, and performing gaze point filtering on the obtained gaze point
(c) moving the mouse cursor or inputting text based on the gaze point coordinates;
(d) executing an operation based on an input signal received from an external input device,
In the step (c), if the position of the mouse cursor is located to the right of the gaze point coordinate, the user inputs a predetermined specific key from the user, hooks a message of the specific key, and moves the position of the mouse cursor to the left by 5 pixels.
The external input device is implemented as a pedal / foot switch to facilitate a physically handicapped person, and when the external input device is pressed, a corresponding message is hooked to generate a left mouse button Down event, and when the external input device is released, the corresponding message is released. Operation method of computer alternative access system using eye tracking and voice recognition that mouse click is operated by hooking and generating UP event of left mouse button.

Images