KR20220095752A - Automatic Scrolling Apparatus Using Gaze Tracking - Google Patents

Abstract

Disclosed is an automatic scrolling device using eye tracking. One aspect of this embodiment relates to an automatic scrolling device which automatically scrolls an output screen by analyzing a user's face image. Provided is an automatic scrolling device, which includes: a communication unit receiving a learning model for deriving a scroll control signal from a face image from the outside; a photographing unit which photographs a user's face image; and an input/output unit outputting contents; and a control unit for controlling whether to scroll an output screen output by the input/output unit by analyzing the degree of concentration on a user's screen from the user's face image captured by the photographing unit using the learning model. The present invention provides a device for scrolling the screen viewed by a user according to user's eye tracking.

Description

Automatic Scrolling Apparatus Using Gaze Tracking

The present invention relates to a device for automatically scrolling according to a user's gaze direction by tracking the gaze.

The content described in this section merely provides background information for the present embodiment and does not constitute the prior art.

In modern society, with the development of communication technology, the use of smart phones is increasing explosively, and accordingly, the spread of smart phones is also increasing.

A smartphone having a touch panel uses the touch panel as an input device. The smartphone may detect various input events input by the user on the touch panel, and may perform a function corresponding thereto.

The smartphone may display various screens (eg, application screens such as browser, E-book, gallery, etc.) displaying a large amount of pages or images, and may detect an event for controlling scrolling on various screens. . The smartphone may scroll and display the screen in response to the scroll event. Scrolling refers to an operation of moving a displayed screen vertically or horizontally.

As the usage of smartphones increases, users often operate smartphones while lying down, but operation of the smartphone while lying down is inconvenient. Although a smartphone holder has been developed to solve this inconvenience, the user may feel inconvenient because the smartphone has to be touched in order to scroll the smartphone.

An embodiment of the present invention aims to provide an apparatus for scrolling a screen viewed by a user according to the user's gaze by recognizing the user's eyeball and tracking the gaze.

According to one aspect of the present invention, in an automatic scrolling device for automatically scrolling an output screen by analyzing a user's face image, a communication unit for receiving a learning model for deriving a scroll control signal from the face image from the outside and the user Whether the output screen output by the input/output unit is scrolled by analyzing the concentration on the user's screen from the user's face image photographed by the photographing unit using the photographing unit for photographing a face image, an input/output unit for outputting content, and the learning model It provides an automatic scrolling device comprising a control unit for controlling the.

According to an aspect of the present invention, the automatic scrolling apparatus further comprises a memory unit for storing the learning model received by the communication unit.

According to an aspect of the present invention, the control unit predicts the movement of the user's pupil or the movement of each feature of the user from the face image, and analyzes the concentration of the user on the screen.

According to an aspect of the present invention, when the movement of the user's pupil is relatively increased or the movement of each feature of the user is relatively decreased, the control unit scrolls the output screen output by the input/output unit in one direction according to the learning model. characterized in that

According to one aspect of the present invention, in an automatic scrolling learning server that analyzes a user's face image and generates a learning model that enables automatic scrolling of an output screen, the face image of a human watching the screen and the human A database for storing an input for scroll control as big data, a controller for learning a human face image in the database and an input for human scroll control, and generating a learning model that receives a face image and outputs a scroll control signal; It provides an automatic scrolling learning server comprising a communication unit for transmitting the learning model generated by the control unit.

According to one aspect of the present invention, the database is characterized in that it stores human face images in each situation in which scroll control is performed.

According to one aspect of the present invention, the control unit is characterized in that it analyzes the concentration of the user's screen from the human face image in the database.

As described above, according to one aspect of the present invention, by tracking the user's gaze and scrolling the screen the user is looking at according to the user's gaze, the user can scroll as desired even if he or she looks at the screen without manipulating the screen. There is this.

1 is a diagram illustrating an automatic scrolling system according to an embodiment of the present invention.
2 is a diagram illustrating a configuration of an automatic scrolling apparatus according to an embodiment of the present invention.
3 is a diagram showing the configuration of an automatic scrolling learning server according to an embodiment of the present invention.
4 is a diagram illustrating a process in which an automatic scrolling learning server or an automatic scrolling device detects a user's eyeball from an image according to an embodiment of the present invention.
5 is a diagram illustrating a process of detecting a pupil by an automatic scrolling learning server or an automatic scrolling device according to an embodiment of the present invention.
6 is a diagram illustrating an example in which an automatic scrolling apparatus scrolls according to a user's gaze according to an embodiment of the present invention.

Since the present invention can have various changes and can have various embodiments, specific embodiments are illustrated in the drawings and described in detail. However, this is not intended to limit the present invention to specific embodiments, and it should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the present invention. In describing each figure, like reference numerals have been used for like elements.

Terms such as first, second, A, and B may be used to describe various elements, but the elements should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be called a power means, and similarly, the power means may also be called the first component. and/or includes a combination of a plurality of related listed items or any of a plurality of related listed items.

When an element is referred to as being “connected” or “connected” to another element, it is understood that it may be directly connected or connected to the other element, but other elements may exist in between. it should be On the other hand, when a certain element is referred to as being “directly connected” or “directly connected” to another element, it should be understood that no other element is present in the middle.

The terms used in the present application are only used to describe specific embodiments, and are not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly dictates otherwise. It should be understood that terms such as “comprise” or “have” in the present application do not preclude the possibility of addition or existence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification in advance. .

Unless defined otherwise, all terms used herein, including technical and scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and should not be interpreted in an ideal or excessively formal meaning unless explicitly defined in the present application. does not

In addition, each configuration, process, process or method included in each embodiment of the present invention may be shared within a range that does not technically contradict each other.

1 is a diagram illustrating an automatic scrolling system according to an embodiment of the present invention.

Referring to FIG. 1 , an automatic scrolling system 100 according to an embodiment of the present invention includes an automatic scrolling device 110 (hereinafter abbreviated as 'device') and an automatic scrolling learning server 120 (hereinafter referred to as 'server'). abbreviated).

The device 110 analyzes the concentration on the user's screen from the face image of the user looking at it, and scrolls the output screen based on the analyzed concentration.

The device 110 includes a touch panel as an input means and is implemented as a device that outputs information such as text, image, or video. An example may be a smartphone or a tablet PC, but is not necessarily limited thereto.

The device 110 captures the face image of the user who views the information output by the device 110 and analyzes the concentration of the user's screen therefrom. The device 110 has a built-in configuration for photographing an image or is connected to an external device to photograph the face image of a user who views the information output by the device 110 . The device 110 acquires a plurality of face images of the user within a preset interval, and extracts the user's facial features and intraocular pupils from within each image. The device 110 receives and stores a learning model (or algorithm) for extracting human features and pupils from the image from the server 120, and analyzes the concentration on the screen in the image using the learning model. The user's emotion or the user's concentration may be derived from the movement of the facial features or pupil. Changes in the movement or size of the user's eyes, eyebrows, pupils, cheeks or mouth are clues that reveal a person's emotions. For example, emotions of fear or sadness may be derived from movements or size changes of eyes and eyebrows, and emotions of joy or disgust (discomfort) may be derived from movements of cheeks or mouths. When the eyes and eyebrows move and the corners of the mouth rise, the device 110 can derive that the user has a feeling of joy, and when the state with the eyes closed for a certain period of time or movement of the features is minimized, the device 110 may derive that the user is feeling bored. The user's concentration on the screen is different according to the emotion he has while watching the screen (content) provided by the device 110 . That is, the device 110 extracts the movement of the features and derives the user's emotion from it, and analyzes how much the user is focused according to the derived user's emotion. Similarly, when the user concentrates on a provided screen, the pupil is still at one part of the screen or moves minutely (within a preset reference value) from one part of the screen to another. In this way, the device 110 derives the user's emotion from the movement of the features or analyzes the concentration of the user on the screen from the movement of the pupil using the learning model.

The device 110 scrolls the output screen according to the degree of concentration on the screen. After predicting the movement of the user's pupil or the overall facial expression change (degree of concentration on the screen), the device 110 automatically scrolls the output screen according to the movement of the pupil. The device 110 scrolls the output screen according to the analyzed user's gaze even if the user's input is not received through the touch panel. A detailed structure will be described later with reference to FIG. 2 .

The server 120 generates a learning model by learning the relationship between the user's concentration and scrolling based on the human face image looking at the device and the input for controlling the human scroll. The server 120 learns numerous human face images and inputs for human scroll control. The server 120 analyzes the degree of concentration (change) on the human screen by analyzing the movement of the pupil or the overall expression of the face (movement of each feature and eyebrows) in the human face image. At the same time, the server 120 analyzes the input related to the scroll control for each human device providing the face image. The input related to the scroll control includes the position of the scroll at each time point, whether the scroll is moved, and the direction in which the scroll is to be moved. The server 120 matches the above-described information for numerous pieces of data, and derives a learning model for changing the face image of a person viewing the device and controlling the scroll. That is, when a human face image is input, the server 120 derives a learning model that derives a scroll control signal as an output by analyzing the concentration on the human screen in the image. The server 120 derives such a learning model and provides it to the device 110 so that the device 110 can scroll the screen using the learning model. A detailed configuration and operation of the server 120 will be described later with reference to FIGS. 3 to 5 .

2 is a diagram illustrating a configuration of an automatic scrolling apparatus according to an embodiment of the present invention.

Referring to FIG. 2 , the device 110 according to an embodiment of the present invention includes a communication unit 210 , a photographing unit 220 , a control unit 230 , a memory unit 240 , and an input/output unit 250 .

The communication unit 210 receives a learning model for scrolling control from the user's image from the server 120 .

The photographing unit 220 captures a face image of a user who uses the device 110 . The photographing unit 220 may be built-in to the device 110 or may be externally connected to the device 110 . The photographing unit 220 obtains the user's face image so that the control unit 230 can analyze the concentration of the user's screen. In order for the control unit 230 to analyze the user's concentration level, the photographing unit 220 captures a plurality of face images of the user within a preset interval.

The control unit 230 analyzes the image captured by the photographing unit 220 based on the received learning model and controls scrolling of a screen output from the input/output unit 250 .

The control unit 230 analyzes the concentration of the user on the screen from the image captured by the photographing unit 220 based on the received learning model. The controller 230 predicts the movement of the pupil or the overall expression of the face (movement of each facial feature and eyebrow) in the image using the learning model, and analyzes the concentration on the screen therefrom. When the user is concentrating on the screen output from the input/output unit 250 , the pupil is stationary on one part of the screen or moves minutely (within a preset reference value) from one part of the screen to another part. On the other hand, if the user checks all the contents of the output screen or is not concentrating, the pupil moves without stopping in a part of the screen or moves with a relatively increased amount of movement (more than a preset reference value). In addition, when the user concentrates while watching the content provided from the screen, he or she moves some or all of the features to reveal the emotion according to the content of the content. On the other hand, when the user checks all the contents provided from the screen or the concentration is low, boredom is revealed or the user is distracted without being able to concentrate, and movement of some or all of the features is minimized. The controller 230 determines whether the movement of some or all of the features is greater than or equal to a preset reference value, and determines whether the user is concentrating on the provided screen and revealing a specific emotion. In order to use such a feature, the control unit 230 extracts the user's face and facial features from the image captured by the photographing unit 220 to analyze the concentration on the user's screen and analyzes their movements.

The memory unit 240 stores the learning model received from the server 120 .

The input/output unit 250 receives an input for outputting content from a user or outputs content according to the user's input.

3 is a diagram showing the configuration of an automatic scrolling learning server according to an embodiment of the present invention.

Referring to FIG. 3 , the server 120 according to an embodiment of the present invention includes a communication unit 310 , a control unit 320 , and a database 330 .

The communication unit 310 transmits the learning model generated by the control unit 320 to the device 110 .

The controller 320 learns a number of human face images stored in the database 330 and inputs for human scroll control, receives the face images, analyzes the concentration (change) on the human screen, and receives scroll control signals therefrom. Create a learning model that outputs

The controller 320 extracts the user's face and facial features from numerous human face images stored in the database 330 . A method for the controller 320 to extract the user's face and features is illustrated in FIGS. 4 and 5 .

4 is a diagram illustrating a process in which an automatic scrolling learning server or an automatic scrolling device detects a user's eyeball from an image according to an embodiment of the present invention, and FIG. 5 is an automatic scrolling learning server according to an embodiment of the present invention. Alternatively, it is a diagram illustrating a process in which the automatic scrolling device detects a pupil.

The controller 320 configures an artificial intelligence-based neural network for extracting the user's face and features from the image.

Artificial intelligence-based neural networks include Support Vector Machine (SVM), Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), Long-Short Term Memory, A classification model based on feature extraction such as LSTM) can be used.

The feature extraction model is largely a face recognition process that recognizes a user's face in an image using a machine learning-based algorithm, a landmark extraction process that extracts landmarks to detect the positions of the eyes and features in the recognized user's face, and image processing It consists of an eye movement tracking process that extracts the pupil and center positions of the eyes from the landmarks of the eyes.

First, the controller 320 uses a HoG (Histogram of Oriented Gradients) feature and an SVM-based face recognition algorithm for face recognition to perform an efficient face recognition function with high speed, low resource requirements, and high accuracy. SVM The HOG-based face detection method using The object of interest is searched for by extracting an object similar to the characteristic of the object of interest from the input image, and the time and effort required to manually search for the object of interest in a wide area can be reduced because the object of interest is automatically found.

Next, in the landmark extraction process, a facial landmark extraction algorithm is used to recognize the location of features in the face recognized in the face recognition process. In this case, the facial landmark extraction algorithm is a Convolutional Neural Network (CNN). When the working principle of CNN is schematically shown, the image is divided into small pieces, and specific features are extracted from the pieces through a small network. The network then moves to the next piece and extracts the features again using the same level in the same way. Create an additional network that moves and extracts different features and repeats the above method to apply the pieces to the network one by one. Afterwards, all the extracted features are well combined to finally judge the image. Through these operations, the control unit 320 extracts features from within the face.

After the feature is extracted, as shown in FIG. 3 , the control unit 320 performs a process of extracting the pupil from the extracted eye region through masking based on a threshold value.

That is, the controller 320 may set a threshold value in order to respond to a change in the environment according to the lighting in the input user's face image, the photographing angle, or the photographing location. The controller 320 may distinguish the pupil from the non-pupil portion in the eye region based on the set threshold value.

Referring back to FIG. 3 , the control unit 320 predicts the movement of the pupil and the change of the expression by using the motion prediction model based on the extracted features. The controller 320 predicts the movement of the pupil or the movement of the entire feature using Recurrent Neural Networks (RNN) or Long-Short Term Memory (LSTM). time), the control unit 320 extracts a feature from a plurality of images successive in time, and predicts the extracted features and movement of the pupil.

The controller 320 learns by matching a human face image with an input for controlling a human scroll, and generates a learning model that the device 110 can use. The controller 320 learns by matching the predicted facial features and pupil movements with an input for controlling a human scroll. As in the above example, the user may input scrolling upward or downward to the device 110 while moving the pupil without stopping at a certain point or showing boredom with an impression. Alternatively, the user may stop the pupil at a certain point or view the provided content, move the facial features, display a happy or sad expression, etc., while not inputting any scrolling input to the device 110 . The control unit 320 learns by matching the motions of the various pupils and features and the accompanying user's scrolling input. The controller 320 learns by matching the above-described numerous data stored in the database 330 . Accordingly, when receiving a human face image, the controller 320 generates a learning model that outputs a scroll control signal. The above-described learning model derives a scroll control signal from the received image by using the scroll operation as a label, such as the position of the scroll, whether the scroll is moved, and the direction in which the scroll is to be moved. An example of such a learning model is shown in FIG. 6 .

6 is a diagram illustrating an example in which an automatic scrolling apparatus scrolls according to a user's gaze according to an embodiment of the present invention. It is exemplified that the device 110 receives the learning model generated by the server 120 through the above-described process and analyzes it using the same.

For example, when the user moves the pupil from top to bottom while watching the screen (input/output unit) of the device 110 as shown in FIG. 6A , the device 110 extracts the pupil from the image and predicts the movement of the pupil. do. Although not shown in FIG. 6 , it is natural that facial expressions can be predicted by movement of features.

As shown in FIG. 6A , when the motion of the user moving the pupil from top to bottom is predicted, the device 110 may derive a control signal for lowering the scroll as shown in FIG. 6B based on the learning model. According to such a control signal, the screen output from the input/output unit 250 may scroll down.

Referring back to FIG. 3 , the controller 320 controls the communication unit 310 to transmit the learning model thus generated to the device 110 .

In order for the controller 320 to generate a learning model, the database 330 stores (as big data) a number of face images of a human watching a screen and an input for controlling a human scroll. In particular, so that the controller 320 can smoothly generate the learning model, the database 330 stores human face images in various situations in which scroll control is performed.

The above description is merely illustrative of the technical idea of this embodiment, and a person skilled in the art to which this embodiment belongs may make various modifications and variations without departing from the essential characteristics of the present embodiment. Accordingly, the present embodiments are intended to explain rather than limit the technical spirit of the present embodiment, and the scope of the technical spirit of the present embodiment is not limited by these embodiments. The protection scope of this embodiment should be interpreted by the following claims, and all technical ideas within the scope equivalent thereto should be interpreted as being included in the scope of the present embodiment.

100: automatic scrolling system
110: automatic scrolling device
120: automatic scrolling learning server
210, 310: communication unit
220: shooting department
230, 320: control unit
240: memory unit
250: input/output unit
330: database

Claims (7)

An automatic scrolling device for automatically scrolling an output screen by analyzing a user's face image,
a communication unit for receiving a learning model for deriving a scroll control signal from an external face image;
a photographing unit for photographing a user's face image;
an input/output unit for outputting content; and
A control unit for controlling whether the output screen output by the input/output unit is scrolled by analyzing the concentration on the user's screen from the user's face image captured by the photographing unit using the learning model
Automatic scrolling device comprising a. According to claim 1,
Automatic scrolling device characterized in that it further comprises a memory unit for storing the learning model received by the communication unit. According to claim 1,
The control unit is
The automatic scrolling device, characterized in that the user's pupil movement or the user's movement of each feature is predicted from the face image to analyze the concentration of the user's screen. 4. The method of claim 3,
The control unit is
An automatic scrolling device, characterized in that, when the movement of the user's pupil is relatively increased or the movement of each feature of the user is relatively decreased, the output screen output by the input/output unit is scrolled in one direction according to the learning model. In an automatic scrolling learning server that analyzes a user's face image and generates a learning model that enables automatic scrolling of an output screen,
a database for storing a human face image watching a screen and an input for human scroll control as big data;
a controller for learning a human face image in the database and an input for human scroll control, and generating a learning model that receives the face image and outputs a scroll control signal; and
Communication unit for transmitting the learning model generated by the control unit
Automatic scrolling learning server comprising a. 6. The method of claim 5,
The database is
Automatic scrolling learning server, characterized in that it stores human face images in each situation of scroll control. 6. The method of claim 5,
The control unit is
Automatic scrolling learning server, characterized in that the concentration on the user's screen is analyzed from the human face image in the database.

Images