KR20220052030A - System for enhancing online learning ability using eye tracking - Google Patents

Abstract

A system for improving an online learning ability using a gaze tracking technology comprises: a photographing device that photographs the pupils of a user and an educational material for which the user is looking at; and a gaze tracking part that extracts a location and direction of the pupil of the user for the educational material from an image photographed by the photographing device, and measures the location on the educational material. Therefore, the present invention is capable of diagnosing a learning ability of a student.

Description

Online learning ability improvement system using eye tracking technology {SYSTEM FOR ENHNCING ONLINE LEARNING ABILITY USING EYE TRACKING}

The following embodiments relate to an online learning ability improvement system using eye tracking technology.

As health and safety issues have recently emerged in society, regular education is being conducted through non-face-to-face online classes at frontline education sites. Due to the sudden shift from the existing face-to-face education, both teachers and students are experiencing considerable confusion.

The establishment of the online class-related system has progressed quickly, making it possible to proceed with the class, but in a situation where the educational effect of the existing face-to-face education is not being achieved, there is an increasing demand for measures to improve the quality of online classes.

In particular, as classes are conducted non-face-to-face, communication and feedback between teachers and students are not being conducted smoothly, such as whether the students understand the content of the class and whether they are taking the class with concentration.

Therefore, there is a need for a method to improve the educational effect through smooth communication and feedback between teachers and students even in a non-face-to-face class environment.

Korean Patent Publication No. 10-2012-0127790

The present invention is to solve the above-described problems, the user's pupil, a photographing device for photographing the educational material that the user is looking at, and the user's pupil position and direction with respect to the educational material in the image photographed by the photographing device It aims to provide an online learning ability improvement system using eye tracking technology that includes an eye tracking unit that extracts and measures a location on educational materials.

The problems of the present invention are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the following description.

The online learning ability improvement system using the eye tracking technology according to an embodiment of the present invention for solving the above problems is a photographing device for photographing the user's pupils, the educational material that the user is looking at, and It includes a gaze tracking unit for measuring the position on the educational material by extracting the location and direction of the user's pupil with respect to the educational material from the image.

According to the present invention, it is possible to check the student's learning ability in real time through eye tracking and analysis of educational data, and based on this, it is possible to diagnose the student's learning ability or provide customized education according to the student.

However, the effects of the present invention are not limited to the above-mentioned effects, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 is an exemplary configuration diagram for each function of an online learning ability improvement system using eye tracking technology according to an embodiment of the present invention.
2 to 4 show an exemplary implementation process of an online learning ability improvement system using an eye tracking technology according to an embodiment of the present invention.
5 is a schematic configuration diagram of an online learning ability improvement system using eye tracking technology according to an embodiment of the present invention.

Advantages and features of the present invention and methods of achieving them will become apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but will be implemented in a variety of different forms, and only these embodiments allow the disclosure of the present invention to be complete, and common knowledge in the technical field to which the present invention belongs It is provided to fully inform the possessor of the scope of the invention, and the present invention is only defined by the scope of the claims.

In the embodiments of the present invention, unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs. have meaning Terms such as those defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning in the context of the related art, and unless explicitly defined in an embodiment of the present invention, an ideal or excessively formal meaning is not interpreted as

The shapes, sizes, proportions, angles, numbers, etc. disclosed in the drawings for explaining the embodiments of the present invention are illustrative and the present invention is not limited to the illustrated matters. In addition, in describing the present invention, if it is determined that a detailed description of a related known technology may unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted. When 'including', 'having', 'consisting', etc. mentioned in this specification are used, other parts may be added unless 'only' is used. When a component is expressed in the singular, cases including the plural are included unless otherwise explicitly stated.

In interpreting the components, it is construed as including an error range even if there is no separate explicit description.

In the case of a description of the positional relationship, for example, when the positional relationship of two parts is described as 'on', 'on', 'on', 'beside', etc., 'right' Alternatively, one or more other parts may be positioned between two parts unless 'directly' is used.

Reference to an element or layer “on” another element or layer includes any intervening layer or other element directly on or in the middle of another element. Like reference numerals refer to like elements throughout.

The size and thickness of each component shown in the drawings are illustrated for convenience of description, and the present invention is not necessarily limited to the size and thickness of the illustrated component.

Each feature of the various embodiments of the present invention may be partially or wholly combined or combined with each other, and as those skilled in the art will fully understand, technically various interlocking and driving are possible, and each embodiment may be implemented independently of each other, It may be possible to implement together in a related relationship.

Hereinafter, an online learning ability improvement system using the eye tracking technology according to the present invention will be described with reference to the accompanying drawings.

The online learning ability improvement system using eye tracking technology extracts the user's pupil position and direction with respect to the educational material from the user's pupil, a photographing device that captures the educational material that the user is looking at, and the image captured by the photographing device to include an eye tracking unit that measures the position on the educational material.

The online learning ability improvement system using eye tracking technology may include an eye tracking device such as a camera that captures a student's gaze, and a display device that provides educational materials. The camera may be a basic camera of the terminal or a separate photographing device. The eye tracking device can also be used with a camera provided in terminals such as tablets, smartphones, and PCs, and the accuracy of eye tracking can be further increased by adding a separate measurement device. The separate eye tracking device may consist of three cameras, and the gaze tracking precision can be improved by tracking the gaze through the central main camera and performing correction through the auxiliary cameras on both sides.

Using eye tracking technology, we measure and analyze three positions: the camera position of the terminal, the position of the educational material on the display, and the position of the student's gaze watching the online class to determine where and for how long the student's gaze stays on the educational material. Data can be obtained by measuring and vector analysis of the movement of the gaze.

The camera position of the terminal is in a fixed state, and since texts and images on pre-prepared educational materials also have preset position point values, it is possible to accurately measure the pupils' pupils by tracking them through triangulation. Real-time measurements can be made by using vector values such as the time the student's gaze stays at a specific point, and the speed of movement.

In the educational material, related information, such as where the characters are already placed, which part is important, which part is difficult, which part to jump to which part, etc., can be set in advance. Educational materials may be in the form of e-books, children's children's books with pictures and characters, lectures in the form of videos including lecturer screens, written contents, and lecture materials, or may be provided in various forms. .

If the student's gaze stays on a certain point for a long time, the system may analyze that point as difficult, or it may analyze that the student does not focus on learning. In the learning content, the order of the gaze is set, but if the student's gaze moves differently, it can be analyzed that the student cannot follow the learning content. If the student's gaze moves to a specific point repeatedly, it can be analyzed as making it difficult or reconfirming the relevant part.

In this way, through eye tracking and analysis of educational data, a student's learning ability can be checked in real time, and based on this, the student's learning ability can be diagnosed or customized education can be provided according to the student. For example, if it is analyzed that a student has difficulty in a specific unit, educational materials with easier difficulty may be presented next, and re-education may be suggested by selecting only the parts that the student did not focus on.

Through this, it is possible to analyze the level of reading learning ability with big data-based artificial intelligence technology to secure high-accuracy, quantified and objective learning ability level status technology. In addition, it is possible to present a learning (cognitive) level and automatically provide recommended books so that reading can acquire rich vocabulary and expressive power, and expand imagination and thinking, which are essential elements in the era of creative convergence.

By accurately tracking pupils that are difficult to measure with fine movements, it is possible to accurately grasp the reading status in real time. Through this, the accuracy of biometric recognition can be increased, so that when the eye tracking system is applied, the position of the pupil can be read to determine which part is being looked at. In particular, 'fixation', where the gaze stays for a while at one point, 'saccade', where the gaze moves rapidly, 'scan path', which is the path of the gaze, Eye tracking is possible focusing on the four analysis elements of 'Revisit'.

The system may be a personal computer such as a desktop computer or a laptop computer, a smart phone, a personal digital assistant (PDA), a portable multimedia player (PMP), a mobile device such as a tablet PC, a wearable device, or a communication device capable of network access such as a server, There is no limit to the type or number of them.

The processor may be configured to process instructions of a computer program by performing input/output operations such as basic arithmetic, logic, and an image optimization method for a compressed package file. Instructions may be provided to the processor via a memory or network interface or bus. The processor may be configured to execute program code. The program code may be stored in a recording device such as a memory.

A processor may be configured to implement a function according to the present embodiment. The processor may be configured such that some components are omitted, additional components not shown are further included, or two or more components are combined according to a method of implementing each function.

The memory is a computer-readable recording medium and may include a random access memory (RAM), a read only memory (ROM), and a permanent mass storage device such as a disk drive. The memory may store an OS, program execution code, and other information. These software components may be loaded from a separate computer-readable recording medium such as a floppy drive, disk, tape, DVD/CD-ROM drive, or memory card. In another embodiment, the software components may be loaded into the memory through a network interface instead of a computer-readable recording medium.

The network interface may be a computer hardware component for connecting each component for performing a function according to the present embodiment to a computer network. The network interface may be connected to a computer network using a wireless connection or a wired connection method.

A bus may enable communication and data transfer between components. The bus may be configured using one or more of a high-speed serial bus, a parallel bus, a storage area network (SAN), and other communication methods.

An artificial intelligence learning model can be utilized, and the convolutional neural network consists of a feature extraction neural network and a classification neural network, and the feature extraction neural network stacks the input signal by sequentially stacking a convolutional layer and a pooling layer. The convolution layer includes a convolution operation, a convolution filter, and an activation function. The calculation of the convolution filter is adjusted according to the matrix size of the target input, but a 9X9 matrix is usually used. The activation function generally uses, but is not limited to, a ReLU function, a sigmoid function, and a tanh function. The pooling layer is a layer that reduces the size of the input matrix, and uses a method of extracting representative values by tying pixels in a specific area. In general, the average value or the maximum value is often used for the calculation of the pooling layer, but is not limited thereto. The operation is performed using a square matrix, which is usually a 9x9 matrix. The convolutional layer and the pooling layer are iterated alternately until the corresponding input becomes small enough while maintaining the difference.

A classification neural network has a hidden layer and an output layer. In a convolutional neural network, there are generally three or more hidden layers, and 100 nodes for each hidden layer are specified, but more or less can be specified in some cases. The activation function of the hidden layer uses a ReLU function, a sigmoid function, and a tanh function, but is not limited thereto. A total of 100 output layer nodes of a convolutional neural network can be made. The output layer activation function of the classification neural network uses the softmax function. The softmax function is a representative function of one-hot encoding, which makes the sum of all output nodes total 1, sets the output of the output node having the largest value to 1, and sets the output of the remaining output nodes to 0. . It is possible to select only one output out of 100 outputs through the softmax function.

A training signal for training a convolutional neural network may be manually generated including information to be modified by a control device manager. The control device manager can manually input the information to be corrected based on the judgment on the need to generate the learning signal, and through this, the database information can be strengthened or weakened according to the type of the learning signal.

The learning signal is created based on the error between the correct answer and the output value, and in some cases, SGD using delta, batch method, or method following the back propagation algorithm can be used. Based on this learning signal, the convolutional neural network performs learning by modifying the existing weights, and in some cases, momentum can be used. A cost function can be used to calculate the error, and a cross entropy function can be used as the cost function.

A convolutional neural network can learn to modify information in a database based on a learning signal. A pre-trained convolutional neural network has 3 or more hidden layers, and each hidden layer can have 50 or more hidden nodes. The activation function of each hidden node may use a ReLU function, a sigmoid function, and a tanh function, but is not limited thereto. The function of the output node can use the softmax function using the one-hot encoding technique. The output selects only one classification according to the one-hot encoding technique, and it is possible to execute a command from the selected classification.

Computing device 9000 may include one or more processors 9100 , memory 9200 , peripheral interface 9300 , input/output subsystem 9400 , power circuitry 9500 , and communication circuitry 9600 . . The computing device 9000 may include some or all of the components for implementation, or may be coupled to some or all of the components for implementation by the input/output subsystem 9400 .

The memory 9200 may include, for example, high-speed random access memory, a magnetic disk, SRAM, DRAM, ROM, flash memory, or non-volatile memory. The memory 9200 may include software modules, instruction sets, or other data necessary for the operation of the computing device 9000 . Access to the memory 9200 from components such as the processor 9100 or the peripheral interface 9300 may be controlled by the processor 9100 .

Peripheral interface 9300 may couple one or more of input peripherals and output peripherals of computing device 9000 to processor 9100 and memory 9200 . The processor 9100 may execute a software module or an instruction set stored in the memory 9200 to perform various functions for the computing device 9000 and process data.

The input/output subsystem 9400 may couple various input/output peripherals to the peripheral interface 9300 . For example, input/output subsystem 9400 may include a controller for coupling peripherals such as monitors, keyboards, mice, printers, touch screens, sensors, etc. to peripheral interface 9300 . Input/output peripherals may be coupled to peripherals interface 9300 without going through input/output subsystem 9400 .

The power circuit 9500 may supply power to all or some of the components of the terminal. For example, power circuit 9500 may include a power management system, one or more power sources, such as batteries or alternating current (AC), charging systems, power failure detection circuits, power converters or inverters, power status indicators or power sources. It may include any other components for creation, management, and distribution.

Communication circuitry 9600 may enable communication with other computing devices using one or more external ports. The communication circuit 9600 may include an RF circuit to transmit and receive RF signals to enable communication with other computing devices.

As an example, the computing device 9000 may have a configuration or arrangement in which some illustrated components are omitted, additional components not illustrated are further included, or two or more components are combined. For example, the computing device for a communication terminal in a mobile environment may further include a touch screen or a sensor, and the communication circuit 9600 includes various communication methods (WiFi, 3G, LTE, Bluetooth, NFC, Zigbee, etc.) Circuitry for RF communication may be included. Components that may be included in the computing device 9000 may be implemented in hardware, software, or a combination of both hardware and software including an integrated circuit specialized for one or more signal processing or applications.

Methods according to an embodiment of the present invention may be implemented in the form of program instructions that can be executed through various computing devices and recorded in a computer-readable medium. In particular, the program according to the present embodiment may be configured as a PC-based program or an application dedicated to a mobile terminal. The application to which the present invention is applied may be installed in the user terminal through a file provided by the file distribution system. As an example, the file distribution system may include a file transmission unit (not shown) that transmits the file according to a request of the user terminal.

The device described above may be implemented as a hardware component, a software component, and/or a combination of the hardware component and the software component. For example, devices and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA). , a programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions, may be implemented using one or more general purpose or special purpose computers. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. A processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, although one processing device is sometimes described as being used, one of ordinary skill in the art will recognize that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that can include For example, the processing device may include a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

Software may comprise a computer program, code, instructions, or a combination of one or more thereof, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be any kind of machine, component, physical device, virtual equipment, computer storage medium or device, to be interpreted by or to provide instructions or data to the processing device. , or may be permanently or temporarily embody in a transmitted signal wave. The software may be distributed over networked computing devices, and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CDROMs and DVDs, and magneto-optical disks such as floppy disks. hardware devices specially configured to store and execute program instructions, such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

Communication connection in the system and method according to the embodiments of the present invention may be configured regardless of communication aspects such as wired communication or wireless communication, and may include a local area network (LAN), a metropolitan area network (MAN) Network) and a wide area network (WAN) may be configured as various communication networks. Preferably, the communication connection in the present specification may be a well-known Internet or World Wide Web (WWW). However, the communication connection is not necessarily limited thereto, and may include a known wired/wireless data communication network, a known telephone network, or a known wired/wireless television communication network in at least a part thereof. For example, the communication connection is a wireless data communication network, such as Wi-Fi communication, WiFi-Direct communication, Long Term Evolution (LTE) communication, and Bluetooth communication (more specifically, low-power Bluetooth (BLE) communication). ; Bluetooth Low Energy)), infrared communication, ultrasonic communication, etc. may be implemented in at least a part of the conventional communication method. For another example, the communication connection may be an optical communication network, which implements at least a part of a conventional communication method such as LiFi (Light Fidelity).

Although embodiments of the present invention have been described in more detail with reference to the accompanying drawings, the present invention is not necessarily limited to these embodiments, and various modifications may be made within the scope without departing from the technical spirit of the present invention. . Accordingly, the embodiments disclosed in the present invention are not intended to limit the technical spirit of the present invention, but to explain, and the scope of the technical spirit of the present invention is not limited by these embodiments. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. The protection scope of the present invention should be construed by the following claims, and all technical ideas within the scope equivalent thereto should be construed as being included in the scope of the present invention.

Claims (1)

A photographing device for photographing the user's pupils and the educational material the user is viewing; and
An online learning ability improvement system using eye tracking technology, comprising a gaze tracking unit that extracts the location and direction of the user's pupil with respect to the educational material from the image captured by the photographing device and measures the location on the educational material.

Images