WO2024090830A1 - Cognitive training method and system based on eye tracking - Google Patents

Abstract

A cognitive training method based on eye tracking according to the present invention may comprise the steps of: providing content including a target stimulus and a distractor stimulus to a virtual space of each of a plurality of participants; monitoring the eye of each of the plurality of participants with respect to the target stimulus and the distractor stimulus on the basis of an eye tracking method; on the basis of a result of the monitoring, calculating a concentration level for the content for each of the plurality of participants; and providing a graphic object classified for each of the plurality of participants to a virtual space of an instructor, and providing information on the concentration level of each of the plurality of participants to the graphic object.

Description

Cognitive training method and system based on eye tracking

The present invention relates to a cognitive training method and system based on eye tracking. More specifically, in learning through virtual space, it relates to a method and system for calculating concentration by analyzing participant's gaze data and providing learning content based on the calculated concentration.

Virtual reality is an interface that creates three-dimensional 3D content of a specific environment or situation through a computer so that people using the 3D content can experience it as if they are actually interacting with the surrounding situation or environment. collectively.

As technology related to virtual reality develops, E-learning technology that conducts education using virtual reality is also developing. In addition, along with the development of virtual reality technology, there is a demand for the development of devices to provide more immersive virtual reality services. In particular, technology for conducting education using content provided through wearable devices, including head mounted displays (HMDs), is being widely used.

In this case, a head-mounted display refers to a digital device that can be worn on the head like glasses to provide multimedia content.

Accordingly, there is currently a growing demand to utilize the aforementioned virtual reality technology in education and training programs.

For example, Korean Patent Publication No. 10-1988110 describes a method of conducting education by evaluating the concentration of users using content provided through a head-mounted display in real time or cumulatively and feeding back the evaluated concentration to the user. It is provided.

However, this method does not provide configuration for an instructor who monitors the concentration of multiple participants and manages the learning process according to the concentration, so in an educational program in which multiple participants participate together, the instructor monitors the concentration of each participant and , there is still a need for technology that allows learning guidance to be conducted according to concentration.

The present invention relates to a method and system for calculating concentration by analyzing gaze data of participants and providing learning content based on the calculated concentration in learning through virtual space.

In order to solve the above-mentioned problems, the cognitive training method based on eye tracking according to the present invention includes the steps of providing content including target stimuli and distracting stimuli to the virtual space of each of the plurality of participants, based on the eye tracking method, Monitoring the gaze of each of the plurality of participants with respect to the target stimulus and the distracting stimulus, calculating a degree of concentration on the content for each of the plurality of participants based on the monitoring result, and in the instructor's virtual space, It may include providing graphic objects classified for each of the plurality of participants and providing information about the concentration of each of the plurality of participants to the graphic object.

Furthermore, the cognitive training method, based on the instructor's input for the graphic object, at least some of the gaze data obtained as a result of the monitoring for the selected participant, the degree of concentration on the content, and the screen at the viewpoint of the selected participant. It may further include providing information through the instructor's virtual space.

Furthermore, the cognitive training method may further include providing concentration-enhancing content that is different from the content through the virtual space of the selected participant in response to the instructor's input through the instructor's virtual space.

Furthermore, calculating the degree of concentration for the content includes identifying a first position of the target stimulus and a second position of the distracting stimulus on the content, based on the monitoring result, the gaze of each of the plurality of participants. It may further include obtaining data, and calculating a degree of concentration on the content based on a comparison of the gaze data with the first location and the second location.

Furthermore, providing the graphic object to the instructor's virtual space includes grouping the concentration of the plurality of participants into a plurality of groups based on a preset standard, and dividing the instructor's virtual space into a plurality of groups. The method may further include providing the graphic object including a visually distinguishable display according to the group.

Furthermore, providing the content to the virtual space of each of the plurality of participants includes receiving an input for the content from among a plurality of content lists, a first object corresponding to the target stimulus, and a first object corresponding to the distracting stimulus. The method may further include arranging a second object on a screen corresponding to the selected content, and providing the content including the first object and the second object to the virtual space of each of the plurality of participants. there is.

Furthermore, the cognitive training method includes receiving a learning evaluation for each of the plurality of participants from the instructor through the instructor's virtual space, and providing a report containing the gaze data and the learning evaluation of each of the plurality of participants. It may further include providing the virtual space to each of the plurality of participants.

Furthermore, the cognitive training method includes acquiring biometric signals of the participant through the virtual space of the participant, and based on the obtained biosignals, calculating performance scores for tasks included in the content for each of the plurality of participants. The method may further include calculating, wherein the graphic object includes information about the concentration level and the performance score of each of the plurality of participants.

Meanwhile, in the cognitive training system based on eye tracking according to the present invention, the control unit of the cognitive training system provides content including target stimuli and distracting stimuli to the virtual space of each of the plurality of participants, and based on the eye tracking method, , Monitoring the gaze of each of the plurality of participants with respect to the target stimulus and the distracting stimulus, and based on the monitoring result, calculating the degree of concentration on the content for each of the plurality of participants, in the virtual space of the instructor, the plurality of participants. Graphic objects classified for each participant may be provided, and information on the concentration of each of the plurality of participants may be provided to the graphic objects.

Meanwhile, it is a program that is executed by one or more processes in an electronic device according to the present invention and can be stored in a computer-readable medium, wherein the program transmits content including a target stimulus and an obstacle stimulus to each of the plurality of participants' virtual providing a space, based on an eye tracking method, monitoring the gaze of each of the plurality of participants with respect to the target stimulus and the distracting stimulus, based on the monitoring result, determining the degree of concentration on the content among the plurality of participants. Includes instructions for performing calculations for each participant, providing graphic objects classified for each of the plurality of participants in the instructor's virtual space, and providing information about the concentration of each of the plurality of participants to the graphic object. can do.

According to various embodiments of the present invention, a cognitive training method and system based on eye tracking provides virtual reality educational content through a wearable device, obtains the concentration of each participant in real time by analyzing the concentration from the participant's gaze information, and It can be monitored.

In addition, the cognitive training method and system based on eye tracking according to various embodiments of the present invention visually displays the concentration of each participant according to the concentration value, allowing the instructor to intuitively understand the real-time concentration of each participant. You can.

Furthermore, the cognitive training method and system based on eye tracking according to various embodiments of the present invention prevents a decrease in the participant's concentration and prevents the participant's learning by providing customized feedback to the participant based on the instructor's concentration on learning. can be supported efficiently.

1 shows a cognitive training system based on eye tracking according to the present invention.

Figure 2 is a conceptual diagram of a cognitive training system based on eye tracking according to the present invention.

Figure 3 is a conceptual diagram of the database stored in the cognitive training system according to this outbreak.

Figure 4 is a flowchart showing a cognitive training method based on eye tracking according to the present invention.

Figure 5 shows a content design screen displayed through an instructor terminal according to the present invention.

Figure 6 shows content displayed through a participant terminal according to the present invention.

Figure 7a shows the monitoring results according to eye tracking of a participant's virtual space according to the present invention.

Figure 7b is a flowchart showing a method of calculating the concentration of a participant based on eye tracking according to the present invention.

Figure 8 shows a graphic object displayed through an instructor terminal according to the present invention.

Figures 9a and 9b illustrate a configuration in which information related to participants according to the present invention is displayed through an instructor terminal.

Figures 10a and 10b illustrate a configuration for providing content to improve concentration to participants based on an instructor's input according to the present invention.

Hereinafter, embodiments disclosed in the present specification will be described in detail with reference to the accompanying drawings. However, identical or similar components will be assigned the same reference numbers regardless of drawing symbols, and duplicate descriptions thereof will be omitted. The suffixes “module” and “part” for components used in the following description are given or used interchangeably only for the ease of preparing the specification, and do not have distinct meanings or roles in themselves. Additionally, in describing the embodiments disclosed in this specification, if it is determined that detailed descriptions of related known technologies may obscure the gist of the embodiments disclosed in this specification, the detailed descriptions will be omitted. In addition, the attached drawings are only for easy understanding of the embodiments disclosed in this specification, and the technical idea disclosed in this specification is not limited by the attached drawings, and all changes included in the spirit and technical scope of the present invention are not limited. , should be understood to include equivalents or substitutes.

Terms containing ordinal numbers, such as first, second, etc., may be used to describe various components, but the components are not limited by the terms. The above terms are used only for the purpose of distinguishing one component from another.

When a component is said to be "connected" or "connected" to another component, it is understood that it may be directly connected to or connected to the other component, but that other components may exist in between. It should be. On the other hand, when it is mentioned that a component is “directly connected” or “directly connected” to another component, it should be understood that there are no other components in between.

Singular expressions include plural expressions unless the context clearly dictates otherwise.

In this application, terms such as “comprise” or “have” are intended to designate the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, but are not intended to indicate the presence of one or more other features. It should be understood that this does not exclude in advance the possibility of the existence or addition of elements, numbers, steps, operations, components, parts, or combinations thereof.

1 shows a cognitive training system based on eye tracking according to the present invention. Figure 2 is a conceptual diagram of a cognitive training system based on eye tracking according to the present invention. Figure 3 is a conceptual diagram of the database stored in the cognitive training system according to this outbreak.

Referring to FIGS. 1 and 2 together, the cognitive training system 100 (hereinafter referred to as the cognitive training system 100) based on eye tracking according to the present invention provides content 210 related to learning to the participant terminal 110. And, a graphic object 240 including information 220 (eg, eye tracking data) acquired through the participant terminal 110 may be provided to the instructor terminal 120.

At this time, the participant terminal 110 and the instructor terminal 120 may be referred to as various types of electronic devices. For example, the participant terminal 110 and the instructor terminal 120 may be referred to as at least one of a smartphone, tablet PC, e-book reader, desktop PC, laptop PC, home appliance device, or wearable device.

However, the participant terminal 110 and the instructor terminal 120 are not limited to the above-mentioned examples, and any device that can provide a virtual space (or virtual reality) service to a user may be sufficient. Below, for convenience of explanation, the participant terminal 110 and the instructor terminal 120 will be described with reference to a head mount display (HMD) (hereinafter, HMD).

The instructor may access (or be connected to) the cognitive training system 100 through the instructor terminal 120. Additionally, a plurality of participants may access (or be connected to) the cognitive training system 100 at the same time through each participant terminal (110-1, 110-2, 110-3, 110-4).

Meanwhile, the cognitive training system 100 according to the present invention may include at least one of a communication unit 150, a storage unit 130, an input unit (not shown), and a control unit 140.

Meanwhile, the cognitive training system 100 according to the present invention may be implemented as an application or software. The cognitive training system 100 implemented as an application in this way is downloaded through a program that can download applications on an electronic device (e.g., Play Store, App Store), or is implemented through an initial installation program on an electronic device. It can be. In this case, the communication unit 150, storage unit 130, input unit, and control unit 140 according to the present invention can be used as components of an electronic device.

Additionally, the cognitive training system 100 according to the present invention may be provided through a web page accessed through the participant terminal 110 and the instructor terminal 120.

According to the present invention, the communication unit 150 may communicate with at least one of the participant terminal 110 and the instructor terminal 120 by wire or wirelessly. The communication unit 150 transmits the content 210 received from the instructor terminal 120 to the participant terminal 110 using wired or wireless communication, or transmits information obtained from the participant terminal 110 to the instructor terminal 120. Can be transmitted.

At this time, the communication unit 150 may support various communication methods according to the communication standard of at least one of the communicating participant terminal 110 and the instructor terminal 120.

For example, the communication unit 150 supports wireless LAN (WLAN), wireless-fidelity (Wi-Fi), wireless fidelity (Wi-Fi) Direct, digital living network alliance (DLNA), wireless broadband (WiBro), and WiMAX ( World Interoperability for Microwave Access), HSDPA (High Speed Downlink Packet Access), HSUPA (High Speed Uplink Packet Access), LTE (Long Term Evolution), LTE-A (Long Term Evolution-Advanced), 5G (5th Generation Mobile Telecommunication) , Bluetooth™ RFID (Radio Frequency Identification), Infrared Data Association (IrDA), UWB (Ultra-Wideband), ZigBee, NFC (Near Field Communication), Wi-Fi Direct, Wireless USB (Wireless Universal Serial Bus) ) may be configured to communicate with at least one of the participant terminal 110 and the instructor terminal 120 using at least one of the technologies.

According to the present invention, the storage unit 130 may be configured to store information related to the content 210 acquired from the instructor terminal 120. Additionally, the storage unit 130 may be configured to store user information and monitoring data (eg, eye tracking data) obtained from the participant terminal 110.

The storage unit 130 according to one embodiment may be provided in the cognitive training system 100 itself. According to another embodiment, at least a portion of the storage unit 130 may be referred to at least one of a cloud server and a database of an external server. In other words, the storage unit 130 is referred to as a space where information related to cognitive training according to an embodiment is stored, and can be understood as having no restrictions on physical space.

At this time, as shown in FIG. 3, the cognitive training information 301 may include i) content information 310, and ii) user information 320.

More specifically, the content information 310 may include at least one of i) target stimulus location 311, ii) interference stimulus location 312, and iii) content information 313 included in the content.

Additionally, the user information 320 may include at least one of i) participant account information 321, ii) participant gaze data 322, iii) participant concentration data 323, and iv) participant learning data 324. there is.

However, the cognitive training information 301 stored in the storage unit 130 is not limited to the above-mentioned example, and may be referred to as various information that is created or transformed during the participant's cognitive training.

The input unit (not shown) according to the present invention is for inputting information from the manager, and the input unit can be a medium between the manager and the cognitive training system 100. More specifically, the input unit may mean an input means for receiving at least some of the cognitive training information 301 related to cognitive training from the manager.

At this time, the input unit may be referred to as various types of input members. The input unit according to one embodiment is a mechanical input means (or mechanical key, for example, a mouse, a joy stick, a physical button, a dome switch, a jog wheel, a jog switch) etc.) and a touch input means. According to another embodiment, the touch input means is referred to as a virtual key, soft key, or visual key displayed on the touch screen through software processing, or is referred to as a virtual key, soft key, or visual key other than the touch screen. It may be referred to as a touch key placed in the part of . Meanwhile, the virtual key or visual key can be displayed on the touch screen in various forms, for example, graphic, text, icon, video or these. It can be made up of a combination of .

Next, the control unit 140 may control the overall operation of the cognitive training system 100. To this end, the control unit 140 includes at least one of the target stimulus and obstacle stimulus setting unit 141, the user eye tracking unit 142, the user achievement determination unit 143, the screen processing unit 144, and the user feedback unit 145. may include.

According to one embodiment, the control unit 140 may process data related to the cognitive training information 301 input through the input unit or received through the communication unit 150. Additionally, the control unit 140 may store data acquired from the participant terminal 110 or the instructor terminal 120 in the storage unit 130.

In addition, the control unit 140 transmits data related to the cognitive training information 301 to the participant terminal 110 and/or the instructor terminal 120, thereby providing the information to the participant terminal 110 and/or the instructor terminal 120. Control can be performed so that information related to content is output through the displayed display.

The control unit 140 may control the screen (or virtual space) displayed through the display provided on the participant terminal 110 and/or the instructor terminal 120 through the screen processing unit 144.

At this time, there is no limitation on the type of display provided on the participant terminal 110 and/or the instructor terminal 120.

Therefore, for convenience of explanation, the operation of the cognitive training system 100 will be described below as being controlled by the control unit 140.

Based on the configuration of the cognitive training system 100 discussed above, below, content 210 related to learning is provided to the participant terminal 110, and information 220 acquired through the participant terminal 110 is provided. The method of providing the graphic object 240 to the instructor terminal 120 will be described in more detail.

Figure 4 is a flowchart showing a cognitive training method based on eye tracking according to the present invention. Figure 5 shows a content design screen displayed through an instructor terminal according to the present invention. Figure 6 shows content displayed through a participant terminal according to the present invention. Figure 7a shows the monitoring results according to eye tracking of a participant's virtual space according to the present invention. Figure 7b is a flowchart showing a method of calculating the concentration of a participant based on eye tracking according to the present invention. Figure 8 shows a graphic object displayed through an instructor terminal according to the present invention. Figures 9a and 9b illustrate a configuration in which information related to participants according to the present invention is displayed through an instructor terminal. Figures 10a and 10b illustrate a configuration for providing content to improve concentration to participants based on an instructor's input according to the present invention.

The control unit 140 according to the present invention can set the positions of the target stimulus 501 and the interference stimulus 502 included in the content by controlling the target stimulus and interference stimulus setting unit 141.

As shown in FIG. 5, the control unit 140 may provide an interface 500 for designing cognitive training content based on eye tracking through the instructor terminal 120.

More specifically, the control unit 140 provides an interface 500 for creating content to be provided to the participant terminal 110 through the instructor terminal 120 by designing one content among the plurality of content lists 510. can be provided.

First, in response to the instructor's input for the first training program 511 among the plurality of content lists 510, the control unit 140 displays the screen 520 corresponding to the selected first training program 511 through the interface 500. ) can be displayed through.

Furthermore, the control unit 140 controls the target stimulus and interference stimulus setting unit 141 to create a first object 531 corresponding to the target stimulus 501 and a second object 532a corresponding to the interference stimulus 501. At least a portion of 532b and 532c may be placed on the screen 520 corresponding to the first training program 511.

For example, based on the instructor's drag input through the virtual space, the control unit 140 selects at least some of the first object 531 corresponding to the target stimulus and the second object 532 corresponding to the distracting stimulus to the first object 532. It can be placed on the screen 520 corresponding to the training program 511.

For another example, in response to a preset object being identified from the screen 520 corresponding to the first training program 511, the control unit 140 displays the first object 531 at the point where the identified object is placed. Alternatively, at least some of the second objects 532 may be arranged.

Furthermore, the control unit 1400 may set the position where the first object 531 is placed as the position of the target stimulus 501, and set the position where the second object 532 is placed as the position of the interference stimulus 502. .

That is, the control unit 140 controls the target stimulus and obstruction stimulus setting unit 141 to set the target stimulus 501 and the obstruction stimulus ( 502) can be set.

Through the above-described configuration, the control unit 140 according to the present invention provides the above-described interface 500 through the instructor terminal 120, allowing the instructor to design content to be provided to the participant terminal 110.

Furthermore, the interface 500 may include a graphic object 540 including a plurality of participant lists. The control unit 140 according to the present invention can provide information related to the learning situation of each participant through a graphic object 540 including a plurality of participant lists. However, a detailed explanation of this is provided below.

The control unit 140 according to the present invention may provide content 620 including the target stimulus 501 and the distracting stimulus 502 to the virtual space of each of the plurality of participants (S401).

More specifically, as shown in FIG. 6, the control unit 140 controls content including a first object 531 corresponding to the target stimulus 501 and a second object 532 corresponding to the interference stimulus 502. (620) can be provided through each participant terminal (110-1, 110-2, 110-3, 110-4) of a plurality of participants.

The control unit 140 uses the screen processing unit 144 to output the content 620 including the target stimulus 501 and the distracting stimulus 502 to the virtual space of the participant 611 through the participant terminal 110. You can control it.

At this time, for example, the content 620 provided in the participant's virtual space may include a background (or screen) such as a classroom, reading room, desk, blackboard, or conference room for learning, but is limited thereto. That is not the case.

In addition, the control unit 140 controls the content ( 620) can be provided through the participant terminal 110 of the participant 611.

However, according to another embodiment (not shown), the controller 140 may control the first object 531 and the second object 532 not to be displayed through the participant terminal 110. That is, the first object ( Content 620 with at least part of 531 or the second object 532 removed may be provided through the participant terminal 110.

Through this, each of the plurality of participants according to the present invention can perform learning using the content 620 provided through the participant terminal 110. For example, the participant 611 memorizes the content included in the content 620 provided through the first participant terminal 110-1, solves a problem, or takes a lecture, and thus participates in the content 620. Learning can be done.

According to the above-described configuration, the cognitive training system 100 according to the present invention provides learning-related content 620 through the participant terminal 110, so that the participant performs learning through the content 620 provided in the virtual space. You can do it.

Furthermore, the control unit 140 according to the present invention can monitor the gaze of each of the plurality of participants toward the target stimulus 501 and the distracting stimulus 502 based on the eye tracking method (S403).

As shown in FIG. 7A, the control unit 140 according to the present invention uses the user eye tracking unit 142 to track a plurality of participants for content 620 including the target stimulus 501 and the distracting stimulus 502. Gaze data 770 (eg, participant gaze data 322 in FIG. 3) can be obtained by tracking each gaze.

More specifically, the control unit 140 according to the present invention acquires gaze data 770 by tracking and monitoring the gaze of the participant 611 in the participant's virtual space, and uses the acquired gaze data 770 as content 620. It can be displayed on the table.

However, according to another embodiment (not shown), the control unit 140 may store the gaze data 770 obtained as a result of gaze monitoring in the storage unit 130 without displaying it on the content 620. .

Furthermore, the control unit 140 tracks and monitors the gaze of the participant 611 in the participant virtual space through the user gaze tracking unit 142, thereby providing gaze data including the movement of the gaze, the position at which the gaze is fixed, or the time at which the gaze is fixed. 770 may be obtained, and a comparison may be performed between the gaze position and the positions of the target stimulus 501 and the distracting stimulus 502.

Through this, the control unit 140 according to the present invention can calculate the degree of concentration on the content 620 for each of the plurality of participants based on the monitoring results (S405).

More specifically, the control unit 140 according to the present invention can calculate the participant's degree of concentration on the content 620 by controlling the user achievement determination unit 143 based on the participant's eye tracking results.

For example, the control unit 140 inputs the gaze data 770 obtained as a result of monitoring the participant's gaze to the user achievement determination unit 143, thereby calculating the participant's degree of concentration on the content 620. .

Below, we will explain in more detail how to calculate the participant's concentration level based on the monitoring results of the participant's gaze.

As shown in FIG. 7B, when the participant's gaze is fixed, the control unit 140 obtains parameters related to the gaze and calculates the concentration of the participant 611 on the content 620 based on the obtained parameters. You can.

First, the control unit 140 can determine the position of the gaze of the participant 611 (S701).

More specifically, the control unit 140 controls the participant 611 based on at least some of the eye (or iris) movement, the number of blinks, and the degree of pupil dilation of the participant 611 obtained through the participant terminal 110. ) can obtain data about the point they are looking at on the content 620.

For example, the control unit 140 acquires the movement of the eye (or iris) of the participant 611 through the participant terminal 110, and based on the acquired eye movement, the participant 611 displays the content 620. ) You can identify the location of a point you are watching on the image.

Furthermore, the control unit 140 acquires the eye movements and blinking times of the participant 611 through the participant terminal 110, and based on the obtained information, the point at which the participant 611 is looking at the content 620 The location and gaze time can be identified.

Furthermore, the control unit 140 may determine whether the gaze position of the identified participant 611 is fixed (S703).

More specifically, the control unit 140 acquires at least some of the eye movement, degree of pupil dilation, and number of blinks of the participant 611 through the participant terminal 110, and based on the obtained information, the participant 611 It can be determined whether the user is looking at a point on the content 620 for more than a preset time.

For example, when the control unit 140 determines that the participant 611 is looking at a point on the content 620 for more than a preset time, the control unit 140 determines that the gaze of the participant 611 is fixed on that point. You can.

If the position of the gaze of the participant 611 is not fixed because the participant 611 is not looking at a specific point, the control unit 140 may again determine the position of the gaze of the participant 611 through step S701.

When the gaze of the participant 611 is fixed on a point on the content 620, the control unit 140 may obtain at least one parameter related to the gaze (S705).

More specifically, when the participant 611 is looking at a point on the content 620 for more than a preset time, at least one parameter related to the gaze of the participant 611 may be obtained.

At this time, at least one parameter related to the gaze is: i) latency, ii) movement distance, iii) movement speed, iv) time fixed at a point after movement, v) target stimulus 501 It may include at least one of accuracy, and vi) gaze fixation ratio of the target stimulus 501 and the distractor stimulus 502.

At this time, the latency period of the gaze may mean the time when the gaze begins to move. In addition, the accuracy of gaze with respect to the target stimulus 501 may mean the ratio of points included in the region of interest including the location of the target stimulus 501 among the points on which the gaze of the participant 611 is fixed. In addition, the fixation ratio of the target stimulus 501 and the distracting stimulus 502 includes the gaze of the participant 611 fixed on the area containing the distracting stimulus 502 and the location of the target stimulus 501. It may refer to the ratio of gaze fixed on the area of interest.

However, parameters related to gaze are not limited to the above-mentioned examples, and can be understood as various parameters as data that can be obtained through monitoring the participant's gaze.

Furthermore, the control unit 140 according to the present invention can calculate the concentration of the participant 611 based on at least one obtained parameter (S707).

More specifically, the control unit 140 adjusts the concentration of the participant 611 by comparing the position where the gaze of the participant 611 is fixed on the content 620 with the positions of the target stimulus 501 and the distracting stimulus 502. It can be calculated.

For example, when the relationship between the position where the gaze of the participant 611 is fixed on the content 620 and the position of the target stimulus 501 decreases below a preset threshold, the control unit 140 controls the It can be judged that concentration has decreased.

For another example, the control unit 140 accumulates the gaze data of the participant 611 for a preset time, and determines the point where the participant's gaze is most often fixed, the distance from the target stimulus 501, and the interfering stimulus ( By comparing the distance with 502), the concentration of the participant 611 can be calculated.

For another example, the control unit 140 acquires the movement of the gaze of the participant 611 for a preset time, and determines whether the direction in which the gaze of the participant 611 is directed according to the movement is toward the target stimulus 501. The participant's (611) degree of concentration on the content (620) can be calculated depending on whether the direction is toward the interfering stimulus (502).

As another example, the control unit 140 calculates the average and standard deviation of the accuracy and/or gaze fixation ratio between the gaze fixation position and the target stimulus 501, and based on the calculated information, the participant 611 The concentration level for content 620 can be calculated.

Meanwhile, the control unit 140 may calculate the concentration of the participant 611 based on the pattern of changes in the fixation position of the participant's gaze.

For example, the control unit 140 determines the first time point when the relationship between the position where the gaze of the participant 611 is fixed and the position of the target stimulus 501 changes to more than a preset threshold, and for the first time after the first time point, A second point in time at which relevance decreases below a preset threshold may be determined. Additionally, the control unit 140 may determine a third time point at which the relevance increases above a preset threshold value for the first time after the second time point.

Accordingly, the control unit 140 calculates the first period between the first time point and the second time point and the second period between the second time point and the third time point, and determines the concentration based on the ratio of the first period and the second period. It can be calculated.

For another example, the control unit 140 may repeat the process of calculating the first period and the second period as described above while the participant 611 continues training. At this time, if each of the first period and the second period is repeated within a preset threshold range, the control unit 140 may determine a correction value for the concentration so that it is proportional to the cycle in which the first period and the second period are repeated. Through this, the control unit 140 can prevent the concentration level from being calculated to be relatively low due to the participant 611's habits or illness.

As another example, the control unit 140 may increase or decrease the correction value based on the ratio of the first period and the second period described above. For example, the control unit 140 may increase the correction value as the first period is larger than the second period, and may decrease the correction value as the first period is smaller than the second period.

Furthermore, the control unit 140 may group the concentration of the plurality of participants on the content 620 calculated into a plurality of groups based on a preset standard.

Through this, the control unit 140 can provide a graphic object that visually displays concentration information for each of the plurality of participants according to the group through the instructor's virtual space. However, a detailed explanation of this will be provided later.

Meanwhile, the control unit 140 acquires the biological signals of the participant 611 through the participant terminal 110, and calculates the performance score for the task (or training) included in the content 620 based on the acquired biological signals. It can be calculated.

For example, the control unit 140 acquires at least some of ground reaction force, photoplethysmogram (PPG), electrocardiogram (ECG), electroencephalography (EEG), and eye tracking data through the participant terminal 110, and based on the acquired data, , the performance result (or performance score) for the task included in the content 620 can be calculated. Additionally, the control unit 140 may calculate a performance score for each bodily function of the participant 611, based on the biosignals described above.

At this time, the performance score may be calculated based on the performance accuracy, number of performance, or difficulty of the task included in the content 620.

In addition, the control unit 140 determines the recommended task difficulty level, task content, or number of task performance times based on the calculated performance score, and provides the information to the participant 611 through a virtual space displayed through the participant terminal 110. You can.

In this regard, the control unit 140 may analyze the concentration and performance scores based on preset standards for each, and change the content provided to the participant 611 based on the analysis results. Here, the preset criteria may be set to different threshold values for each concentration and performance score.

For example, if the performance score is above the first threshold and the concentration is below the threshold, the control unit 140 may determine that the content provided to the participant 611 is lower than the level of the participant 611 and increase the difficulty of the task. there is.

For another example, when the performance score is above the second threshold and the concentration is below the threshold, it is determined that the participant 611 is having difficulty using the participant terminal 110 and a method of using the participant terminal 110 is determined. We can provide content that teaches. At this time, the second preset threshold value for the performance score may be set lower than the first threshold value.

For another example, if the performance score is less than the second threshold and the concentration is more than the threshold, the content provided to the participant 611 may be determined to be higher than the participant's level and the difficulty of the task may be lowered.

For another example, when the performance score is less than the second threshold and the concentration is less than the threshold, it is determined that the concentration of the participant 611 has decreased, and content that provides attention or relaxation content may be provided. .

Through the above-described configuration, the cognitive training system 100 according to the present invention calculates the participant's learning degree (or task performance degree) based on the participant's gaze data and other biological signals, thereby providing a task suitable for the participant's learning degree. Content including can be provided through the participant's virtual space.

Furthermore, the control unit 140 according to the present invention can provide graphic objects divided by a plurality of participants in the instructor's virtual space and provide information about the concentration of each of the plurality of participants to the graphic object 240 ( S407).

The control unit 140 may display information about the concentration (or learning achievement) of each of the plurality of participants through the graphic object 240. That is, depending on the degree of concentration, the visual appearance of the graphic object 240 may be configured differently. For example, the control unit 140 may display the graphic object 240 in different colors according to the concentration level or learning achievement level. Alternatively, the control unit 140 may include information (eg, text information) about the participant's concentration or learning achievement level in the graphic object 240.

More specifically, as shown in FIG. 8, the control unit 140 according to the present invention displays graphic objects that are visually distinguished according to the concentration calculated for each plurality of participants in the instructor's virtual space provided on the instructor terminal 120. (240) can be provided.

For example, the control unit 140 may display participant 2, participant 5, participant 7, participant 9, participant 11, and participant 12 as having “normal” levels of concentration through the graphic object 240. At this time, the normal grade may be represented by a graphic object 240 having a first color (eg, green).

In addition, the control unit 140 can display through the graphic object 240 that Participant 1, Participant 6, and Participant 10 have concentration levels of “insufficient”, and Participant 3 and Participant 8 have concentration levels of “poor”. there is. At this time, the insufficient grade may be represented by a graphic object 240 having a second color (e.g., yellow), and the poor grade may be represented by a graphic object 240 having a third color (e.g., red). there is. In this way, the control unit 140 can display the status of multiple participants in different visual appearances so that the instructor can understand them at a glance. That is, in the present invention, different visual features for a plurality of different grades may be matched and stored in the storage unit 130 as matching information. And, the control unit 140 can intuitively display information about the participant through the graphic object 240 that references this matching information.

However, the classification according to the level of concentration displayed by the control unit 140 through the graphic object 240 and the display method according to the classification are not limited to the above-mentioned examples, and information related to the concentration obtained through monitoring of gaze It can be understood as a variety of means to provide.

In addition, as described above, the control unit 140 provides graphic objects 240 according to the concentration (or academic achievement) of a plurality of participants in the instructor's virtual space provided on the instructor terminal 120. This can be done in real time while the participants are using the content provided to each participant, or it can be done after the use of the content is completed. Through the above-described configuration, the cognitive training system 100 according to the present invention can determine the concentration of each participant based on the participant's eye tracking and display the concentration of each participant to the instructor through the instructor's virtual space.

Furthermore, the control unit 140 according to the present invention is based on the instructor's input on some of the graphic objects 240 divided by a plurality of participants, the gaze data obtained for the selected participant, the degree of concentration, and the screen at the selected participant's viewpoint. At least part of it can be provided through the instructor's virtual space.

In other words, the participant's virtual space can be provided as the instructor's virtual space. At this time, the instructor can access (or visit) the participant's virtual space in a real-time situation where the participant is learning. To this end, the control unit 140 may connect the instructor's terminal 120 to the participant's virtual space. In this case, the participant's virtual space may be provided to the instructor's terminal 120. At this time, an avatar corresponding to the instructor may be added to each virtual space of the instructor terminal 120 and the participant's terminal.

The instructor's access to the participant's virtual space may be accomplished at the request of at least one of the instructor or the participant. The instructor can access the participant virtual space by selecting the graphic object 240 corresponding to each participant, as seen above, through the instructor terminal. When the graphic object 240 is selected from the instructor terminal, the control unit 140 may perform a series of controls to connect the instructor terminal to the participant's virtual space.

Additionally, when a command requesting the instructor's intervention is transmitted from the participant terminal 110, the control unit 140 may connect the instructor terminal to (or visit) the participant's virtual space.

To this end, the participant terminal 110 may receive a command requesting the instructor's intervention from the participant 611. For example, a command requesting the instructor's intervention may be a manipulation of an input means provided on the participant terminal 110, such as a physical button, or may be an input through a separately provided input means. For another example, a command requesting the instructor's intervention may correspond to gaze movement to a preset point in the virtual space provided to the participant 611.

Meanwhile, when the instructor is connected to the participant's virtual space, the instructor terminal may provide at least some of the gaze data, concentration, and screen at the selected participant's viewpoint to the content provided in the participant's virtual space. You can. Furthermore, the participant's learning content is provided to the instructor's terminal in the participant's virtual space, which allows the instructor to monitor the participant in real time.

Referring to FIGS. 8, 9A, and 9B together, the control unit 140 collects gaze data 951, 952 of “Participant 3” based on the instructor’s input regarding “Participant 3” among the graphic objects 240; The concentration level 901 and the screen 930 from the perspective of “Participant 3” can be provided through the instructor’s virtual space provided on the instructor terminal 120.

As shown in FIGS. 9A and 9B, the control unit 140 according to the present invention controls the concentration level 901 and learning information 910 of the selected participant based on the instructor's input for some of the graphic objects 240. The screen 930 from the selected participant's viewpoint can be provided to the instructor's virtual space. At this time, the participant's gaze data 951 and 952 may overlap on the screen 930 from the participant's viewpoint and be provided to the instructor's virtual space. Meanwhile, as described above, the concentration level 901 may be displayed to be visually distinguished according to the level of concentration, as shown in FIG. 8.

For example, the screen 930 of the participant's viewpoint provided in the instructor's virtual space may include gaze data 951 about the position or point where the participant's gaze rests, as shown in FIG. 9A. The control unit 140 may overlap the graphic object for the location or point where the user's gaze rests on content including graphic objects corresponding to the target stimulus and the distracting stimulus, respectively, and provide the information to the instructor as the participant's gaze data. Through this, the control unit 140 can intuitively recognize which stimulus the participant is paying attention to and which stimulus was interrupted.

For another example, the screen 930 from the participant's viewpoint provided in the instructor's virtual space may include gaze data 952 about the order in which the participant's gaze stayed, as shown in FIG. 9B. The control unit 140 may detect a point (for example, a gaze point) where the participant's gaze remains for more than a preset time in the content provided to the participant. In addition, when a plurality of gaze points are detected, information about the order in which the plurality of gaze points stay can be displayed together with the plurality of gaze points. As shown in FIG. 9B, when eight gaze points are detected, the control unit 140 can provide the eight gaze points to the instructor terminal by attaching a number according to the time order in which each gaze stopped. At this time, information about the gaze point including order information may be displayed overlapping content including graphic objects corresponding to the target stimulus and the distracting stimulus, respectively.

Through this, the instructor can intuitively recognize the order in which participants look at the content and focus their attention on content containing graphic objects corresponding to target and distracting stimuli, respectively.

The instructor can provide real-time feedback on the participant's learning in the participant's virtual space provided on the instructor terminal. In other words, the instructor and participant can access the participant's virtual space together, and the instructor can transmit real-time feedback commands about the participant's learning on the participant's virtual space through the instructor terminal. Through this, participants can receive learning guidance from the instructor in real time.

Furthermore, the control unit 140 according to the present invention can provide information related to the learning situation of each participant through a graphic object 540 including a plurality of participant lists. More specifically, the control unit 140 according to the present invention can display each of the plurality of participants to be visually distinguished according to the calculated concentration through a graphic object 540 including a list of the plurality of participants.

Also, for example, learning information 910 may include the participant's i) learning level 902, ii) performance score 902, iii) turnaround time (“” and iv) heart rate 905; It is not limited and can be understood to include various types of information about the participant's learning level or biosignals.

As shown in FIG. 9A, the control unit 140 can display the participant's gaze data to be visually distinguished according to the location of the gaze and the time or frequency at which the gaze is fixed at a certain location. For example, the control unit 140 may display the participant's gaze to be visually distinguished through color turbidity, brightness, etc. depending on the time the participant's gaze is fixed on a specific location.

Referring to FIG. 9B according to another embodiment, the control unit 140 may display the gaze data of the participant in quantification of the time or frequency that the gaze is fixed at a certain position according to the location of the gaze. For example, the control unit 140 may display the frequency (e.g., 3 times) with which the participant's gaze is fixed on a specific location in the instructor's virtual space.

Through the above-described configuration, the cognitive training system 100 according to the present invention displays the concentration of each participant through a graphic object in the instructor's virtual space, allowing the instructor to monitor the concentration of each of a plurality of participants. Furthermore, the cognitive training system 100 according to the present invention responds to an input to a graphic object containing information on the concentration of each plurality of participants, and provides the learning situation of the selected participant to the instructor's virtual space, allowing the instructor to determine the concentration level of each participant. Accordingly, the learning situation of each learner can be checked.

Furthermore, the control unit 140 according to the present invention can use the user feedback unit 145 to provide feedback to improve the concentration of the participant 611 through the participant's 611 virtual space.

More specifically, as shown in FIGS. 10A and 10B, the control unit 140 may provide feedback that is distinct from the content 620 through the virtual space of the participant 611 based on the instructor's input.

The control unit 140 according to the present invention responds to the instructor's input while displaying information related to a specific participant through the instructor terminal 120, as shown in FIG. 9A or 9B, and provides feedback through the virtual space of the participant. can do.

According to one embodiment, the control unit 140 may display a preset graphic object (for example, an arrow) through the participant's virtual space in response to the instructor's input through the instructor terminal 120.

For example, as shown in FIG. 10A, the control unit 140 may respond to the instructor's input through the instructor terminal 120 and provide a pop-up window 1011 notifying the content change through the participant's virtual space. there is.

Accordingly, the control unit 140 may provide preset concentration improvement content that is different from the previously provided content 620 through the participant's virtual space. At this time, the concentration improvement content can be understood as content in which the difficulty of the task (or training) included in the already provided content 620 has been lowered, concentration training content, concentration improvement content, learning method guidance content, or virtual space usage guidance content. You can. Concentration improvement content can be understood as content that includes images input by an instructor to improve the concentration of participants.

Content conversion can be made upon request from the instructor's terminal through the participant's virtual space discussed above. If the instructor connected to the participant's virtual space determines that the participant needs learning (concentration, learning how to use, etc.) different from the current learning through a separate virtual space, the instructor will switch the participant's virtual space to the system. You can request it. In this case, the control unit 140 provides learning content according to learning (concentration, learning how to use, etc.) different from the current learning in a separate virtual space for both the instructor and the participant, or for the participant alone, as shown in FIG. 10A. can do.

For example, the control unit 140 may provide preset terminal usage guidance content that is different from the previously provided content 620 through the participant's virtual space. At this time, the terminal usage guide content may be content that teaches how to use the participant terminal 110. In this case, the control unit 140 may store the state of the virtual space currently being provided to the participant 611 in the storage unit 130 before the content provided to the participant 611 is switched. Through this, after the content provided to the participant 611 is converted and learning about the converted content is completed, the control unit 140 provides the previous content to the participant 611 again through the participant terminal 110. can do.

For another example, in response to the instructor's input through the instructor terminal 120, a blinking or highlighting effect may be provided for the content 620 that is already being provided.

Referring to FIG. 10B according to another embodiment, the control unit 140 may output the input voice 1012 through the participant terminal 110 based on the instructor's voice input through the instructor terminal 120. More specifically, the control unit 140 may record the instructor's voice through the instructor terminal 120 and output the recorded voice 1012 through the participant terminal 110 in response to the instructor's input.

Alternatively, the control unit 140 may output sound effects previously input through the participant terminal 110 in response to the instructor's input through the instructor terminal 120.

According to another embodiment (not shown), the control unit 140 responds to the instructor's input through the instructor terminal 120 and provides preset haptic feedback (e.g., vibration) through at least some of the participant terminals 110. ) can be provided.

On the other hand, the control unit 140 according to the present invention can provide the above-described feedback through the virtual space of a plurality of participants whose concentration calculated through eye tracking is less than a preset threshold.

Alternatively, the control unit 140 may provide the above-described feedback through the virtual space of a participant whose concentration calculated through eye tracking decreases by more than a preset value over a preset period of time among a plurality of participants.

Through the above-described configuration, the cognitive training system 100 according to the present invention can respond to a decrease in the participant's concentration by providing feedback to the participant's virtual space according to the instructor's input. Additionally, through this, the cognitive training system 100 can support efficient learning of participants.

Meanwhile, the control unit 140 according to the present invention can receive learning evaluation input for each of the plurality of participants from the instructor through the instructor's virtual space.

More specifically, the control unit 140 according to the present invention evaluates learning for each of the plurality of participants as a result of the instructor monitoring the learning situation and concentration of each of the plurality of participants through the instructor virtual space displayed on the instructor terminal 120. can be input.

Furthermore, the control unit 140 may generate a report including learning evaluation and gaze data input for each of the plurality of participants and provide the report through the virtual space of each of the plurality of participants.

That is, the control unit 140 controls the user feedback unit 145 to generate a report containing at least some of the learning evaluation and gaze data for each of the plurality of participants, and to send the generated report through the virtual space of each participant. can be provided.

Meanwhile, computer-readable media includes all types of recording devices that store data that can be read by a computer system. Examples of computer-readable media include HDD (Hard Disk Drive), SSD (Solid State Disk), SDD (Silicon Disk Drive), ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc. There is.

Furthermore, the computer-readable medium may be a server or cloud storage that includes storage and can be accessed by electronic devices through communication. In this case, the computer can download the program according to the present invention from a server or cloud storage through wired or wireless communication.

Furthermore, in the present invention, the computer described above is an electronic device equipped with a processor, that is, a CPU (Central Processing Unit), and there is no particular limitation on its type.

Meanwhile, the above detailed description should not be construed as restrictive in all respects and should be considered illustrative. The scope of the present invention should be determined by reasonable interpretation of the appended claims, and all changes within the equivalent scope of the present invention are included in the scope of the present invention.

Claims (10)

1. Providing content including a target stimulus and a distracting stimulus to the virtual space of each of the plurality of participants;

   Monitoring the gaze of each of the plurality of participants with respect to the target stimulus and the distracting stimulus based on an eye tracking method;

   Based on the monitoring results, calculating a concentration level for the content for each of the plurality of participants; and

   A cognitive training method based on eye tracking, comprising providing graphic objects classified for each of the plurality of participants in an instructor's virtual space, and providing information about the concentration of each of the plurality of participants to the graphic objects.
2. According to paragraph 1,

   Based on the instructor's input for the graphic object, gaze data obtained as a result of the monitoring for the selected participant, concentration on the content, and at least a portion of the screen at the selected participant's viewpoint are provided through the instructor's virtual space. A cognitive training method based on eye tracking, further comprising the step of:
3. According to paragraph 2,

   In response to the instructor's input through the instructor's virtual space, providing concentration-enhancing content that is distinct from the content through the selected participant's virtual space. A cognitive training method based on eye tracking.
4. According to paragraph 1,

   The step of calculating the concentration level for the content is,

   identifying a first location of the target stimulus and a second location of the distracting stimulus on the content;

   Based on the monitoring results, acquiring gaze data of each of the plurality of participants; and

   A cognitive training method based on eye tracking, further comprising calculating a degree of concentration on the content based on a comparison of the gaze data with the first location and the second location.
5. According to paragraph 4,

   The step of calculating concentration on content based on comparison of the gaze data with the first location and the second location includes:

   determining a first viewpoint at which the gaze of each of the plurality of participants is fixed at the first position;

   For the first time after the first viewpoint is determined, determining a second viewpoint at which the gaze of each of the plurality of participants is moved to the second location;

   For the first time after the second viewpoint is determined, determining a third viewpoint at which the gaze of each of the plurality of participants is fixed to the first position;

   calculating a first period between the first time point and the second time point and a second period between the second time point and the third time point; and

   A cognitive training method based on eye tracking, comprising the step of calculating concentration based on the ratio of the first period and the second period.
6. According to paragraph 1,

   The step of providing the graphic object to the instructor's virtual space is:

   Grouping the plurality of participants into a plurality of groups based on a preset standard; and

   A cognitive training method based on eye tracking, further comprising providing the graphic object including a visually distinguishable mark according to the group through the instructor's virtual space.
7. According to paragraph 1,

   Obtaining the participant's biological signals through the participant's virtual space; and

   Based on the obtained biosignals, calculating a performance score for a task included in the content for each of the plurality of participants, further comprising:

   The graphic object is,

   A cognitive training method based on eye tracking, comprising information on the concentration level and performance score of each of the plurality of participants.
8. In clause 7,

   Analyzing the concentration and the performance score based on preset standards for each of the concentration and the performance score; and

   A cognitive training method based on eye tracking, further comprising the step of switching content provided to the participant based on the analysis results.
9. In a cognitive training system based on eye tracking,

   a communications department designed to communicate with participants and instructors; and

   A control unit providing content including a target stimulus and a distracting stimulus to the virtual space of each of the plurality of participants,

   The control unit,

   Based on an eye tracking method, monitor the gaze of each of the plurality of participants with respect to the target stimulus and the distracting stimulus,

   Based on the monitoring results, the degree of concentration on the content is calculated for each of the plurality of participants,

   A cognitive training system, characterized in that: providing graphic objects classified for each of the plurality of participants in the instructor's virtual space, and providing information about the concentration of each of the plurality of participants to the graphic objects.
10. A program that is executed by one or more processes in an electronic device and can be stored in a computer-readable medium,

    The above program is,

    Providing content including a target stimulus and an interfering stimulus to the virtual space of each of the plurality of participants;

    Monitoring the gaze of each of the plurality of participants with respect to the target stimulus and the distracting stimulus based on an eye tracking method;

    Based on the monitoring results, calculating a concentration level for the content for each of the plurality of participants; and

    A computer comprising instructions for providing graphic objects classified for each of the plurality of participants in the instructor's virtual space and providing information about the concentration of each of the plurality of participants to the graphic objects. A program stored on a readable recording medium.

Images