KR101881092B1 - Eye-ear attention system and method for study - Google Patents

Abstract

Disclosed are an audio-video linked recognition training system to prevent lowering of concentration, and a method thereof. According to one embodiment of the present invention, the audio-video linked recognition training system comprises: an eye tracking module mounted on a display which displays visual content, and tracking a user′s gaze to generate eye tracking information; a plurality of auditory modules attached to two or more of upper, lower, left, and right sides of the display and outputting audio content; a content database linking and storing the visual and audio content; and a training unit controlling output of the video and audio content, and using the eye tracking information as a user′s reaction in accordance with output of either or both of the video and audio content to perform recognition training, and evaluating a training result.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an eye-

The present invention relates to an audiovisual and interlocking cognitive training system and method.

Concentration (or attention) refers to the ability to concentrate consciousness on a single occasion, and is basically a required skill in conversation with the other person in social life, and in performing lessons and tasks.

Attention is also a significant problem in the problems that arise after stroke, dementia, and brain damage. This lack of concentration can lead to serious problems in interpersonal relationships as well as learning disabilities.

In recent years, the number of people who are suffering from such attention deterioration is increasing. The number of people suffering from the disease and brain damage due to stroke, which is the second mortality rate, dementia with three times the population increase over 10 years, traumatic brain injury caused by traffic accidents, . In addition, the number of school-age populations with inborn sensory processing and information processing problems, such as an increase in children and adolescents with attention deficit hyperactivity disorder, is increasing. Finally, due to the explosion of smartphone usage rates, there has also been an increase in the deterioration of attention due to multitasking.

In order to solve these social and demographic problems, there is a need for professional training tools to resist the deterioration of attention. Korean Patent Laid-Open No. 10-2015-0122877 discloses a technique for improving concentration using brain waves.

A computer-based cognitive training program is a software therapy tool that enhances and enhances cognitive function by consistently and repeatedly training in a specifically reduced cognitive domain due to cognitive impairment. However, the use of cognitive rehabilitation programs using such computer software is poor. This is due to the lack of scientific and technical programs dedicated to cognitive concentration training that are based on literary grounds.

In general, some of the programs that are suitable for clinical use include special education software, learning software, or game machines that are easily available for treatment. In the end, it is a reality that there is a very lack of specialized and highly skilled computer - based cognitive training programs and contents developed for people with low concentration.

Korean Patent Laid-Open No. 10-2015-0122877 (published on November 11, 2015) - Method and system for improving concentration using neurofeedback

The present invention is to provide an audiovisual and interactive cognitive training system and method capable of IT fusion training to prevent concentration loss through active use of visual and auditory and to enhance the concentration of learning or work.

An object of the present invention is to provide an audiovisual interlocking type cognitive training system and method capable of easily performing concentration training by installing an eye tracking device and an auditory device on a display and utilizing concentration training content.

Other objects of the present invention will become more apparent through the following preferred embodiments.

According to an aspect of the present invention, there is provided an audiovisual interlocked cognitive training system, comprising: an eye tracking module attached to a display for displaying visual content, the eye tracking module for tracking the user's eyes and generating eye tracking information; A plurality of audio modules attached to two or more of upper, lower, right, and left sides of the display and outputting auditory contents; A contents database in which the visual contents and the auditory contents are interlocked and stored; And a training unit for controlling the output of the visual content and the auditory content, performing cognitive training using the eye tracking information in response to one or more of the visual content and the output of the auditory content, and evaluating the training result An audiovisual and interlocking cognitive training system is provided.

And a calibration unit for detecting the position of the user and guiding the movement of the user to a correct position for cognitive training.

Or a calibration unit for detecting the position of the user and adjusting the position to be auditory center by individually adjusting the respective sound magnitudes output from the plurality of audio modules corresponding to the position.

Wherein each of the plurality of audio modules includes an ultrasonic sensor for measuring a distance to the user through ultrasonic waves, wherein the calibration unit sets the distance measured by the ultrasonic sensor as a radius and the center of each of the plurality of audio modules An intersection where a plurality of virtual circles meet can be detected as the position of the user.

Or each of the plurality of audio modules includes a speaker for outputting different sound signals in a plurality of zones, and the calibration unit superimposes the zones in which the sound signals heard by the user are most widely output for each of the audio module The overlapping area can be detected as the position of the user.

Wherein the training unit sets different sounds to be output from each of the plurality of audio modules, and outputs the position information corresponding to one of the different sounds, which is determined as a problem, Cognitive training and evaluation can be performed based on the correspondence of the user's sight line information.

The different sounds may be set to differ in decibel.

Other aspects, features, and advantages will become apparent from the following drawings, claims, and detailed description of the invention.

According to the embodiment of the present invention, it is possible to prevent the deterioration of concentration through active use of sight and hearing, and to perform IT fusion training to enhance the concentration of learning or work.

In addition, since an eye tracking device and an auditory device can be installed on the display and the concentration training content can be utilized, the concentration training can be easily performed, and there is no effect on the place.

1 is a schematic configuration diagram of an audiovisual and interlocked cognitive training system according to an embodiment of the present invention;
2 is a flowchart of an audiovisual and interlocking cognitive training method according to an embodiment of the present invention;
FIG. 3 is a view illustrating an audiovisual and interlocking type cognitive training system according to an exemplary embodiment of the present invention,
4 is a view for explaining the calibration of the user position according to the first embodiment,
5 is a view for explaining the calibration of the user position according to the second embodiment,
FIG. 6 is a flowchart of an audiovisual-linked cognitive training method according to another embodiment of the present invention;
FIGS. 7 and 8 illustrate an example of contents reproduction for audiovisual and interlocking type cognitive training;
FIG. 9 is a view illustrating an audiovisual-linked cognitive training content according to another embodiment of the present invention;
10 is a configuration diagram of an automatic training program interlocked with an evaluation tool executed in an audiovisual and ancillary cognitive training system according to another embodiment of the present invention;

While the invention is susceptible to various modifications and alternative forms, specific embodiments thereof are shown by way of example in the drawings and will herein be described in detail. It is to be understood, however, that the invention is not to be limited to the specific embodiments, but includes all modifications, equivalents, and alternatives falling within the spirit and scope of the invention.

It is to be understood that when an element is referred to as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, . On the other hand, when an element is referred to as being "directly connected" or "directly connected" to another element, it should be understood that there are no other elements in between.

The terms first, second, etc. may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, the terms "comprises" or "having" and the like refer to the presence of stated features, integers, steps, operations, elements, components, or combinations thereof, But do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or combinations thereof.

It is to be understood that the components of the embodiments described with reference to the drawings are not limited to the embodiments and may be embodied in other embodiments without departing from the spirit of the invention. It is to be understood that although the description is omitted, multiple embodiments may be implemented again in one integrated embodiment.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Reference will now be made in detail to the preferred embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to the like elements throughout. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the following description, well-known functions or constructions are not described in detail since they would obscure the invention in unnecessary detail.

2 is a flowchart of an audiovisual and interlocking cognitive training method according to an embodiment of the present invention. FIG. 3 is a flowchart illustrating a cognitive- FIG. 4 is a view for explaining the calibration of the user position according to the first embodiment, and FIG. 5 is a view for explaining the calibration of the user position according to the second embodiment And FIG.

1 to 5 show a display 10, an audiovisual interlocked cognitive training system 100, an eye tracking module 110, a camera unit 112, an audio module 120, 120a to 120c, (Collectively referred to as '122'), ultrasonic sensors 124a to 124d (hereinafter also collectively referred to as '124'), a calibration unit 130, a training unit 140, A database 150 is shown.

The audiovisual and interlocking cognitive training system 100 according to an embodiment of the present invention is characterized in that it utilizes both the sight and the hearing of a user (or a trainee) to prevent a decrease in concentration and to enhance the concentration of learning or work do.

The audiovisual interlocking cognitive training system 100 includes an eye tracking module 110, an auditory module 120, a training unit 140, and a content database 150. [ And may further include a calibration unit 130 as needed.

The eye tracking module 110 and the audio module 120 in the audio-visual interlocking type cognitive training system 100 have a structure that can be attached to the known display 10. Here, the display 10 refers to a display device capable of displaying visual contents such as a computer monitor, a tablet PC, and the like.

The eye tracking module 110 is a module for tracking a user's gaze and includes a camera unit 112 and an eye tracking unit (not shown).

The camera unit 112 photographs a face image of a user located in front of the display 10. [ The face image of the user photographed by the camera unit 112 is transmitted to the eye tracking unit.

The eye tracking unit recognizes the eyeball in the face image of the user and generates eye tracking information by tracking the position the user is viewing through eye tracking. The eye tracking technique is a matter which is obvious to a person skilled in the art to which the present invention belongs, and a detailed description thereof will be omitted.

The audio module 120 is a module for outputting auditory contents, and includes an audio reproducing unit (not shown) and a speaker 122.

The audio reproduction unit reproduces the auditory contents used for the concentration training. The speaker 122 externally outputs a sound signal (sound) corresponding to the auditory contents reproduced by the audio reproducing section.

The auditorium module 120 may be installed at two or more of the upper, lower, right, and left sides of the display 10. In the case of an auditorium module installed under the display 10, the module may be integrated with the eye tracking module 110. That is, the audio tracking module 110 may be equipped with one audio module (that is, an audio reproducing unit and a speaker 122). The calibrating unit 130 may be configured to perform a more accurate training than the case where the user is positioned in front of the display 10 in which the eye tracking module 110 and the plurality of audio modules 120 are installed, And can guide the user to move to an appropriate position (correct position) if the user is not in the proper position.

The guidance for the user may be displayed as a visual message via the display 10 or as an audible message through the speaker 122 of the audio module 120. [ For example, a visual message and / or audible message may include a direction close to the display, a direction away from the display, or a left or right direction parallel to the display.

Or the calibration unit 130 may adjust the size of the sound signal output from the speaker 122 of each auditory module 120 based on the detected user's position so that the audiovisual and interlocking awareness training is performed more accurately .

According to the first embodiment for identifying the user's position, the auditory module 120 may further include an ultrasonic sensor 124. The ultrasonic sensor 124 searches for a user located in front and measures the distance to the user.

In this embodiment, the auditory module 120 may be attached to two or more of the upper, lower, right, and left sides of the display 10. Therefore, it is possible to determine that the user is located at a point where a virtual circle whose distance is measured by a plurality of mutually spaced ultrasonic sensors 124 is a radius, and where a virtual circle meets.

Referring to FIG. 4, when the distance to the user is R1 based on the measurement result of the first ultrasonic sensor 124a and the distance to the user based on the measurement result of the second ultrasonic sensor 124b is R2 Is shown.

In this case, the calibration unit 130 includes a virtual first circle having the first ultrasonic sensor 124a as its center and a radius of R1, a hypothetical first circle having the second ultrasonic sensor 124b as a center, It can be determined that the user 1 is located at the intersection where the two circles meet.

According to the second embodiment for identifying the user's position, the speaker 122 provided in the auditory module 120 may be a directional speaker capable of adjusting the direction of a sound signal to be output.

The first sound signal output through the first speaker 122a may be divided into a plurality of zones and output. For example, as shown in FIG. 5, three zones A1, A2, and A3 may be separately output.

In this case, the first sound signal output for each zone can be set differently. For example, a sound may be output in the area A1, the area A2, and the area A3.

The second sound signal output through the second speaker 122b may be divided into a plurality of zones and output. For example, as shown in FIG. 5, three zones B1, B2, and B3 may be separately output.

In this case, the second sound signal output for each zone can be set differently. For example, it is possible to output a sound of 'A' in the B1 zone, 'I' in the B2 zone, and 'Da' in the B3 zone.

The first sound signal and the second sound signal may be output with a predetermined time interval for each of the separated regions. That is, the first sound signal is output to one of the three zones A1 to A3, and the second sound signal is also output to one of the three zones B1 to B3.

A zone based on the first sound signal and a zone based on the second sound signal may also have overlapped zones A1B3, A2B3, A3B3, A2B2, A3B2 and A3B1.

The user can be informed of the loudest sound among the first sound signal outputted through the first speaker 122a and the second sound signal outputted through the second speaker 122b. The user input can be done through a separate input device (keyboard, mouse, etc.) or a touch screen display. Alternatively, the eye tracking module 110 may be used to move the eyes to the sound examples displayed on the screen, thereby allowing the user input.

In this case, for example, when the user indicates that the user has recognized that the sound 'O' and the sound 'I' are heard the greatest, the calibration unit 130 outputs the sound 'A' It can be determined that the user 1 is located in the A3B2 zone where the corresponding B21 zone is overlapped.

The calibration unit 130 may display a visual message on the display 10 that guides the user movement based on the detected user location or may output an audible message through the speaker 122 to allow the user to move to an appropriate location have.

Or the calibration unit 130 may adjust the size of the sound signal output from the speaker 122 of each auditory module 120 based on the detected position of the user so that the audiovisual interlocking type awareness training is performed more accurately .

In the case of the time content output through the display 10, it may suffice if the user is located in front of the display 10 and the user can see the display 10. Thus, calibration for visual content can be omitted.

However, in the case of auditory contents, since a plurality of auditory modules 120 are arranged and output sound signals separated from each other, the user position is an important factor for efficient concentration training.

In this case, even if the user does not move, the audio signal output from the auditory module 120 closer to the user is reduced based on the user position and the audio signal output from the remote auditory module 120 to the user is increased, The effect can be the same as that located at the center.

For this, the calibration unit 130 may set the magnitude of the sound signal output through the speaker 122 in proportion to the distance between the user position and the audio module 120.

According to the third embodiment for identifying the user's position, the eye tracking module 110 can be used. The user's location can be determined by using the camera unit 112 and the IR sensor (not shown) included in the eye tracking module 110.

The position of the entire head is grasped around the eye of the user analyzed by the i-tracking unit. And calculates the error with the position where the eye tracking is optimized, and generates the necessary movement information of the user according to the error.

Then, the display 10 displays the current state of the user. Together with this, the user is presented with necessary movement information (eg, left / right or front / rear movement distance). The user can move to the optimized position in accordance with the instructions, so that the calibration of the user position can be made.

When the calibration of the user position by the calibration unit 130 is completed, the training unit 140 performs concentration training on the user using the audio-visual and interlocking training contents stored in the content database 150.

The visual content is output via the display 10. And auditory content to be output via the corresponding auditorium module 120.

The user's line of sight according to the output of the visual content or the output of the auditory contents can be detected through the eye tracking module 110. [ Therefore, when the visual line position (or visual line movement) corresponding to the contents of the outputted visual contents / auditory contents is detected, it can be determined that the user is performing the concentration training properly. In other cases, since the user is not performing the concentration training properly, the training unit 140 may alert the user or induce the user's interest through the change of the content.

The contents database 150 stores visual contents and / or auditory contents used for concentration training. And content for calibration for user location calibration can also be stored.

FIG. 6 is a flowchart of an audiovisual-linked cognitive training method according to another embodiment of the present invention, and FIGS. 7 and 8 illustrate contents reproduction for audiovisual-linked cognitive training.

The audiovisual-linked cognitive training evaluation method according to the present embodiment is performed through the audiovisual-linked cognitive training system 100, and can be applied to real three-dimensional (3D) sound training using auditory sense.

3D sound is a very important factor in the training of concentration through hearing. In the past, most of the time, when making 3D sound, the position of sound is arbitrarily adjusted by using C #.

However, in this case, the position of the sound is unclear and it is difficult for an elderly user to clearly determine the position of the sound. For example, it sounds similar, but it is difficult to judge whether it is right or center.

Accordingly, in order to realize a real three-dimensional sound, the eye tracking module 110 and the detachable audio module 120 are required as in the present embodiment.

The eye tracking module 110 is attached to the bottom of the display 10 as shown in Fig. In the eye tracking module 110, a sound reproducing unit and a speaker corresponding to one audio module can be mounted.

The auditorium modules 120 are also attached to the upper and left and right sides of the display 10, respectively.

The eye tracking module 110 and the audio module 120 are connected to each other via a USB port.

In this case, a concentration training method through auditory interlocking with the eye tracking module 110 is shown in the following figures.

So that the predetermined sound is outputted through the auditory module 120 (step S300).

In this case, different sound may be output to each of the plurality of audio modules 120. [ For example, a bird sound may be outputted on the left side, and a beep sound may be outputted on the right side.

A sound relating to a predetermined specific animal (for example, a beast) is output through a visual message through the display 10 or through an auditory message through the audio module 120 before or after the sound output, You can instruct or guide the user to take the part that is supposed to be there.

When the user looks at a predetermined sound output side, the eye tracking module 110 tracks the user's eyes and detects where the user's eyes are staying (step S310).

When a point where the user's eyesight stays for a predetermined time or longer is detected, the user can display the hidden animal at the corresponding point or display the place where the target animal is hidden (Step S320). When the place where the target animal is hiding is matched, a predetermined score is obtained. By repeating this process, it is possible to quantify the concentration training performance while inducing the user's interest.

That is, cognitive training and evaluation can be performed on the basis of the correspondence between the position information corresponding to any one of the different sounds and the user's gaze information derived from the eye tracking information generated by the eye tracking module 110.

In another embodiment, the first speaker 122 may start sound output through coding using C # or the like to move the sound output to another second speaker 122, It is also possible to train to move along. In this case, whether the user's gaze has been properly moved can be confirmed through the eye tracking module 110, and it is also possible to confirm whether or not the concentration training is properly performed.

Or a plurality of audio modules 120 may be interlocked and controlled so that a three-dimensional sound corresponding to the sound source is output while the sound source is moved. In this case, the user's eyes can be moved along with the sound movement effect, so that the concentration training can be performed. In addition, the user may confirm whether the user's gaze moves following the sound movement effect through the eye tracking module 110, and perform evaluation of concentration training according to the result.

In step S300, it is also possible to set the size (decibel) of the sound output at the time of sound output through the plurality of audio modules 120 to be different according to the position.

In this case, the user may try to evaluate the hearing concentration of the user by letting the user take a larger sound output. The user's gazing position can be confirmed through the eye tracking module 110.

Alternatively, the user may be able to evaluate the auditory attention by causing the user to input or select what sound is output at a particular location. For example, when the left speaker outputs the word 'Hangul' at 10 dB and the right speaker outputs the word 'English' at 30 dB, the user may be asked what the sound is from the left. In this case, not only the size of the sound but also the direction of the user can be guided to have high concentration ability by keeping it in mind.

FIG. 9 is a view illustrating audiovisual-linked cognitive training contents according to another embodiment of the present invention.

Referring to Fig. 9, the display 10 displays a clock. The second hand moves every second, and the sound of the second hand moving one by one is outputted through the audio module 120. [

In this case, the user watches the clock carefully, and if the second hand moves a plurality of units instead of one at a time, the user can perform the cognitive training by performing the predetermined response.

For example, in the difficulty level 1, the second hand can be set to be 10 times as long as it moves 4 times at a time, reacting for 3 minutes, and moving 4 times. In the second level of difficulty, the second hand can be set to respond by moving three spaces at a time, while moving for three minutes while moving three spaces to ten. In the 3-step difficulty, you can set the second hand to respond by moving 2 spaces at a time, and 10 times when moving 2 spaces while proceeding for 3 minutes. In the 4-step difficulty, the second hand can respond by moving 2 spaces at a time, and can be set to 10 times while moving for 2 minutes while proceeding for 5 minutes.

The training unit 140 may be able to calculate the concentration training result by comprehensively reflecting the accuracy of the response, the number of times the gaze deviates from the second hand path, and the entire gaze tracking path to the predetermined result calculation algorithm.

10 is a configuration diagram of an automatic training program linked to an evaluation tool executed in an audiovisual and ancillary cognitive training system according to another embodiment of the present invention.

When you click the start evaluation button on the main screen of the automatic training program, evaluation starts. And check the evaluation score according to the evaluation progress. The difficulty level can be adjusted automatically according to the score.

Cognitive training is automatically started according to the difficulty level that is automatically adjusted, and the problem corresponding to the difficulty level is performed.

When the cognitive training at this stage is finished, the evaluation window will open automatically and evaluation will proceed.

You can check your score and move on to the next step or re-train this step, depending on your score. Scores can be automatically saved in the database.

The above-described audio-visual interlocking cognitive training method can be implemented as a computer-readable code on a computer-readable recording medium. The computer-readable recording medium includes all kinds of recording media storing data that can be decoded by a computer system. For example, it may be a ROM (Read Only Memory), a RAM (Random Access Memory), a magnetic tape, a magnetic disk, a flash memory, an optical data storage device, or the like. In addition, the computer-readable recording medium may be distributed and executed in a computer system connected to a computer network, and may be stored and executed as a code readable in a distributed manner.

It will be apparent to those skilled in the art that various modifications and variations can be made in the present invention without departing from the spirit or scope of the invention as defined in the appended claims. It will be understood that the invention may be varied and varied without departing from the scope of the invention.

10: Display 100: Audio-visual interlocking cognitive training system
110: Eye tracking module 112:
120, 120a to 120c: Audio module 122a to 122d: Speaker
124a to 124d: Ultrasonic sensor 130: Calibration unit
140; Training Department 150: Content Database

Claims (7)

As an audio-visual interlocking cognitive training system,
An eye tracking module attached to a display for displaying visual contents and tracking eye lines of a user to generate eye tracking information;
A plurality of audio modules attached to two or more of upper, lower, right, and left sides of the display and outputting auditory contents;
A contents database in which the visual contents and the auditory contents are interlocked and stored;
A training unit for controlling the output of the visual content and the auditory content, performing cognitive training using the eye tracking information as a user response according to one or more outputs of the visual content and the auditory content, and evaluating a training result; And
And a calibration unit for detecting the position of the user and adjusting the position of the user to be the auditory center by individually adjusting the respective sound magnitudes output from the plurality of audio modules corresponding to the position of the user,
Wherein each of the plurality of audio modules includes a speaker capable of adjusting a direction of a sound signal to separately output different sound signals in a plurality of zones,
Wherein the calibration unit detects, as the position of the user, a superimposition zone in which the zones where the sound signals heard by the user are most likely to be superimposed on each other for the plurality of auditorium modules,
Wherein the calibration unit proportionally sets the sound size according to the position of the user and the distance between the plurality of audio modules,
Wherein the training unit sets different sounds to be output from each of the plurality of audio modules,
The cognitive training and evaluation are performed on the basis of the correspondence between the position information corresponding to one of the different sounds determined as the problem and the user's gaze information derived from the eye tracking information generated by the eye tracking module Visual - cognitive cognitive training system.
delete delete delete delete delete The method according to claim 1,
Wherein the different sounds are set to be different in decibels.

Images