KR102559461B1 - Apparatus for improving auditory function using user interaction - Google Patents

Abstract

An apparatus for providing step-by-step training and testing for cognitive and auditory function improvement according to the present invention outputs music to a user and trains the user to perform training to improve at least one of the user's cognitive function and auditory function. An output unit for outputting training induction information for inducing the user to perform; an input unit that senses or receives training performance information from the user performing the training; and a processor that calculates training performance accuracy information based on the training performance information and sets training step information of the training based on whether or not the training performance accuracy information is greater than or equal to reference training accuracy information.

Description

Apparatus for improving auditory function using user interaction}

The present invention relates to an apparatus, method, and program for providing step-by-step training and testing for improving cognitive function and auditory function.

Statistically, it has been investigated that about 22% of the Korean population has hearing loss, and about 30% or more of the elderly aged 65 or older are patients with moderate or higher hearing loss who require hearing aids such as hearing aids. In modern society, hearing loss is gradually increasing due to environmental factors such as excessive noise exposure and cell phone use. Cognitive disorders such as dementia along with presbyopia can be cited as medical problems that frequently occur in the elderly. Among them, mild cognitive impairment (MCI) is classified as an intermediate stage between normal aging and dementia as cognitive function, especially memory, is lowered compared to the same age group. The prevalence of MCI is approximately 28%, estimated to be more than a quarter of the population aged 65 years or older. About 10-15% of MCI patients develop dementia, especially Alzheimer's disease, and are classified as a high-risk group. MCI is clinically important because it is the stage in which Alzheimer's can be detected at its earliest stage and is also known to have the best therapeutic effect.

Senile hearing loss occurs due to the degeneration of the peripheral auditory organs with age. The incidence of dementia is increased by 2 to 5 times in hearing-impaired patients. This is because hearing-impaired patients do not hear well even when sound stimuli come in, and the central auditory nerve degenerates, resulting in overall cognitive function deterioration.

Accordingly, there is a need to preemptively conduct tests on auditory and cognitive functions, and systematically provide training to improve auditory and cognitive functions according to the test results.

However, in the prior art, rehabilitation programs are held offline, and thus, in the case of the elderly who have difficulty moving, the accessibility of the rehabilitation programs is low. In addition, in the prior art, there is a problem in that rehabilitation of the elderly having various levels of cognitive function and auditory function is not provided customized to each individual level, and a uniform rehabilitation program that does not consider the level of the elderly is provided.

The present invention provides step-by-step training capable of improving the cognitive function and auditory function in response to the user's cognitive function and auditory function, and can provide retraining of insufficient training by examining the cognitive function and auditory function after training.

The objects of the present invention are not limited to the above-mentioned objects, and other objects and advantages of the present invention not mentioned above can be understood by the following description and will be more clearly understood by the examples of the present invention. It will also be readily apparent that the objects and advantages of the present invention may be realized by means of the instrumentalities and combinations indicated in the claims.

An apparatus for providing step-by-step training and testing for cognitive and auditory function improvement according to the present invention outputs music to a user and trains the user to perform training to improve at least one of the user's cognitive function and auditory function. An output unit for outputting training induction information for inducing the user to perform; an input unit that senses or receives training performance information from the user performing the training; and a processor that calculates training performance accuracy information based on the training performance information and sets training step information of the training based on whether or not the training performance accuracy information is greater than or equal to reference training accuracy information.

Preferably, the processor may control the output unit and the input unit so that the user performs another training among a plurality of training based on whether the training stage information of the training indicates a training change stage.

Preferably, when the training performance accuracy information is greater than or equal to reference training accuracy information and the training stage information of a recently performed training does not indicate a training change stage, the processor sets the training stage information to be increased, and controls the output unit and the input unit so that the user performs training corresponding to the training stage information set upward.

Preferably, the processor may control the output unit and the input unit so that the user re-performs recently performed training when the training performance accuracy information is less than the reference training accuracy information.

Preferably, the processor may control the output unit and the input unit so that the user performs training different from a recently performed training when the training stage information of the training indicates a training change stage.

Preferably, the output unit may output test guidance information for inducing the user to perform a test for checking at least one of the user's cognitive function and auditory function.

Preferably, the input unit may sense or receive test execution information from the user who performs the test in response to the test guidance information.

Preferably, the processor calculates test performance accuracy information and average accuracy information of the test performance accuracy information based on the test performance information, and determines whether or not to re-execute the training based on whether the average accuracy information is greater than or equal to reference test accuracy information.

Preferably, the processor may control the output unit and the input unit so that the user re-performs a training corresponding to a test for which the average accuracy information was calculated when the average accuracy information is less than the reference test accuracy information.

The computer program according to the present invention may be combined with a computer, which is hardware, and stored in a computer-readable recording medium to perform the method.

According to the above, the present invention provides the user with step-by-step training to improve the cognitive and auditory functions optimized for the user's cognitive and auditory functions, and tests the cognitive and auditory functions after training, thereby monitoring the training performance in real time.

1 is a diagram illustrating an example of an apparatus for providing step-by-step training and testing for improving cognitive function and auditory function according to an embodiment of the present invention and a user using the same.
2 is a block diagram of an apparatus for providing step-by-step training and testing for cognitive function and auditory function improvement according to an embodiment of the present invention.
3 is a diagram for explaining a case in which a user performs a first step of music-based multi-task training during training through an apparatus for providing step-by-step training and testing for improving cognitive function and auditory function according to an embodiment of the present invention.
4 is a diagram for explaining a case in which a user performs a second step of music-based multi-task training during training through an apparatus for providing step-by-step training and testing for cognitive function and auditory function improvement according to an embodiment of the present invention.
5 is a diagram for explaining a case in which a user performs a third step of music-based multi-task training during training through an apparatus for providing step-by-step training and testing for cognitive function and auditory function improvement according to an embodiment of the present invention.
6 is a diagram for explaining a case in which a user performs a fourth step of music-based multi-task training during training through an apparatus for providing step-by-step training and testing for cognitive function and auditory function improvement according to an embodiment of the present invention.
FIG. 7 is a diagram for explaining a case in which a user performs musical instrument playing training during training through an apparatus for providing step-by-step training and testing for cognitive function and auditory function improvement according to an embodiment of the present invention.
FIG. 8 is a diagram for explaining a case in which a user performs task performance training in music during training through an apparatus for providing step-by-step training and testing for cognitive function and auditory function improvement according to an embodiment of the present invention.
9 is a diagram for explaining a case in which a user performs singing training during training through an apparatus for providing step-by-step training and testing for cognitive function and auditory function improvement according to an embodiment of the present invention.
FIG. 10 is a diagram for explaining a case in which a user performs instrument discrimination training during training through an apparatus for providing step-by-step training and testing for cognitive function and auditory function improvement according to an embodiment of the present invention.
FIG. 11 is a diagram for explaining a case in which a user performs speech training according to a piece of music during training through an apparatus for providing step-by-step training and testing for cognitive function and auditory function improvement according to an embodiment of the present invention.
12 is a diagram for explaining a case in which a user performs a fifth step of music-based multi-task training during training through an apparatus for providing step-by-step training and testing for cognitive function and auditory function improvement according to an embodiment of the present invention.
13 is a diagram for explaining a case in which a user performs a sixth step of music-based multi-task training during training through an apparatus for providing step-by-step training and testing for cognitive function and auditory function improvement according to an embodiment of the present invention.
14 is a diagram for explaining a case in which a user performs a seventh step of music-based multi-task training during training through an apparatus for providing step-by-step training and testing for improving cognitive function and hearing function according to an embodiment of the present invention.
15 is a diagram for explaining a case in which a user performs an eighth step of music-based multi-task training during training through an apparatus for providing step-by-step training and testing for improving cognitive function and auditory function according to an embodiment of the present invention.
16 is a flowchart illustrating a method of providing step-by-step training and testing for cognitive function and auditory function improvement according to an embodiment of the present invention.

Hereinafter, various embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that this is not intended to limit the present invention to specific embodiments, and includes various modifications, equivalents, and/or alternatives of the embodiments of the present invention. In connection with the description of the drawings, like reference numerals may be used for like elements.

In this document, expressions such as "has", "may have", "includes", or "may include" indicate the presence of a corresponding feature (e.g., a component such as a number, function, operation, or part), and do not exclude the presence of additional features.

In this document, expressions such as "A or B", "at least one of A and/and B", or "one or more of A or/and B" may include all possible combinations of the items listed together. For example, "A or B", "at least one of A and B", or "at least one of A or B" can refer to (1) including at least one A, (2) including at least one B, or (3) including both at least one A and at least one B.

Expressions such as "first", "second", "first", or "second" used in this document may modify various elements, regardless of order and/or importance, and are used to distinguish one element from another element, but do not limit the elements. For example, a first user device and a second user device may represent different user devices regardless of order or importance. For example, without departing from the scope of rights described in this document, a first element may be called a second element, and similarly, the second element may also be renamed to the first element.

When an element (eg, a first element) is referred to as being “(functionally or communicatively) coupled with/to” or “connected to” another element (eg, a second element), it will be understood that the element may be directly coupled to the other element or connected through another element (eg, a third element). On the other hand, when an element (eg, a first element) is referred to as being “directly connected” or “directly connected” to another element (eg, a second element), it may be understood that no other element (eg, a third element) exists between the element and the other element.

As used herein, the expression "configured to" may be used interchangeably with, for example, "suitable for", "having the capacity to", "designed to", "adapted to", "made to", or "capable of" depending on the circumstances. The term "configured (or set) to" may not necessarily mean only "specifically designed to" hardware. Instead, in some contexts, the phrase "a device configured to" may mean that the device is "capable of" in conjunction with other devices or components. For example, the phrase "a processor configured (or set) to perform A, B, and C" may refer to a dedicated processor (e.g., embedded processor, MCU) to perform those operations, or a generic-purpose processor (e.g., CPU, AP) capable of performing those operations by executing one or more software programs stored in memory.

Terms used in this document are only used to describe a specific embodiment, and may not be intended to limit the scope of other embodiments. Singular expressions may include plural expressions unless the context clearly dictates otherwise. Terms used herein, including technical or scientific terms, may have the same meaning as commonly understood by a person of ordinary skill in the technical field described in this document. Among the terms used in this document, terms defined in general dictionaries may be interpreted as having the same or similar meanings as those in the context of the related art, and unless explicitly defined in this document, ideal or excessively formal meanings. In some cases, even terms defined in this document cannot be interpreted to exclude the embodiments of this document.

Hereinafter, an apparatus for providing step-by-step training and testing for cognitive function and auditory function improvement will be described in detail with reference to the drawings.

1 is a diagram showing an example of an apparatus for providing step-by-step training and examination for improving cognitive function and hearing function according to an embodiment of the present invention and an example of a user using the same, and FIG.

1 and 2, an apparatus 100 for providing step-by-step training and testing for improving cognitive function and hearing function according to an embodiment of the present invention may include an output unit 110, an input unit 120, a processor 130, a communication unit 140, and a storage unit 150. Among the components of the apparatus 100 for providing step-by-step training and examination for cognitive and auditory function improvement according to the present invention, only the components related to the process of training the user's hearing and cognitive functions, examining the hearing and cognitive functions, and providing training results and test results will be described. Therefore, those skilled in the art related to the apparatus 100 for providing step-by-step training and examination for cognitive function and auditory function improvement that other general-purpose components may be further included in addition to the components shown in FIG. 2 Those with ordinary knowledge can understand.

According to an embodiment of the present invention, the apparatus 100 for providing step-by-step training and examination for improving cognitive function and hearing function is controlled and manipulated by a user, can be mounted in a certain place or carried, plays a sound source or outputs a signal for reproducing a sound source, and displays and receives various information for training and examination. At this time, being portable means that there is no inconvenience in holding or carrying the device 100, which provides step-by-step training and testing for cognitive function and auditory function improvement. Also, a device for reproducing a sound source refers to a device that generates sound audible by a user from data stored in the form of a digital or analog signal or outputs a signal that generates a sound audible by a user. Examples of the device 100 that provides step-by-step training and testing to improve cognitive and auditory functions include a personal computer (PC), mobile phone, personal digital assistant, MPEG Audio layer-3 Player, Compact Disc player, and Portable Media Player. It includes all devices that display information that a user can watch, or that senses and receives information from a user.

According to an embodiment of the present invention, the apparatus 100 for providing step-by-step training and testing for improving cognitive function and auditory function may allow a user to first perform a screening test for cognitive function and auditory function.

Here, the screening test may be one or more of pure-tone audiometry, Korean-Montreal-Cognitive-Assessment (K-MoCA), and Korean version of Mini-Mental Status Examination (MMSE-K).

Thereafter, the apparatus 100 for providing step-by-step training and testing for improving cognitive and auditory functions according to an embodiment of the present invention may perform the first to fourth steps of training for improving cognitive and auditory functions.

At this time, the apparatus 100 for providing step-by-step training and testing for cognitive function and auditory function improvement according to an embodiment of the present invention sets the training level to the next level when the performance accuracy information at each training level is equal to or greater than the reference training accuracy information, and allows the user to perform the training. Here, the apparatus 100 for providing step-by-step training and examination for cognitive function and auditory function improvement according to an embodiment of the present invention may provide a plurality of training included in the first training category and a plurality of training included in the second training category, each of the plurality of training corresponding to the first training category may consist of first to fourth steps, and each of the plurality of training corresponding to the second training category may consist of fifth to eighth steps.

According to an embodiment of the present invention, the apparatus 100 for providing step-by-step training and examination for improving cognitive function and auditory function can check whether the training performance accuracy information is equal to or greater than the reference training accuracy information whenever the user completes each step for any one of a plurality of trainings included in each of the first and second training categories, and if the training performance accuracy information is equal to or greater than the reference training accuracy information, the level can be increased so that the user can perform training in the next step.

According to an embodiment of the present invention, the apparatus 100 for providing step-by-step training and testing for improving cognitive function and auditory function allows the user to perform other training that has not been performed among the plurality of trainings included in the first training category when the user has performed all steps of any one of a plurality of trainings included in the first training category.

In addition, the apparatus 100 for providing step-by-step training and testing for cognitive function and auditory function improvement according to an embodiment of the present invention allows the user to perform other training that has not been performed among the plurality of trainings included in the second training category when the user has performed all steps of any one of a plurality of trainings included in the second training category.

Here, the plurality of trainings may be music-based multi-task training, instrument playing training, task performance training in music, singing training, instrument discrimination training, and speaking training in music.

To this end, if the training is music-based multi-task training, the output unit 110 of the apparatus 100 for providing step-by-step training and testing for improving cognitive function and auditory function according to an embodiment of the present invention can output music and sheet music information corresponding to music or output music and hand image information as training guidance information, as shown in FIGS. 3 to 6.

In order to perform the above functions, the output unit 110 may include a sound output module and a display module.

Then, if the training is music-based multi-task training, the input unit 120 of the apparatus 100 for providing step-by-step training and testing for cognitive function and auditory function improvement according to an embodiment of the present invention may sense motion information of one or more limbs of the user's hand and user's foot or sensor the shape information of the user's hand to generate training performance information, as shown in FIGS. 3 to 6.

In order to perform the above function, the input unit 120 may include a gyro sensor module that senses movements of the user's hands and feet, a motion pressure sensor module, and a camera module that captures the shapes of the user's hands and feet.

If the training is music-based multi-task training, the processor 130 of the apparatus 100 for providing step-by-step training and testing for cognitive function and auditory function improvement according to an embodiment of the present invention may calculate training performance accuracy information based on the degree of correspondence between sheet music information or hand image information and limb movement information or training performance information.

Referring further to FIG. 3, when the training is tapping, which is the first step of music-based multi-task training, the output unit 110 displays sheet music information along with music as training guidance information, and guides the user to tap their fingers according to the progress of the music.

In this case, the input unit 120 may sense finger movement information as training performance information whenever a user taps a finger.

The processor 130 may calculate training performance accuracy information based on the time difference between the time of tapping and the beat of the note through the finger motion information.

In addition, the processor 130 may control the output unit 110 to display training performance accuracy information corresponding to the time point of a beat of a musical note as color information or numerical information in real time.

When the output of the music ends, the processor 130 may display training performance accuracy information for the entire time the music is output.

If the training performance accuracy information is less than the reference training accuracy information (eg, accuracy of 80%), the processor 130 may allow the user to retrain the first stage of music-based multi-task training.

Meanwhile, the processor 130 allows the user to perform the next training step if the training performance accuracy information is equal to or greater than the reference training accuracy information, but allows the user to perform other training when the latest training step is the fourth step.

Referring further to FIG. 4 , when the training is clapping, which is the second step of music-based multi-task training, the output unit 110 displays sheet music information along with music as training guidance information, and induces the user to clap as the music progresses.

In this case, the input unit 120 may sense whether or not the user claps information as training performance information whenever the user claps.

The processor 130 may calculate training performance accuracy information based on the time difference between the timing of the clap and the beat of the musical note based on the clap information.

In addition, the processor 130 may control the output unit 110 to display training performance accuracy information corresponding to the time point of a beat of a musical note as color information or numerical information in real time.

When the output of the music ends, the processor 130 may display training performance accuracy information for the entire time the music is output.

If the training performance accuracy information is less than the reference training accuracy information (eg, accuracy of 80%), the processor 130 may allow the user to retrain the second step of the music-based multi-task training.

Meanwhile, the processor 130 allows the user to perform the next training step if the training performance accuracy information is equal to or greater than the reference training accuracy information, but allows the user to perform other training when the latest training step is the fourth step.

5 , when the training is the third step of music-based multi-task training, namely clap and foot stomp, the output unit 110 displays sheet music information along with music as training guidance information, and induces the user to clap or stomp according to the progress of the music.

At this time, the input unit 120 may sense whether or not the user claps or stomps as training performance information whenever the user claps or stomps.

The processor 130 may calculate training execution accuracy information based on the time difference between the timing of the clap or foot stomp and the timing of the beat of the note through the clap information or the foot stomp information.

In addition, the processor 130 may control the output unit 110 to display training performance accuracy information corresponding to the time point of a beat of a musical note as color information or numerical information in real time.

When the output of the music ends, the processor 130 may display training performance accuracy information for the entire time the music is output.

If the training performance accuracy information is less than the reference training accuracy information (eg, accuracy of 80%), the processor 130 may allow the user to retrain in the third step of the music-based multi-task training.

Meanwhile, the processor 130 allows the user to perform the next training step if the training performance accuracy information is equal to or greater than the reference training accuracy information, but allows the user to perform other training when the latest training step is the fourth step.

Referring further to FIG. 6 , when the training is finger matching, which is the fourth step of music-based multi-task training, the output unit 110 changes and sequentially displays the hand image information that any two of the fingers touch with music as training induction information, and induces the user to touch fingers similarly to the hand shape of the hand image information.

At this time, the input unit 120 may sense the user's hand shape information as training performance information whenever the hand image information is changed.

The processor 130 may calculate training performance accuracy information based on a shape difference between the output hand image information and the sensed user's hand shape information.

Also, the processor 130 may control the output unit 110 to display training performance accuracy information as text information whenever hand image information is changed.

When the output of the music ends, the processor 130 may display training performance accuracy information for the entire time the music is output.

If the training performance accuracy information is less than the reference training accuracy information (eg, accuracy of 80%), the processor 130 may allow the user to retrain in the fourth step of the music-based multi-task training.

Meanwhile, the processor 130 allows the user to perform the next training step if the training performance accuracy information is equal to or greater than the reference training accuracy information, but allows the user to perform other training when the latest training step is the fourth step.

If the training is musical instrument playing training, the output unit 110 of the apparatus 100 for providing step-by-step training and testing for cognitive function and auditory function improvement according to an embodiment of the present invention may output music, sheet music information corresponding to the music, and instrument image information as training guidance information.

In addition, if the training is musical instrument playing training, the input unit 120 of the apparatus 100 for providing step-by-step training and testing for cognitive function and auditory function improvement according to an embodiment of the present invention may sense instrument movement information of a virtual instrument virtually played by a user and generate training performance information.

To this end, the input unit 120 may include or be connected to a virtual instrument including a motion sensor module and an acceleration sensor module.

If the training is musical instrument playing training, the processor 130 of the apparatus 100 for providing step-by-step training and examination for improving cognitive function and auditory function according to an embodiment of the present invention may calculate training performance accuracy information based on the degree of correspondence between musical score information, musical instrument image information, and musical instrument movement information.

7 , when the training is musical instrument playing training, the output unit 110 displays sheet music information along with music (instrument sound) corresponding to the type of instrument selected by the user as training induction information, and induces the user to play the virtual instrument according to the progress of the music.

At this time, the output unit 110 can quickly set the bpm of the output music as the musical instrument playing training level increases. For example, the output unit 110 may set the bpm of music to 70 in the first stage of musical instrument playing training, and increase the bpm by 10 whenever one stage is raised.

In this case, the input unit 120 may sense musical instrument movement information as training performance information whenever the user plays the virtual musical instrument, that is, moves the musical instrument.

The processor 130 may calculate training performance accuracy information based on the time difference between the movement time of the virtual musical instrument and the beat time of the musical note through the musical instrument motion information.

In addition, the processor 130 may control the output unit 110 to display training performance accuracy information corresponding to the time point of a beat of a musical note as color information or numerical information in real time.

When the output of the music ends, the processor 130 may display training performance accuracy information for the entire time the music is output.

If the training performance accuracy information is less than the reference training accuracy information (eg, 80% accuracy), the processor 130 may allow the user to retrain the current stage of musical instrument performance training.

Meanwhile, the processor 130 allows the user to perform the next training step if the training performance accuracy information is equal to or greater than the reference training accuracy information, but allows the user to perform other training when the latest training step is the fourth step.

According to an embodiment of the present invention, the output unit 110 of the apparatus 100 for providing step-by-step training and examination for cognitive function and auditory function improvement may output music, figure selection guidance voice, and example figure images as training guidance information if the training is task performance training in music.

In addition, if the training is task performance training in music, the input unit 120 of the apparatus 100 for providing step-by-step training and testing for cognitive function and hearing function improvement according to an embodiment of the present invention receives figure selection information from the user and generates training performance information.

To this end, the input unit 120 may include or be connected to a virtual instrument including a motion sensor module and an acceleration sensor module.

If the training is task performance training in music, the processor 130 of the apparatus 100 for providing step-by-step training and testing for improving cognitive function and hearing function according to an embodiment of the present invention may calculate training performance accuracy information based on the degree of correspondence between figure selection induction voice and example figure image and figure selection information.

8 , when the training is task performance training in music, the output unit 110 displays music, figure selection guidance voice, and example figure images as training guidance information, and guides the user to select a figure according to the progress of the music.

In this case, the output unit 110 may increase the level of difficulty of answering the figure selection indicated by the output figure selection guide voice as the level of task performance training in the music increases. For example, the output unit 110 may output a figure selection guidance voice so as to select one figure color in the first step of task performance training in music, three figure colors in the second step, one figure color and one figure type in the third step, and three figure colors and three figure types in the fourth step.

At this time, the input unit 120 may receive figure selection information from the user.

The processor 130 may calculate training execution accuracy information based on a difference between the figure selection information and the correct answer for figure selection indicated by the figure selection guidance voice.

When the output of the music ends, the processor 130 may display training performance accuracy information for the entire time the music is output.

If the training performance accuracy information is less than the reference training accuracy information (eg, 80% accuracy), the processor 130 may cause the user to retrain the current stage of task performance training in music.

Meanwhile, the processor 130 allows the user to perform the next training step if the training performance accuracy information is equal to or greater than the reference training accuracy information, but allows the user to perform other training when the latest training step is the fourth step.

9, when the training is singing training, the output unit 110 displays music (accompaniment of a song), lyric information, and pitch information as training induction information, and induces the user to sing according to the progress of the music.

At this time, the output unit 110 may change the type of music output so that the difficulty of singing increases as the level of singing training increases. For example, the output unit 110 may output music to sing songs in a major key in the first step of singing training, children's songs in a minor key in the second step, adult in a major key in the third step, and adult in a minor key in the fourth step.

At this time, the input unit 120 may receive sound information of the song being sung from the user.

To this end, the input unit 120 may include a voice recognition module.

The processor 130 may calculate training performance accuracy information based on a difference between pitch information of a song and pitch information of a song sung by a user.

When the output of the music ends, the processor 130 may display training performance accuracy information for the entire time the music is output.

The processor 130 may cause the user to retrain the current stage of singing training if the training performance accuracy information is less than the reference training accuracy information (eg, accuracy of 80%).

Meanwhile, the processor 130 allows the user to perform the next training step if the training performance accuracy information is equal to or greater than the reference training accuracy information, but allows the user to perform other training when the latest training step is the fourth step.

Further referring to FIG. 10 , when the training is instrument discrimination training, the output unit 110 may output a musical instrument sound as training induction information, display musical instrument image information, and induce the user to select a musical instrument type according to the musical instrument sound.

In this case, the output unit 110 may change the type and number of output musical instrument sounds so that the level of difficulty in selecting the musical instrument increases as the instrument discrimination training level increases. For example, the output unit 110 may output one musical instrument sound in the first step of the instrument discrimination training, two different instrument sounds of different musical instruments in the second step, three musical instrument sounds of different musical instruments in the third step, and two different musical instrument sounds of the same musical instrument type in the fourth step.

Here, musical instruments may be a classification of musical instruments and may include, for example, wind instruments, percussion instruments, and string instruments. On the other hand, detailed musical instruments are classifications in which types of musical instruments are subdivided in one musical instrument category. For example, musical instrument wind instruments may include woodwind instruments and brass instruments as detailed musical instruments.

At this time, the input unit 120 may receive musical instrument selection information from the user.

The processor 130 may calculate training performance accuracy information based on a difference between the output instrument sound and the instrument selection information input from the user.

If the training performance accuracy information is less than the reference training accuracy information (eg, accuracy of 80%), the processor 130 may allow the user to retrain the current stage of instrument discrimination training.

Meanwhile, the processor 130 allows the user to perform the next training step if the training performance accuracy information is equal to or greater than the reference training accuracy information, but allows the user to perform other training when the latest training step is the fourth step.

11, when the training is speaking along with music, the output unit 110 outputs music as training guidance information, an announcer voice to be spoken by the user, and voice text information indicating the announcer voice, and induces the user to speak along with the announcer's voice.

At this time, the output unit 110 may reduce the ratio (SNR) of the loudness of the announcer's voice to the loudness of the output music so that the level of speaking difficulty increases as the level of speech-in-the-music training increases. For example, the output unit 110 can output music and announcer voice so that the speech-in-music training has a 23dB SNR in the first step, 20dB SNR in the second step, 16dB SNR in the third step, and 13dB SNR in the fourth step.

At this time, the input unit 120 may receive a user voice from the user.

The processor 130 may calculate training performance accuracy information based on a difference between the output announcer voice and the user voice.

The processor 130 may display training performance accuracy information for a predetermined number of announcer voice outputs (10 times) or during music output.

If the training performance accuracy information is less than the reference training accuracy information (eg, accuracy of 80%), the processor 130 may allow the user to retrain the current stage of speech-in-music training.

Meanwhile, the processor 130 allows the user to perform the next training step if the training performance accuracy information is equal to or greater than the reference training accuracy information, but allows the user to perform other training when the latest training step is the fourth step.

Through this, the apparatus 100 for providing step-by-step training and testing for cognitive function and auditory function improvement according to an embodiment of the present invention allows the user to perform the first to fourth steps of music-based multi-task training, musical instrument playing training, task performance training in music, singing training, instrument discrimination training, and speaking along the music set included in the first training category.

Meanwhile, according to an embodiment of the present invention, the apparatus 100 for providing step-by-step training and testing for improving cognitive function and auditory function allows the user to perform two steps for each of a plurality of training steps included in the first training category as a test to check at least one of the user's cognitive function and auditory function.

Thereafter, the apparatus 100 for providing step-by-step training and testing for cognitive function and auditory function improvement according to an embodiment of the present invention calculates average accuracy information of test performance accuracy information according to the user's test performance, and if the average accuracy information is equal to or greater than the reference test accuracy information, the user can perform training included in the second training category.

Conversely, in the apparatus 100 for providing step-by-step training and testing for improving cognitive function and auditory function according to an embodiment of the present invention, if the average accuracy information is less than the reference test accuracy information, the user re-performs the corresponding training.

A process for allowing a user to perform a plurality of exercises included in the second training category in the apparatus 100 for providing step-by-step training and testing for improving cognitive function and auditory function according to an embodiment of the present invention will be described.

Referring further to FIG. 12, when the training is finger separation, which is the fifth step of music-based multi-task training, the output unit 110 displays sheet music information along with music as training guidance information, and the user according to the progress of the music. You can induce to separate your thumb and little finger.

At this time, the input unit 120 may sense finger separation information as training performance information whenever the user separates the thumb and little finger.

The processor 130 may calculate training performance accuracy information based on the time difference between the time of separation and the beat of the note through the finger separation information.

In addition, the processor 130 may control the output unit 110 to display training performance accuracy information corresponding to the time point of a beat of a musical note as color information or numerical information in real time.

When the output of the music ends, the processor 130 may display training performance accuracy information for the entire time the music is output.

If the training performance accuracy information is less than the reference training accuracy information (eg, accuracy of 80%), the processor 130 may allow the user to retrain the fifth step of the music-based multi-task training.

Meanwhile, if the training performance accuracy information is equal to or greater than the reference training accuracy information, the processor 130 allows the user to perform the next training step, but if the most recently performed training step is the eighth step, the user can perform another training.

Referring further to FIG. 13 , when the training is rock paper scissors, which is the sixth step of music-based multi-task training, the output unit 110 displays win/loss information along with the opponent's hand image information as training induction information, and induces the user to play rock paper scissors according to the opponent's hand image information and win/loss information as the music progresses.

In this case, the input unit 120 may sense the user's hand shape information as the user's training performance information whenever an image of the opponent's hand is output.

The processor 130 may determine a win or loss of rock paper scissors based on the output image information of the opponent's hand and the sensed shape information of the user's hand, and may calculate training performance accuracy information based on a difference between the displayed win or loss information and the determined win or loss.

Also, the processor 130 may control the output unit 110 to display training performance accuracy information as text information whenever hand image information is changed.

When the output of the music ends, the processor 130 may display training performance accuracy information for the entire time the music is output.

If the training performance accuracy information is less than the reference training accuracy information (eg, 80% accuracy), the processor 130 may allow the user to retrain the sixth step of the music-based multi-task training.

Meanwhile, if the training performance accuracy information is equal to or greater than the reference training accuracy information, the processor 130 allows the user to perform the next training step, but if the most recently performed training step is the eighth step, the user can perform another training.

Referring further to FIG. 14, when the training is figure drawing, which is the seventh step of music-based multi-task training, the output unit 110 displays figure drawing guide line information along with music as training guidance information, and induces the user to draw figures according to the progress of the music.

In this case, the input unit 120 may sense shape information and position information of the user's hand whenever drawing guide line information is output.

The processor 130 may calculate training performance accuracy information based on a difference between a figure drawing guide line indicated by the figure drawing guide line information and a position of the user's hand.

Also, the processor 130 may control the output unit 110 to display performance accuracy information as color information or text information whenever figure drawing guide line information is changed.

When the output of the music ends, the processor 130 may display training performance accuracy information for the entire time the music is output.

If the training performance accuracy information is less than the reference training accuracy information (eg, accuracy of 80%), the processor 130 may allow the user to retrain in the seventh step of the music-based multi-task training.

Meanwhile, the processor 130 allows the user to perform the next training step if the training performance accuracy information is equal to or greater than the reference training accuracy information, but allows the user to perform other training when the latest training step is 8 steps.

15, when the training is the eighth step of music-based multi-task training, i.e. following complicated hand gestures, the output unit 110 displays hand gesture image information along with music as training guidance information, and induces the user to make hand gestures that are similar to the hand gesture images according to the progress of the music.

In this case, the input unit 120 may sense the shape information of the user's hand whenever hand motion image information is output.

The processor 130 may calculate training performance accuracy information based on a difference between a hand gesture represented by the hand gesture image information and a hand gesture represented by the sensed shape information.

Also, the processor 130 may control the output unit 110 to display performance accuracy information as color information or text information whenever hand motion image information is changed.

When the output of the music ends, the processor 130 may display training performance accuracy information for the entire time the music is output.

If the training performance accuracy information is less than the reference training accuracy information (eg, accuracy of 80%), the processor 130 may allow the user to retrain the eighth step of the music-based multi-task training.

Meanwhile, the processor 130 allows the user to perform the next training step if the training performance accuracy information is equal to or greater than the reference training accuracy information, but allows the user to perform other training when the latest training step is 8 steps.

According to an embodiment of the present invention, the apparatus 100 for providing step-by-step training and testing for cognitive function and auditory function improvement may allow the user to perform the fifth to eighth steps of musical instrument playing training.

At this time, the output unit 110 can quickly set the bpm of the output music as the musical instrument playing training level increases. For example, the output unit 110 may set the bpm of music to 80 in the fifth stage of musical instrument playing training, and increase the bpm by 10 whenever one stage is raised.

If the training performance accuracy information is less than the reference training accuracy information (eg, 80% accuracy), the processor 130 may allow the user to retrain the current stage of musical instrument performance training.

Meanwhile, if the training performance accuracy information is equal to or greater than the reference training accuracy information, the processor 130 allows the user to perform the next training step, but if the most recently performed training step is the eighth step, the user can perform another training.

According to an embodiment of the present invention, the apparatus 100 for providing step-by-step training and testing for cognitive function and auditory function improvement may allow a user to perform fifth to eighth steps of task performance training in music.

At this time, the output unit 110 outputs 5 figure colors and 5 figure types excluding the specific figure color and specific figure type in the fifth step of task performance training in music, 5 figure colors and 5 figure types excluding the specific figure color and specific figure type in the sixth step (three figure colors and 3 figure types are continuously output), and in the seventh step, 5 card colors, 5 figure types, and 5 figure colors , In the eighth step, a figure selection guidance voice may be output to select 5 card colors, 5 figure types, and 5 figure colors (three figure colors and three figure types are continuously output).

If the training performance accuracy information is less than the reference training accuracy information (eg, 80% accuracy), the processor 130 may allow the user to retrain the current stage of musical instrument performance training.

Meanwhile, if the training performance accuracy information is equal to or greater than the reference training accuracy information, the processor 130 allows the user to perform the next training step, but if the most recently performed training step is the eighth step, the user can perform another training.

According to an embodiment of the present invention, the apparatus 100 for providing step-by-step training and testing for cognitive function and auditory function improvement may allow the user to perform the fifth to eighth steps of singing training.

At this time, the output unit 110 may change the type of music output so that the difficulty of singing increases as the level of singing training increases. For example, the output unit 110 may output music to sing the latest songs in major keys, minor keys, and key keys, respectively, in step 8.

If the training performance accuracy information is less than the reference training accuracy information (eg, 80% accuracy), the processor 130 may allow the user to retrain the current stage of musical instrument performance training.

Meanwhile, if the training performance accuracy information is equal to or greater than the reference training accuracy information, the processor 130 allows the user to perform the next training step, but if the most recently performed training step is the eighth step, the user can perform another training.

According to an embodiment of the present invention, the apparatus 100 for providing step-by-step training and testing for cognitive function and auditory function improvement may allow the user to perform the fifth to eighth steps of instrument discrimination training.

In this case, the output unit 110 may change the type and number of output musical instrument sounds so that the level of difficulty in selecting the musical instrument increases as the instrument discrimination training level increases. For example, the output unit 110 may output three musical instrument sounds of different musical instruments in the fifth step of the instrument discrimination training, two musical instrument sounds of the same detailed musical instrument in the sixth step, three musical instrument sounds of the same detailed musical instrument in the seventh step, and four musical instrument sounds of different musical instruments in the eighth step.

If the training performance accuracy information is less than the reference training accuracy information (eg, 80% accuracy), the processor 130 may allow the user to retrain the current stage of musical instrument performance training.

Meanwhile, if the training performance accuracy information is equal to or greater than the reference training accuracy information, the processor 130 allows the user to perform the next training step, but if the most recently performed training step is the eighth step, the user can perform another training.

According to an embodiment of the present invention, the apparatus 100 for providing step-by-step training and testing for cognitive function and auditory function improvement may allow a user to perform the fifth to eighth steps of speech-in-music training.

At this time, the output unit 110 may reduce the ratio (SNR) of the loudness of the announcer's voice to the loudness of the output music so that the level of speaking difficulty increases as the level of speech-in-the-music training increases. For example, the output unit 110 can output music and an announcer's voice so that 10dB SNR is 10dB SNR in the 5th step, 6dB SNR is 6dB SNR in the 6th step, 3dB SNR is 3dB SNR in the 7th step, and 0dB SNR is 0dB SNR in the 8th step.

If the training performance accuracy information is less than the reference training accuracy information (eg, 80% accuracy), the processor 130 may allow the user to retrain the current stage of musical instrument performance training.

Meanwhile, if the training performance accuracy information is equal to or greater than the reference training accuracy information, the processor 130 allows the user to perform the next training step, but if the most recently performed training step is the eighth step, the user can perform another training.

Through this, the apparatus 100 for providing step-by-step training and testing for cognitive function and auditory function improvement according to an embodiment of the present invention allows the user to perform the fifth to eighth steps of music-based multi-task training, musical instrument playing training, task performance training in music, singing training, instrument discrimination training, and speaking along the music set, respectively, included in the second training category.

Meanwhile, the apparatus 100 for providing step-by-step training and testing for improving cognitive function and hearing function according to an embodiment of the present invention allows the user to perform two steps for each of a plurality of training steps included in the second training category as a test to check at least one of the user's cognitive function and auditory function.

Thereafter, the apparatus 100 for providing step-by-step training and testing for improving cognitive function and auditory function according to an embodiment of the present invention calculates average accuracy information of test performance accuracy information according to the user's test performance, and if the average accuracy information is equal to or greater than the reference test accuracy information, training and testing are terminated.

Conversely, in the apparatus 100 for providing step-by-step training and testing for improving cognitive function and auditory function according to an embodiment of the present invention, if the average accuracy information is less than the reference test accuracy information, the user re-performs the corresponding training.

The communication unit 140 transmits and receives data between the device 100 providing step-by-step training and testing for improving cognitive function and auditory function and an external device. Depending on the use environment of the device 100 for providing step-by-step training and testing for cognitive and auditory function improvement, the communication unit 140 may not exist in the device 100 for providing step-by-step training and testing for cognitive and auditory function improvement.

The storage unit 150 is a conventional storage medium, and those skilled in the art related to the apparatus 100 for providing step-by-step training and testing for cognitive function to test hearing and auditory function improvement can know that the storage unit 150 includes a hard disk drive (HDD), a read only memory (ROM), a random access memory (RAM), a flash memory, and a memory card. .

On the other hand, the apparatus 100 for providing step-by-step training and testing for cognitive and auditory function improvement according to the present invention outputs music to the user and trains the user to improve one or more of the user's cognitive function and auditory function. An output unit 110 that outputs training induction information to induce the user to perform; an input unit 120 that senses or receives training performance information from a user performing training; and a processor 130 that calculates training performance accuracy information based on the training performance information and sets training stage information of training based on whether or not the training performance accuracy information is equal to or greater than the reference training accuracy information.

The processor 130 may control the output unit 110 and the input unit 120 so that the user performs another training among a plurality of training based on whether the training stage information of the training indicates a training change stage.

When the training performance accuracy information is greater than or equal to the reference training accuracy information and the training stage information of the most recently performed training does not indicate a training change stage, the processor 130 sets the training stage information upward and controls the output unit 110 and the input unit 120 so that the user performs training corresponding to the upwardly set training stage information.

The processor 130 may control the output unit 110 and the input unit 120 so that the user re-performs the recently performed training when the training performance accuracy information is less than the reference training accuracy information.

The processor 130 may control the output unit 110 and the input unit 120 to allow the user to perform training different from the recently performed training when training stage information of training indicates a training change stage.

The output unit 110 may output test guidance information for inducing the user to perform a test for checking one or more of the user's cognitive function and auditory function.

The input unit 120 may sense or receive test execution information from a user performing an examination corresponding to the examination guidance information.

The processor 130 may calculate test performance information and average accuracy information of the test performance information based on the test performance information, and determine whether to re-execute the training based on whether the average accuracy information is greater than or equal to the reference test accuracy information.

If the average accuracy information is less than the reference test accuracy information, the processor 130 may control the output unit 110 and the input unit 120 so that the user re-performs the training corresponding to the test for which the average accuracy information was calculated.

On the other hand, a method for providing step-by-step training and testing for cognitive and auditory function improvement according to the present invention includes an output unit outputting music to a user and training to improve one or more of the user's cognitive function and auditory function. Outputting training induction information that induces the user to perform training; sensing or receiving training performance information from a user performing training by an input unit; and calculating, by a processor, training performance accuracy information based on the training performance information, and setting training stage information of training based on whether or not the training performance accuracy information is greater than or equal to reference training accuracy information.

The method for providing step-by-step training and testing for improving cognitive function and auditory function according to the present invention may further include controlling the output unit 110 and the input unit 120 so that the user performs another training among a plurality of training based on whether the training stage information of the training indicates a training change stage.

A method for providing step-by-step training and testing for cognitive and auditory function improvement according to the present invention includes outputting test guidance information for inducing a user to perform a test to check at least one of the user's cognitive and auditory functions; sensing or receiving test execution information from a user performing an examination in response to the examination guidance information by an input unit; and calculating, by a processor, test performance accuracy information and average accuracy information of the test performance accuracy information based on the test performance information, and determining whether to re-perform the training based on whether the average accuracy information is greater than or equal to the reference test accuracy information.

On the other hand, according to another embodiment of the present invention, the apparatus 100 for providing step-by-step training and testing for improving cognitive function and auditory function outputs music to the user and trains the user to improve the user's cognitive function. an input unit 120 that senses or receives training performance information from a user performing training; and a processor 130 that calculates training performance accuracy information based on the training performance information and sets training stage information of training based on whether or not the training performance accuracy information is equal to or greater than the reference training accuracy information.

The processor 130 may control the output unit 110 and the input unit 120 so that the user performs another training among a plurality of training based on whether the training stage information of the training indicates a training change stage.

Training may include music-based multitasking training, instrument playing training, and task performance training in music.

When the training performance accuracy information is greater than or equal to the reference training accuracy information and the training stage information of the most recently performed training does not indicate a training change stage, the processor 130 sets the training stage information upward and controls the output unit 110 and the input unit 120 so that the user performs training corresponding to the upwardly set training stage information.

If the training is music-based multi-task training, the output unit 110 may output music and sheet music information corresponding to the music or output music and hand image information as training induction information.

If the training is music-based multi-task training, the input unit 120 may sense motion information of one or more limbs of the user's hands and feet or sense information on the shape of the user's hands to generate training performance information.

If the training is music-based multi-task training, the processor 130 may calculate training performance accuracy information based on a degree of correspondence between sheet music information or hand image information and limb movement information or training performance information.

If the training is musical instrument playing training, the output unit 110 may output music, sheet music information corresponding to the music, and musical instrument image information as training induction information.

If the training is musical instrument playing training, the input unit 120 may sense musical instrument motion information of a virtual musical instrument virtually played by the user and generate training performance information.

If the training is musical instrument playing training, the processor 130 may calculate training performance accuracy information based on the degree of correspondence between sheet music information, musical instrument image information, and musical instrument motion information.

If the training is task performance training in music, the output unit 110 may output music, figure selection guidance voice, and example figure images as training guidance information.

If the training is task performance training in music, the input unit 120 may receive figure selection information from the user and generate it as training performance information.

If the training is task performance training in music, the processor 130 may calculate training execution accuracy information based on the degree of correspondence between the figure selection guidance voice, the example figure image, and the figure selection information.

The processor 130 may control the output unit 110 and the input unit 120 to allow the user to perform training different from the recently performed training when training stage information of training indicates a training change stage.

Meanwhile, a method for providing step-by-step training and testing for improving cognitive function and auditory function according to another embodiment of the present invention includes an output unit outputting music to a user and training to improve the user's cognitive function. Outputting training induction information that induces the user to perform training; sensing or receiving training performance information from a user performing training by an input unit; and calculating, by a processor, training performance accuracy information based on the training performance information, and setting training stage information of training based on whether or not the training performance accuracy information is greater than or equal to reference training accuracy information.

A method for providing step-by-step training and testing for improving cognitive function and auditory function according to another embodiment of the present invention may further include controlling an output unit and an input unit so that a user performs other training among a plurality of training based on whether the training stage information of the training indicates a training change stage.

So far, the present invention has been mainly looked at with respect to preferred embodiments. Those skilled in the art to which the present invention belongs will understand that the present invention can be implemented in a modified form without departing from the essential characteristics of the present invention. Therefore, the disclosed embodiments should be considered from a descriptive point of view rather than a limiting point of view. The scope of the present invention is shown in the claims rather than the foregoing description, and all differences within the equivalent scope should be construed as being included in the present invention.

As described above, although the present invention has been described by the limited embodiments and drawings, the present invention is not limited thereto, and various modifications and variations are possible within the scope of equivalents of the technical spirit of the present invention and the claims to be described below by those skilled in the art to which the present invention belongs.

Claims (1)

an output unit which outputs music to a user and outputs training induction information for inducing the user to perform training to improve at least one of the user's cognitive function and auditory function;
an input unit that senses or receives training performance information from the user performing the training; and
A processor for calculating training performance accuracy information based on the training performance information and setting training step information of the training based on whether the training performance accuracy information is greater than or equal to reference training accuracy information;
The processor
Controlling the output unit and the input unit so that the user performs another training among a plurality of training based on whether the training stage information of the training indicates a training change stage;
the output section
Outputting test guidance information for inducing the user to perform a test to check at least one of the user's cognitive function and auditory function;
the input part
Sensing or receiving test performance information from the user performing the test in response to the test guidance information;
The processor
Calculate test performance accuracy information and average accuracy information of the test performance accuracy information based on the test performance information, and determine whether to re-execute the training based on whether the average accuracy information is equal to or greater than reference test accuracy information;
the output section
When the training is the 6th step among the 1st to 8th steps of music-based multi-task training, win/loss information is output together with the opponent's hand image information of the rock-paper-scissors game as the training induction information to induce the user to play the rock-paper-scissors game according to the opponent's hand image information and the win/loss information,
the input part
When the training is the 6th step among the 1st to 8th steps of music-based multi-task training, whenever the other hand image is output, the user's hand shape information is sensed as the training performance information of the user to generate the training performance information;
The processor
When the training is the 6th step among the 1st to 8th steps of music-based multi-task training, the outcome of the rock-paper-scissors game is determined based on the output image information of the opponent's hand and the sensed shape information of the user's hand, and the training performance accuracy information is calculated based on the difference between the displayed win/loss information and the determined win/loss;
the output section
When the training is singing training, outputting music, lyric information, and pitch information as the training guidance information, and inducing the user to sing according to the progress of the music,
the output section
If the training is the singing training, the type of music output is changed so that the singing difficulty increases as the singing training step increases, but in the first stage of the singing training, the nursery rhyme in a major key, in the second stage of the singing training, in the minor key, in the third stage of the singing training, whether it is a major adult, a minor adult in the fourth stage of the singing training, and a major year in the fifth stage of the song training In the sixth stage of the singing training, the popular songs by year in minor keys, in the seventh stage of the singing training, the latest songs in major and minor keys, in the eighth stage of the singing training, the latest songs in major keys, minor keys, and telegrams, outputting the music for the user to sing,
the input part
When the training is the singing training, receiving sound information of the song being sung from the user,
The processor
When the training is the singing training, calculating the training performance accuracy information based on the difference between the output pitch information of the song of the music and the pitch information of the song sung by the user;
the output section
When the training is instrument discrimination training, the sound of the instrument is output as the training guidance information and image information of the instrument is displayed to induce the user to select a type of instrument according to the sound of the instrument;
the output section
When the training is the instrument discrimination training, the type and number of musical instrument sounds output are changed so that the musical instrument selection difficulty increases as the instrument discrimination training step increases, and in the first stage of the instrument discrimination training, one instrument sound, in the second stage of the instrument discrimination training, two different musical instrument sounds, three different musical instrument sounds in the third stage of the instrument discrimination training, and the same instrument but two different musical instruments in the fourth stage of the instrument discrimination training In the fifth stage of the instrument discrimination training, three musical instrument sounds of different musical instruments are output, in the sixth stage of the instrument discrimination training, two musical instrument sounds of the same detailed instrument type, in the seventh stage of the instrument discrimination training, three musical instrument sounds of a detailed musical instrument type, and four musical instrument sounds of different musical instruments in the eighth stage of the instrument discrimination training,
The musical instruments
It is a classification of instruments,
The detailed musical instruments are
It is a subdivided classification of the types of instruments included in one musical instrument,
the input part
When the training is the instrument discrimination training, musical instrument selection information is input from the user;
The processor
When the training is the instrument discrimination training, calculating the training performance accuracy information based on a difference between the output sound of the instrument and the instrument selection information input from the user;
the output section
If the training is musical instrument playing training, output music, sheet music information corresponding to the music, and musical instrument image information as the training guidance information;
the input part
If the training is musical instrument playing training, sensing musical instrument movement information of a virtual musical instrument virtually played by a user to generate the training performance information;
the input part
Characterized in that if the training is task performance training in music, shape selection information is input from the user and generated as the training performance information
Hearing function improvement device using user interaction.