KR20220059621A - Electronic apparatus for performing auditory training with music, method, and computer program - Google Patents

Abstract

The present invention relates to an auditory training method for an electronic device. The auditory training method comprises: a detection training step of receiving a first user input for a first query inquiring whether a performance sound is heard; a discrimination training step of receiving a second user input for a second query inquiring whether the rhythms or melodies of two sounds sound different from each other; a confirmation training step of receiving a third user input for a third query requesting to select an instrument corresponding to a sound; an understanding training step of receiving a fourth user input for a fourth query related to a sound including playing sounds of a plurality of musical instruments; and a retraining step of further performing at least one of the detection training step based on the first to fourth user inputs, the discrimination training step, the confirmation training step, and the understanding training step. The present invention provides an electronic device capable of performing auditory training based on various queries using musical instrument sounds.

Description

ELECTRONIC APPARATUS FOR PERFORMING AUDITORY TRAINING WITH MUSIC, METHOD, AND COMPUTER PROGRAM }

The present disclosure relates to an electronic device for performing auditory training, and more particularly, to an electronic device for performing auditory training including a process of detecting, discriminating, confirming, understanding, and the like, a performance sound.

The prevalence of hearing impairment in Korea ranks third after physical disability and visual impairment among 16 disability types, and it is estimated that approximately 6.5% (3,136,110 people) of the total population are deaf (Ministry of Health and Welfare, 2014). This number has been steadily increasing every year since 2005 due to the early detection of congenital hearing loss, the increase in geriatric hearing loss due to the rapid aging of the population, the spread of portable audio devices and the increase in noise-induced hearing loss due to industrialization. is projected to account for about 10% of the total population, similar to developed countries.

Deaf people have difficulties in communicating primarily due to hearing loss, and secondarily, they have economic and psychosocial limitations. However, with the rapid development of engineering and medical technology, the latest hearing aids are equipped with a certain degree of artificial intelligence technology, and cochlear implantation is possible from 12 months of age. Park et al, 2018). Nevertheless, it is still difficult to recognize the other person's speech in an environment where there is everyday noise outside the quiet listening environment, which does not show a significant difference between the normal hearing person and the language recognition ability, and there are many restrictions in pursuing the enjoyment of sound through music (Nelson et al, 2003).

Specifically, the current cochlear cochlear speech processing technique, which acoustically analyzes and encodes sound input from the outside and transmits it as an electrical signal, can deliver most of the envelope information, but the temporal fineness that fills the auditory filter widened due to hearing loss Structural information does not convey detailed information (Hopkins et al, 2008). As a result, even if the temporal microstructure information is not sufficient in a quiet environment, there is no great difficulty in recognizing a language because envelope information is given only by wearing a hearing aid and a cochlear implant, but in a noisy environment, even if envelope information is given Unfortunately, because there is not enough information on the temporal microstructure, it is impossible to understand speech (Moore, 2008), and several factors converge to cause distortion even in music listening, which requires comprehensive recognition (Looi et al, 2012).

On the other hand, interestingly, as the part of the brain that plays a major role in understanding spoken language is known to play a major role in processing music, the need for auditory training through musical instruments is emerging (Gfeller) et al, 2016. Anderson S et al, 2013. Asaridou et al, 2013).

However, technology development for a device or system capable of automatically providing auditory training through the playing sound of a musical instrument has not progressed significantly.

Registered Patent Publication No. 10-20535800000

The present disclosure provides an electronic device capable of performing auditory training based on various queries using sound played by a musical instrument.

The present disclosure provides an electronic device capable of automatically designing an effective retraining process based on various characteristics of a user input and a hearing ability training method.

The objects of the present invention are not limited to the objects mentioned above, and other objects and advantages of the present invention not mentioned may be understood by the following description, and will be more clearly understood by the examples of the present invention. It will also be readily apparent that the objects and advantages of the present invention may be realized by the means and combinations thereof indicated in the appended claims.

The auditory training method of an electronic device according to an embodiment of the present disclosure includes outputting a first sound composed of a performance sound of a musical instrument, displaying a first query for inquiring whether a performance sound is heard, and performing the first query A detection training step of receiving a first user input in response to, outputting a second sound and a third sound each composed of a performance sound of a musical instrument, and at least one of a rhythm and a melody of the second sound and the third sound A discrimination training step of displaying a second query inquiring whether or not sounds different from each other, receiving a second user input in response to the second query, outputting a fourth sound composed of a performance sound of one instrument, and plural Displaying a third query for requesting to select a musical instrument corresponding to the fourth sound from among musical instruments of , and receiving a third user input for selecting at least one of the plurality of musical instruments in response to the third query step, outputting a fifth sound including performance sounds of a plurality of musical instruments, displaying a fourth query related to at least one of the plurality of musical instruments corresponding to the fifth sound, and responding to the fourth query Receiving a user input, an understanding training step, a detection training step, a discrimination training step, a confirmation training based on at least one of the first user input, the second user input, the third user input, and the fourth user input and a retraining step, further performing at least one of the step and the comprehension training step.

In this case, in the retraining step, when the first user input is an incorrect answer, a detection training step, a discrimination training step, a confirmation training step, and an understanding training step are additionally performed, and the first user input is a correct answer and the second user When the input is an incorrect answer, a discrimination training step, a confirmation training step, and an understanding training step are additionally performed, and when the first user input and the second user input are each correct answer and the third user input is an incorrect answer, a confirmation training step; The understanding training step may be additionally performed, and when the first user input, the second user input, and the third user input are each correct answer and the fourth user input is an incorrect answer, the understanding training step may be additionally performed.

The hearing ability training method of the electronic device includes: a detection training step performed in the retraining step according to the difficulty level of at least one of the first to fourth queries and the correct answer of at least one of the first to fourth user inputs; The method may further include adjusting the difficulty of at least one of the discrimination training step, the confirmation training step, and the understanding training step.

In this case, in the discrimination training step, the second query corresponding to the first difficulty level may be displayed. In this case, in the retraining step, if the second user input received according to the second query in the discrimination training step is an incorrect answer, a discrimination training step corresponding to a second difficulty level equal to or lower than the first difficulty level is performed. In the discrimination training step, if the second user input received according to the second query is the correct answer, a discrimination training step corresponding to a third difficulty level higher than or equal to the first difficulty level is additionally performed. can

Here, in the hearing ability training method of the electronic device, when the second user input received according to the second query in the discrimination training step is a correct answer, the time taken for receiving the second user input is determined in the discrimination training step The method may further include comparing an average time required to receive a user input obtained based on the training history. In this case, in the retraining step, if the time taken to receive the second user input is longer than or equal to the average time plus the threshold time, the discrimination training step corresponding to the first difficulty level is additionally performed, , when the time taken to receive the second user input is shorter than the average time plus the threshold time, a discrimination training step corresponding to a difficulty higher than the first difficulty level may be additionally performed.

Meanwhile, in the confirmation training step, a fourth sound composed of a performance sound of one of a plurality of musical instruments corresponding to a plurality of groups including at least one of a keyboard instrument, a wind instrument, a percussion instrument, and a string instrument is output, and the plurality of The third query for requesting to select a musical instrument corresponding to the fourth sound from among musical instruments may be displayed, and a third user input for selecting at least one of the plurality of musical instruments may be received in response to the third query. In this case, in the hearing ability training method of the electronic device, when the third user input received according to the third query in the confirmation training step is an incorrect answer, the instrument selected according to the third user input is applied to the fourth sound It may further include the step of identifying whether it is included in the same group as the corresponding musical instrument. In the retraining step, when the instrument selected according to the third user input is included in the same group as the instrument corresponding to the fourth sound, the confirmation training step is additionally performed a first number of times, and the third When the instrument selected according to the user input is not included in the same group as the instrument corresponding to the fourth sound, the confirmation training step may be additionally performed a second number of times greater than the first number.

Meanwhile, the method for training the performance of the electronic device may further include measuring the level of ambient noise. In the retraining step, at least one of the detection training step, the discrimination training step, the confirmation training step, and the understanding training step may be additionally performed based on the degree of difficulty corresponding to the measured ambient noise level. .

An electronic device according to an embodiment of the present disclosure includes a speaker, a display, and a processor connected to the speaker and the display. The processor outputs a first sound composed of a performance sound of a musical instrument through the speaker, displays a first query for inquiring whether a performance sound is heard through the display, and responds to the first query 1 performing a detection training step of receiving a user input, outputting a second sound and a third sound each composed of a performance sound of a musical instrument through the speaker, and rhythm and melody of the second sound and the third sound A second query for inquiring whether at least one of the sounds is different from each other is displayed through the display, and a discrimination training step of receiving a second user input in response to the second query is performed, and the playing sound of one instrument outputting a fourth sound composed of A confirmation training step of receiving a third user input for selecting at least one of the musical instruments of performing a comprehension training step of displaying a fourth query related to at least one of the plurality of instruments through the display, and receiving a fourth user input in response to the fourth query, the first user input, the first At least one of a detection training step, a discrimination training step, a confirmation training step, and an understanding training step is additionally performed based on at least one of the second user input, the third user input, and the fourth user input.

The electronic device and the auditory training method according to the present disclosure are effective in that it is possible to collect information suitable for each stage of auditory training by outputting the performance sound of a musical instrument in various forms and providing various queries corresponding thereto.

The electronic device and the auditory ability training method according to the present disclosure have an effect of providing an efficient retraining process in which the individual auditory ability of the user is considered according to the characteristics of the user input collected for each step.

1 is a block diagram illustrating a configuration of an electronic device according to an embodiment of the present disclosure;
2 is a flowchart illustrating a method for training an auditory ability of an electronic device according to an embodiment of the present disclosure;
3 is a view for explaining a detection training step performed by an electronic device according to an embodiment of the present disclosure;
4A to 4C are diagrams for explaining a discrimination training step performed by an electronic device according to an embodiment of the present disclosure;
5 is a view for explaining a verification training step performed by an electronic device according to an embodiment of the present disclosure;
6 is a view for explaining an understanding training step performed by an electronic device according to an embodiment of the present disclosure;
7 is an algorithm for explaining a hearing training method according to an embodiment of the present disclosure;
8 is a flowchart for explaining a hearing training method according to an embodiment of the present disclosure, and
9 is a block diagram illustrating a detailed configuration of an electronic device according to various embodiments of the present disclosure;

Prior to describing the present disclosure in detail, a description will be given of the description of the present specification and drawings.

First, terms used in the present specification and claims have been selected in consideration of functions in various embodiments of the present disclosure. However, these terms may vary depending on the intention or legal or technical interpretation of a person skilled in the art, and the emergence of new technology. Also, some terms are arbitrarily selected by the applicant. These terms may be interpreted in the meanings defined herein, and in the absence of specific definitions, they may be interpreted based on the general content of the present specification and common technical common sense in the art.

Also, the same reference numerals or reference numerals in each drawing attached to this specification indicate parts or components that perform substantially the same functions. For convenience of description and understanding, the same reference numbers or reference numerals are used in different embodiments. That is, even though all components having the same reference number are illustrated in a plurality of drawings, the plurality of drawings do not mean one embodiment.

In addition, in this specification and claims, terms including an ordinal number such as “first” and “second” may be used to distinguish between elements. This ordinal number is used to distinguish the same or similar elements from each other, and the meaning of the term should not be construed as limited due to the use of the ordinal number. As an example, the use order or arrangement order of components combined with such an ordinal number should not be limited by the number. If necessary, each ordinal number may be used interchangeably.

In this specification, the singular expression includes the plural expression unless the context clearly dictates otherwise. In the present application, terms such as "comprises" or "consisting of" are intended to designate that the features, numbers, steps, operations, components, parts, or combinations thereof described in the specification exist, and are intended to indicate that one or more other It should be understood that this does not preclude the possibility of addition or presence of features or numbers, steps, operations, components, parts, or combinations thereof.

In an embodiment of the present disclosure, terms such as “module”, “unit”, “part”, etc. are terms for designating a component that performs at least one function or operation, and such component is hardware or software. It may be implemented or implemented as a combination of hardware and software. In addition, a plurality of "modules", "units", "parts", etc. are integrated into at least one module or chip, except when each needs to be implemented in individual specific hardware, and thus at least one processor. can be implemented as

In addition, in an embodiment of the present disclosure, when it is said that a certain part is connected to another part, this includes not only a direct connection but also an indirect connection through another medium. In addition, the meaning that a certain part includes a certain component means that other components may be further included, rather than excluding other components, unless otherwise stated.

1 is a block diagram illustrating a configuration of an electronic device according to an embodiment of the present disclosure.

Referring to FIG. 1 , the electronic device 100 may include a speaker 110 , a display 120 , and a processor 130 .

The electronic device 100 may be implemented as various devices such as a smartphone, a tablet PC, a notebook PC, a desktop PC, a kiosk, and a TV.

The speaker 110 is a configuration for providing a variety of information aurally, and may include various conventional configurations for converting an electrical audio signal to output a sound.

The display 120 is configured to visually provide various information.

The display 120 may be implemented as a liquid crystal display (LCD), a plasma display panel (PDP), an organic light emitting diode (OLED), a transparent OLED (TOLED), a micro LED, etc., but is not limited thereto. It may include various known types of displays.

The display 120 may be implemented in the form of a touch screen capable of sensing a user's touch manipulation, or may be implemented as a flexible display that can be folded or bent.

The processor 130 is a configuration for overall control of the electronic device 100 by being connected to components included in the electronic device 100 .

The processor 130 may perform operations according to various embodiments to be described later by executing at least one instruction stored in the memory of the electronic device 100 .

The processor 130 is a general-purpose processor such as a central processing unit (CPU) and an application processor (AP), a graphics-only processor such as a graphic processing unit (GPU), a vision processing unit (VPU), etc. It may be implemented as an intelligence-only processor or the like. In addition, the processor 140 may include a volatile memory such as SRAM.

2 is a flowchart illustrating a method for training an auditory ability of an electronic device according to an embodiment of the present disclosure.

Referring to FIG. 2 , the processor 130 may perform steps S210 to S250 using the speaker 110 and the display 120 .

Specifically, the processor 130 sequentially performs the detection training (S210), the discrimination training (S220), the confirmation training (S230), and the understanding training (S240), while according to the user input received in each of the above steps. Retraining (S250) may be performed.

Hereinafter, each of the above-described steps will be described in more detail with reference to the drawings.

Detection training ( S210 ) is a step for confirming whether a performance sound is heard by a user who is the target of auditory training.

In the detection training ( S210 ), the processor 130 according to an embodiment of the present disclosure may output a sound composed of a performance sound of a musical instrument through the speaker 110 .

Here, the sound may include a performance sound of an instrument arbitrarily selected from among a plurality of preset musical instruments (eg, piano, organ, flute, clarinet, drum, cymbal, guitar, violin, etc.).

Also, the sound may include a performance sound for playing a song arbitrarily selected from among a plurality of preset songs/songs.

In addition, the processor 130 may display on the display 120 a query to inquire whether a performance sound is heard.

As a result, user input in response to the query may be received.

Here, the user input may correspond to a selection of whether or not a performance sound is heard (eg, yes/no).

In relation to this, FIG. 3 is a diagram for explaining a detection training step performed by an electronic device according to an embodiment of the present disclosure. FIG. 3 assumes that the electronic device 100 is implemented as a tablet PC.

Referring to FIG. 3 , the electronic device 100 may output a sound 310 including a performance sound of a musical instrument.

In addition, the electronic device 100 may display the UI 320 including the query “Do you hear the sound of the instrument playing?”

In this case, the electronic device 100 may receive a user input for selecting “yes” or “no” included in the UI 320 .

As the user input to the UI 320 is received, the processor 130 may proceed to the discrimination training (S220) step.

In this case, before proceeding to the discrimination training ( S220 ), the processor 130 may display information on whether the user input is correct or not.

As a specific example, when a user input of touching “yes” of the UI 320 is received while the sound 310 including the sound of the instrument is output, the processor 130 provides information indicating that the user input is the correct answer (ex. . “Correct answer”, “That’s right. The instrument was playing. It was the sound of the violin playing.”) can be displayed on the display 120 .

Meanwhile, when a user input is not received even after a preset time has elapsed from the time when the UI 320 is displayed, the processor 130 may proceed to the discrimination training step S220 in a state in which the user input is not received.

The discrimination training ( S220 ) is a step for confirming whether the user, who is the target of the auditory training, can distinguish the difference between the played sounds. Specifically, the discrimination training ( S220 ) is a step for confirming whether the user can recognize the sameness/difference between the melody and/or the rhythm of the played sound.

In the discrimination training ( S220 ), the processor 130 may output two sounds each composed of a performance sound of one instrument through the speaker 110 .

Here, each sound may include a performance sound obtained by performing an arbitrarily selected song/song by an arbitrarily selected instrument. In this case, the instruments of the two sounds may be the same or different, and the songs/songs of the two sounds may also be the same or different.

In addition, the processor 130 may display a query to inquire whether at least one of the rhythms and melodies of the two sounds sounds different from each other.

Since rhythm and melody are elements constituting the song (song) itself, the corresponding query may simply consist of a query asking whether the song/song of each sound sounds the same or different.

As a result of the query being displayed in this way, a user input in response to the query may be received.

As a specific example, a user input indicating that the first played song and the second played song are the same/different songs may be received.

Meanwhile, according to another embodiment of the discrimination training (S220) step, the processor 130 may display a query to inquire whether the instruments of the two sounds sound different from each other.

In this case, a user input indicating that the instrument playing the first song and the instrument playing the second song are the same/different instruments may be received.

4A to 4C are diagrams for explaining a discrimination training step performed by an electronic device according to an embodiment of the present disclosure.

Referring to FIG. 4A , the electronic device 100 may output a first sound 411 including a performance sound of a musical instrument.

In addition, the electronic device 100 may display a UI 421 guiding that two performance sounds are output.

Subsequently, referring to FIG. 4B , the electronic device 100 may output a second sound 411 including a performance sound of one instrument.

And, referring to FIG. 4C , the electronic device 100 may display a UI 422 including a query “Is the first played song and the second played song the same?”

However, although the query of FIG. 4C corresponds to a query asking whether the rhythm and melody of two songs are the same regardless of whether the instruments are the same, there are other various applications that test the ability to discriminate between two or more sounds. Of course, you can ask questions.

Here, as the user input is received, “yes”, “no”, “listen again”, etc. may be selected.

When “listen again” is selected, the UI 422 of FIG. 4C may be provided after the process of FIGS. 4A to 4B is performed again.

As the user input to the UI 422 is received, the processor 130 may proceed to the confirmation training (S230) step.

In this case, before proceeding to the confirmation training (S230) step, the processor 130 may display information on whether the user input is correct or not.

As a specific example, it is assumed that the rhythm and melody of the sounds 411 and 412 output in FIGS. 4A and 4B are different from each other.

At this time, when a user input of touching “No” of the UI 420 of FIG. 4C is received, the processor 130 provides information indicating that the user input is the correct answer (eg, “Correct”, “That’s right. First song and two The second song is different.”) may be displayed on the display 120 .

Meanwhile, when a user input is not received even after a preset time has elapsed from the time when the UI 422 is displayed, the processor 130 may proceed to the confirmation training step S230 in a state in which the user input is not received.

In the confirmation training (S230) step, how well the user who is the target of the auditory command understands the specific characteristics of the played sound (eg, what the instrument is, the speed of the rhythm, the rhythm, the number of notes composing the melody, the atmosphere of the melody, etc.) This is a step to check whether or not

In the confirmation training ( S230 ), the processor 130 according to an embodiment of the present disclosure may output a sound composed of a performance sound of one instrument.

The present sound may be a performance sound of one arbitrarily selected instrument from among a plurality of preset musical instruments. A song/song of a performance sound included in the present sound may also be arbitrarily selected.

Here, the processor 130 may control the display 120 to display a query requesting to select a musical instrument corresponding to the output sound from among a plurality of preset musical instruments.

As a result, a user input for selecting at least one of a plurality of preset musical instruments in response to a corresponding query may be received.

As a specific example, the processor 130 may display UI items representing each of a plurality of preset musical instruments. In addition, a user's touch touching at least one of the UI items may be input. As a result, the instrument indicated by the touched UI item may be selected according to the user input.

In relation to this, FIG. 5 is a diagram for explaining a confirmation training step performed by an electronic device according to an embodiment of the present disclosure.

Referring to FIG. 5 , the electronic device 100 may output a sound 510 including a performance sound of a musical instrument through the speaker 110 .

Here, the electronic device 100 may display the UI 520 including a query to inquire whether the output performance sound is of a musical instrument, such as a piano, a violin, a xylophone, or a guitar.

In this case, the electronic device 100 may receive a user input for selecting (touching) one of the musical instrument items included in the UI 520 .

As the user input to the UI 520 is received, the processor 130 may proceed to the understanding training (S240) step.

In this case, before proceeding to the understanding training (S240) step, the processor 130 may display information on whether the user input is correct or not.

As a specific example, it is assumed that the sound 510 output in FIG. 5 includes a piano performance sound.

At this time, if a user input for selecting a violin in the UI 520 of FIG. 5 is received, the processor 130 provides information indicating that the user input is not a correct answer (eg, “not correct answer”, “no. It was the sound of the piano playing.”) may be displayed on the display 120 .

On the other hand, if the user input is not received even after a preset time has elapsed from the time when the UI 520 of FIG. 5 is displayed, the processor 130 may proceed to the understanding training (S240) step in a state in which the user input is not received. there is.

The understanding training ( S240 ) is a step for confirming whether the user who is the target of the auditory training recognizes more in-depth information (eg, harmony) about the performance sound.

In the comprehension training ( S240 ), the processor 130 according to an embodiment of the present disclosure may output a sound including performance sounds of a plurality of musical instruments through the speaker 110 .

Here, the sound may include performance sounds played by different musical instruments according to appropriate harmony.

In this case, the processor 130 may control the display 120 to display a query related to at least one of a plurality of musical instruments of the output sound (played sound).

For example, this query may include a variety of in-depth questions, such as how many instruments are being played, which instrument plays the highest scale, and which instrument plays the lowest scale.

In addition, the processor 130 may receive a user input in response to a corresponding query.

In relation to this, FIG. 6 is a diagram for explaining an understanding training step performed by an electronic device according to an embodiment of the present disclosure.

Referring to FIG. 6 , the electronic device 100 may output a sound 610 including performance sounds played by a plurality of musical instruments.

In addition, the electronic device 100 may display the UI 620 including a query asking how many instruments are being played.

Here, a user input for selecting the number of musical instruments being played may be received.

As the user input to the UI 620 is received, the processor 130 may proceed to the retraining (S250) step.

In this case, before proceeding to the retraining (S250) step, the processor 130 may display information on whether the user input is correct or not.

However, if the user input is not received even after a preset time has elapsed from the time when the UI 620 of FIG. 6 is displayed, the processor 130 may go to the retraining (S250) step in a state in which the user input is not received. there is.

As such, as the performance sound of one or more instruments and the question/answer for the performance sound are used in each stage, auditory training using only relatively intuitive elements (instrument, rhythm, melody) is possible.

In particular, according to prior studies, it has been found that the effect of auditory training on relatively young infants among experimental subjects who received cochlear implantation is greater than the effect of auditory training on subjects in adolescence or older.

Therefore, there is a need for subjects who have undergone cochlear implantation to start receiving auditory training as early as possible from infancy.

Considering this, the fact that the auditory training method according to the present disclosure uses only intuitive elements (instrument, rhythm, melody) according to musical instrument performance is a great advantage in that it can be easily accessed even by subjects at a young age. .

Meanwhile, the retraining step S250 is a step for additionally performing at least one of detection training, discrimination training, confirmation training, and understanding training.

Specifically, the processor 130 may additionally perform at least one training process of detection/discrimination/confirmation/understanding based on at least one of the user inputs received in steps S210 to S240 described above.

According to an embodiment of the present disclosure, the processor 130 may additionally perform at least one step according to whether the received user input is a correct answer.

In relation to this, FIG. 7 is an algorithm for explaining a hearing ability training method according to an embodiment of the present disclosure.

Referring to FIG. 7 , after the above-described steps S210 to S240 are performed, the retraining step ( S2500 ) may additionally perform at least one of steps S210 to S240 according to the determination process of S710 to S740 .

For example, when the user input received in step S210 (detection training) is an incorrect answer (S710 - N), the processor 130 may additionally perform detection training, discrimination training, confirmation training, and understanding training (S210 to S210 to N). S240).

As another example, if the user input received in step S210 is a correct answer (S710 - Y), but the user input received in step S220 (discrimination training) is an incorrect answer (S720 - N), the processor 130 performs discrimination training, confirmation training And understanding training may be additionally performed (S220 to S240).

As another example, if the user input received in steps S210 and S220 is a correct answer (S710 - Y, S720 - Y), but the user input received in step S230 (confirmation training) is an incorrect answer (S730 - N), the processor 130 may additionally perform confirmation training and understanding training (S230 and S240).

As another example, if the user input received in steps S210 to S230 is a correct answer (S710 - Y, S720 - Y, S730 - Y), but the user input received in step S240 (confirmation training) is an incorrect answer (S740 - N), Understanding training may be additionally performed (S240).

Considering the above-described examples of detection, discrimination, confirmation, and understanding training, the later the training is, the higher the difficulty (for example, the difficulty of discrimination training is generally higher than that of detection training), There is an advantage that step-by-step retraining according to the difficulty is possible through the embodiment of 7 .

Meanwhile, unlike FIG. 7 , an embodiment in which all detection/discrimination/confirmation/understanding training is performed in the retraining step ( S250 ) regardless of whether the user inputs received in steps S210 to S240 are correct is also possible.

Meanwhile, according to an embodiment of the present disclosure, the processor 130 determines whether at least one of the user inputs received in steps S210 to S240 is correct and/or the difficulty of at least one of the queries provided in steps S210 to S240. , it is possible to adjust the difficulty of at least one of the detection training step, the discrimination training step, the confirmation training step, and the understanding training step performed in the retraining step ( S250 ).

In relation to this, FIG. 8 is a flowchart illustrating a hearing ability training method according to an embodiment of the present disclosure.

Referring to FIG. 8 , after steps S210 to S240 are performed, the processor 130 may determine the difficulty and/or number of at least one of detection, discrimination, confirmation, and understanding training to be performed in retraining ( S810 ).

And, in the retraining step (S820), the processor 130 may additionally perform at least one of detection, discrimination, confirmation, and understanding training according to the determined difficulty/number of times.

Specifically, the processor 130 may adjust the difficulty of training included in the retraining step S250 according to whether at least one of the user inputs received in steps S210 to S240 is correct.

As an example, it may be assumed that step S220 (discrimination training) corresponds to the first difficulty level. That is, the difficulty level of answering the query provided in step S220 corresponds to the first level of difficulty.

Here, if the user input received according to the query in step S220 is an incorrect answer, in the retraining step ( S250 ), discrimination training may be additionally performed according to a second difficulty level that is lower than or equal to the first difficulty level.

On the other hand, if the user input received according to the query in step S220 is the correct answer, in the retraining step ( S250 ), discrimination training may be additionally performed according to a third difficulty level that is higher than or equal to the first difficulty level.

In addition, the processor 130 may adjust the difficulty of training included in the retraining step ( S250 ) according to the time it takes for at least one of the user inputs received in steps S210 to S240 to be received.

As a specific example, when the user input received according to the query provided in the discrimination training step S220 is the correct answer, the processor 130 receives the user input from the time taken to receive the corresponding user input (eg, from the time the query is displayed). time) can be compared with the average time required to receive user input.

The average time may be obtained based on a training history of one or more discrimination training steps performed in the past. As an example, the processor 130 determines that the user input in the past discrimination training stage through the (same) user's past discrimination training history (eg, time taken to receive user input, correct answer rate, number of training times/frequency, etc.) You can calculate the average of the time it took to be received.

Here, if the time taken to receive the user input in step S220 is longer than or equal to the average time plus the threshold time, in the retraining step (S250), the processor 130 according to the same difficulty as the difficulty of step S220 A discrimination training step may be additionally performed.

On the other hand, if the time taken to receive the user input in step S220 is shorter than the average time plus the threshold time, in the retraining step (S250), the processor 130 performs discrimination training according to the difficulty higher than that of the step S220. Additional steps may be performed.

As such, by performing customized retraining that takes into account not only the correct answer but also the time taken to select the correct answer, there is an effect that optimal hearing training for an individual user can be performed.

Meanwhile, the processor 130 may adjust the number of times of training performed in the retraining step ( S250 ) according to at least one of the user inputs received in steps S210 to S240 .

According to an embodiment, it is assumed that the user inputs received in steps S210, S220, and S240 (detection, discrimination, understanding) are correct answers, but the user input received in steps S230 (confirmation) is an incorrect answer.

In this case, in the retraining step (S250), the processor 130 additionally performs the detection training, the discrimination training, and the understanding training the same number of times, but the confirmation training can be additionally performed more times than the corresponding number. .

Meanwhile, according to an embodiment, it is assumed that the electronic device 100 outputs a sound composed of a performance sound of one instrument among a plurality of instruments corresponding to a plurality of groups in step S230 (confirmation).

The plurality of groups may include, for example, at least one of a keyboard instrument, a wind instrument, a percussion instrument, and a string instrument by being classified according to a sound generation principle, but in addition to various criteria (ex. traditional instrument/modern instrument, oriental instrument/western instrument) may include groups separated by .

In this case, in step S230 , the electronic device 100 displays a query requesting to select a musical instrument that played the output sound (played sound) from among a plurality of musical instruments corresponding to a plurality of groups, and responds to the query (user input). ) can be received.

Here, if the received user input is an incorrect answer (: select another instrument), the processor 130 identifies whether the instrument selected according to the user input is included in the same group as the instrument (correct answer) that played the output sound. can

If the selected instrument is included in the same group as the instrument that played the corresponding sound, in the retraining step S250 , the processor 130 may additionally perform the confirmation training step a first number of times.

For example, it is assumed that the performance sound output in step S230 (confirmation) is a performance sound of a clarinet, and the instrument selected by the user is a saxophone included in the same group (string instrument) as the clarinet. In this case, the processor 130 may perform the confirmation training three times in the retraining step (S250).

On the other hand, if the selected instrument is not included in the same group as the instrument that played the sound, in the retraining step (S250), the processor 130 additionally performs the confirmation training step a second number of times greater than the first number. can

For example, it is assumed that the performance sound output in step S230 (confirmation) is a performance sound of a clarinet, and the instrument selected by the user is a piano belonging to a group different from the clarinet. In this case, the processor 130 may perform the confirmation training 6 times more than 3 times in the retraining step (S250).

As such, when the instrument selected by the user is not the correct answer in the confirmation training step ( S230 ), the electronic device 100 according to the present embodiment determines whether the selected instrument is similar to the played instrument (eg, included in the same group) It can measure the user's cognitive abilities differently and perform the retraining process differently. As a result, the optimal customized retraining process for individual users (audience training subjects) can be automatically and delicately constructed.

Meanwhile, according to an embodiment, it is assumed that in step S240 (understanding), the electronic device 100 outputs a sound composed of playing sounds of a plurality of musical instruments corresponding to the same group.

In addition, in step S240 (understanding), it is assumed that a query is provided to inquire about the number of musical instruments that have played the played sound.

In this case, if the user input in response to the above-mentioned query is the correct answer, in the understanding training process of step S250 (retraining), the processor 130 outputs a sound composed of the playing sounds of a plurality of instruments corresponding to a plurality of groups. and can provide related queries.

On the other hand, if the user input in response to the above-mentioned question is an incorrect answer, in the understanding training process of step S250 (retraining), the processor 130 still outputs a sound composed of the playing sounds of a plurality of instruments corresponding to the same group and , and can provide related queries.

Meanwhile, according to an embodiment, it is assumed that in step S240 (understanding), the electronic device 100 outputs sounds composed of performances of a plurality of musical instruments corresponding to a plurality of groups (eg, two groups).

In addition, in step S240 (understanding), it is assumed that a query is provided to inquire about the number of musical instruments that have played the played sound.

At this time, if the user input in response to the above-mentioned query is the correct answer, in the understanding training process of step S250 (retraining), the processor 130 performs a higher number than the above-described plurality of groups (eg, two groups). A sound composed of playing sounds of a plurality of musical instruments corresponding to a plurality of groups (eg, three groups) may be output, and a related query may be provided.

On the other hand, if the user input in response to the above-mentioned query is an incorrect answer, in the understanding training process of step S250 (retraining), the processor 130 outputs a sound composed of the playing sounds of a plurality of instruments corresponding to the same group, You can provide related queries.

Since the difficulty of comprehension training increases as musical instruments corresponding to various groups are played together, retraining with a difficulty level suitable for the user's cognitive ability may be performed according to the above-described embodiment.

On the other hand, the processor 130 according to an embodiment of the present disclosure may adjust the difficulty of the retraining step (S250) according to the ambient noise.

For example, the processor 130 may measure the level of ambient noise using at least one sensor (eg, a noise sensor (microphone)).

And, in the retraining step (S250), the processor 130 may additionally perform at least one of detection training, discrimination training, confirmation training, and understanding training based on the difficulty corresponding to the measured magnitude of ambient noise. .

Specifically, as the level of ambient noise increases, the processor 130 may lower the difficulty of at least one training additionally performed in the retraining step. Conversely, as the level of ambient noise decreases, the processor 130 may increase the difficulty of at least one training additionally performed in the retraining step.

Considering the general situation of a cochlear implant recipient who has difficulty in distinguishing/recognizing various sounds as the noise increases, the above-described embodiment of adjusting the difficulty according to ambient noise may have a significant advantage in auditory training.

Meanwhile, the sound levels (eg, dB) output in the above-described steps S210 to S250 may be within a sound level range preset according to the user.

In the case of users who have received a cochlear implant and/or wear a hearing aid, since sound recognition characteristics are different for each user, a sound loudness range that is comfortable for each user may be determined according to an expert or a guideline.

In this case, as a user manipulation by the user himself/herself, a guardian, or an expert is input to the electronic device 100 , a sound volume range for the user may be set.

On the other hand, according to an embodiment of the present disclosure, the processor 130 uses at least one artificial intelligence model trained to predict the correct answer for each step of the user to identify the performance of hearing training (improving the user's hearing). can

This artificial intelligence model may be trained to predict whether or not the corresponding user is correct in the auditory training performed thereafter, based on whether the user input is correct for each training stage of the specific user for a preset period.

This artificial intelligence model may be trained through the electronic device 100 and/or at least one external device, and may be stored in the memory of the electronic device 100 as data in the form of a neural network. This artificial intelligence model consists of CNN (Convolutional Neural Network), DNN (Deep Neural Network), RNN (Recurrent Neural Network), RBM (Restricted Boltzmann Machine), DBN (Deep Belief Network), BRDNN (Bidirectional Recurrent Deep Neural Network), GAN It may correspond to (Generateve Adhesive Network) and Deep Q-Networks, but is not limited to the above-described examples.

As an example, the present artificial intelligence model may be trained according to information on each training step performed for a week and whether a user input received in each training step is correct or not.

Information on each training stage includes information on whether each training stage is detection/discrimination/confirmation/understanding, information on the type and number of instruments related to the output sound (played sound), and the performance of the output sound It may include information about the rhythm and melody, information about the output query, and the like.

Here, the artificial intelligence model may be trained by using information about each training step as training data for the input, and using whether the user input received in each training step is correct or not as training data for the output.

When the artificial intelligence model is trained according to the above-described process, the processor 130 may input sound and query information output in the currently performed training stage (eg, discrimination) to the artificial intelligence model.

Here, the artificial intelligence model may determine whether a user input to be received with respect to a corresponding query is a correct answer or an incorrect answer (in this case, output data may be obtained in the form of a correct answer probability and an incorrect answer probability).

In addition, the processor 130 may evaluate the training performance by comparing whether the actual received user input is correct or not and the determination content (correct answer) of the artificial intelligence model.

For example, when the user input is a correct answer even though the artificial intelligence model predicts an incorrect answer, the processor 130 may increase the performance score.

As another example, if the user input is an incorrect answer even though the artificial intelligence model predicts the correct answer, the processor 130 may decrease the performance score.

As another example, when a user input that matches the prediction (correct or incorrect) of the artificial intelligence model is received, the processor 130 may maintain a performance score.

Meanwhile, the processor 130 may additionally train/update the artificial intelligence model according to a user input received while auditory training is continuously performed.

According to various studies in the prior art, it has been known that the effect is more evident when hearing training is performed for a specific user for a relatively long time. However, the specific period or method is not clearly defined. By using the artificial intelligence model as in the above-described embodiment, the user's hearing improvement trend can be clearly explained, and accordingly, important information for deriving the optimal period or optimal method of the individual or universal hearing training process can be collected. It is meaningful to have

On the other hand, the above-described various embodiments may be combined, combined, or applied in various ways within the limits that do not conflict with each other. That is, the hearing ability training method of the electronic device according to the present disclosure is not limited to each of the above-described various embodiments.

9 is a block diagram illustrating a detailed configuration of an electronic device according to various embodiments of the present disclosure;

Referring to FIG. 9 , the electronic device 100 may further include a memory 140 , a user input unit 150 , a communication unit 160 , etc. in addition to the speaker 110 , the display 120 , and the processor 130 .

The memory 140 is a configuration for storing an operating system (OS) for controlling overall operations of the components of the electronic device 100 and at least one instruction or data related to the components of the electronic device 100 . .

The processor 130 may perform operations according to various embodiments of the present disclosure by executing at least one instruction stored in the memory 140 .

As an example, the memory 140 may include information on at least one application for performing the hearing training method according to the present disclosure.

In this case, the processor 130 may perform the hearing ability training method according to various embodiments of the present disclosure by executing the corresponding application.

The user input unit 150 is configured to receive a user's command for the electronic device 100 and/or user information required for the operation of the electronic device 100 .

The user input unit 150 may include various components such as one or more buttons, a touch screen, a microphone, and a camera.

For example, the user input received in each of the above-described training steps ( S210 to S250 ) may be a touch method for a touch screen provided on the display 120 , but is not limited thereto.

The communication unit 160 may include a circuit as a configuration for the electronic device 100 to communicate with at least one external device.

The communication unit 160 may be connected to an external device based on a network implemented through wired communication and/or wireless communication. In this case, the communication unit 160 may be directly connected to an external device, or may be connected to an external electronic device through one or more external servers (eg, Internet Service Providers (ISPs)) that provide a network.

The network may be a personal area network (PAN), a local area network (LAN), a wide area network (WAN), etc. depending on the area or size, and depending on the openness of the network, an intranet, It may be an extranet or the Internet.

Wireless communication includes long-term evolution (LTE), LTE Advance (LTE-A), 5th generation (5G) mobile communication, code division multiple access (CDMA), wideband CDMA (WCDMA), universal mobile telecommunications system (UMTS), WiBro (Wireless Broadband), GSM (Global System for Mobile Communications), DMA (Time Division Multiple Access), WiFi (Wi-Fi), WiFi Direct, Bluetooth, NFC (near field communication), at least one of the communication methods such as Zigbee may include

Wired communication may include at least one of communication methods such as Ethernet, optical network, USB (Universal Serial Bus), and ThunderBolt. Here, the communication unit 150 may include a network interface or a network chip according to the above-described wired/wireless communication method. Meanwhile, the communication method is not limited to the above-described example, and may include a communication method newly appearing according to the development of technology.

As an example, the processor 130 may be connected to at least one external server through the communication unit 160 by executing an application for performing the above-described auditory training method according to various embodiments of the present disclosure.

In this case, the processor 130 may receive application-related information (eg, a web page, sound information, UI information, etc.) from an external server through the communication unit 160 .

And, in the process of performing each of the training steps ( S210 to S250 ), the processor 130 may output various sounds/UIs according to the received information.

Meanwhile, the processor 130 may transmit an audio signal for the sound output in each of the training steps S210 to S250 to at least one speaker device through the communication unit 160 .

As a specific example, the processor 130 may transmit a corresponding audio signal to a speaker device such as a wireless earphone, a wireless headphone, or a wireless speaker through a Bluetooth communication method.

As a result, the above-described queries for inducing user input in each of the steps S210 to S250 may be output through the above-described speaker device.

Meanwhile, although not shown, the electronic device 100 may further include a (wired) audio output terminal in addition to the speaker.

In this case, the electronic device 100 may transmit an audio signal for the sound output in each training step ( S210 to S250 ) to at least one speaker device (eg, earphone/headphone, speaker, etc.) through the audio output terminal. there is. As a result, sound may be output from the corresponding speaker device.

Meanwhile, the various embodiments described above may be implemented in a recording medium readable by a computer or a similar device using software, hardware, or a combination thereof.

According to the hardware implementation, the embodiments described in the present disclosure are ASICs (Application Specific Integrated Circuits), DSPs (digital signal processors), DSPDs (digital signal processing devices), PLDs (programmable logic devices), FPGAs (field programmable gate arrays) ), processors, controllers, micro-controllers, microprocessors, and other electrical units for performing other functions may be implemented using at least one.

In some cases, the embodiments described herein may be implemented by the processor itself. According to the software implementation, embodiments such as the procedures and functions described in this specification may be implemented as separate software modules. Each of the above-described software modules may perform one or more functions and operations described herein.

On the other hand, the computer instructions or computer program for performing the processing operation in the electronic device 100 according to various embodiments of the present disclosure described above is a non-transitory computer-readable medium. can be stored in When the computer instructions or computer program stored in the non-transitory computer-readable medium are executed by the processor of the specific device, the specific device performs the processing operation in the electronic device 100 according to various embodiments described above.

The non-transitory computer-readable medium refers to a medium that stores data semi-permanently, not a medium that stores data for a short moment, such as a register, cache, memory, etc., and can be read by a device. Specific examples of the non-transitory computer-readable medium may include a CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM, and the like.

In the above, preferred embodiments of the present disclosure have been illustrated and described, but the present disclosure is not limited to the specific embodiments described above, and is generally used in the technical field belonging to the present disclosure without departing from the gist of the present disclosure as claimed in the claims. Various modifications may be made by those having the knowledge of

100: electronic device 110: speaker
120: display 130: processor

Claims (9)

In the hearing training method of an electronic device,
a detection training step of outputting a first sound composed of a performance sound of a musical instrument, displaying a first query to inquire whether a performance sound is heard, and receiving a first user input in response to the first query;
outputting a second sound and a third sound composed of a performance sound of a musical instrument, respectively, and displaying a second query to inquire whether at least one of a rhythm and a melody of the second sound and the third sound is different from each other; , a discrimination training step of receiving a second user input in response to the second query;
outputting a fourth sound composed of a performance sound of one instrument, displaying a third query requesting to select a musical instrument corresponding to the fourth sound from among a plurality of instruments, and responding to the third query from among the plurality of instruments a confirmation training step of receiving a third user input for selecting at least one;
outputting a fifth sound including performance sounds of a plurality of musical instruments, displaying a fourth query related to at least one of the plurality of musical instruments corresponding to the fifth sound, and a fourth user input responding to the fourth query To receive, understanding training stage; and
At least one of a detection training step, a discrimination training step, a confirmation training step, and an understanding training step is added based on at least one of the first user input, the second user input, the third user input, and the fourth user input Performing with, retraining step; including, auditory training method of an electronic device. According to claim 1,
The retraining step is
If the first user input is an incorrect answer, additionally performing a detection training step, a discrimination training step, a confirmation training step, and an understanding training step,
When the first user input is a correct answer and the second user input is an incorrect answer, a discrimination training step, a confirmation training step, and an understanding training step are additionally performed;
When the first user input and the second user input are each correct answer and the third user input is an incorrect answer, a confirmation training step and an understanding training step are additionally performed,
When the first user input, the second user input, and the third user input are each a correct answer and the fourth user input is an incorrect answer, a comprehension training step is additionally performed. According to claim 1,
The hearing ability training method of the electronic device,
A detection training step, discrimination training step, confirmation training step, and understanding performed in the retraining step according to the difficulty of at least one of the first to fourth queries and whether at least one of the first to fourth user inputs is correct. Adjusting the difficulty of at least one of the training steps; further comprising, the hearing ability training method of the electronic device. 4. The method of claim 3,
The discrimination training step is
Display the second query corresponding to the first difficulty level,
The retraining step is
If the second user input received according to the second query in the discrimination training step is an incorrect answer, additionally performing a discrimination training step corresponding to a second difficulty level that is lower than or equal to the first difficulty level;
In the discrimination training step, when the second user input received according to the second query is a correct answer, a discrimination training step corresponding to a third difficulty level higher than or equal to the first difficulty level is additionally performed. training method. 5. The method of claim 4,
The hearing ability training method of the electronic device,
When the second user input received according to the second query in the discrimination training step is the correct answer, the time taken to receive the second user input is the received user input based on the training history of the discrimination training step Comparing with the average time spent on; further comprising,
The retraining step is
If the time taken to receive the second user input is longer than or equal to the average time plus the threshold time, a discrimination training step corresponding to the first difficulty level is additionally performed;
When the time taken to receive the second user input is shorter than the average time plus the threshold time, a discrimination training step corresponding to a difficulty higher than the first difficulty level is additionally performed. Way. According to claim 1,
The verification training step is
Outputs a fourth sound composed of a performance sound of one of a plurality of musical instruments corresponding to a plurality of groups including at least one of a keyboard instrument, a wind instrument, a percussion instrument, and a string instrument, and corresponds to the fourth sound among the plurality of musical instruments Displaying the third query requesting to select a musical instrument to be used, receiving a third user input for selecting at least one of the plurality of musical instruments in response to the third query,
The hearing ability training method of the electronic device,
In the confirmation training step, when the third user input received according to the third query is an incorrect answer, identifying whether the instrument selected according to the third user input is included in the same group as the instrument corresponding to the fourth sound further including;
The retraining step is
When the instrument selected according to the third user input is included in the same group as the instrument corresponding to the fourth sound, the confirmation training step is additionally performed a first number of times,
When the musical instrument selected according to the third user input is not included in the same group as the musical instrument corresponding to the fourth sound, the confirmation training step is additionally performed a second number of times greater than the first number. training method. According to claim 1,
The method of training the performance of the electronic device,
Measuring the level of ambient noise; further comprising,
The retraining step is
The performance training method of an electronic device, wherein at least one of the detection training step, the discrimination training step, the confirmation training step, and the understanding training step is additionally performed based on the degree of difficulty corresponding to the measured ambient noise level. In an electronic device,
speaker;
display; and
Including; a processor connected to the speaker and the display;
The processor is
A first sound composed of a performance sound of a musical instrument is output through the speaker, a first query for inquiring whether a performance sound is heard is displayed through the display, and a first user input in response to the first query is received. receiving, performing a detection training step;
A second inquiring whether at least one of a rhythm and a melody of the second sound and the third sound is different from each other by outputting a second sound and a third sound each composed of a performance sound of a musical instrument through the speaker performing a discrimination training step of displaying a query through the display and receiving a second user input in response to the second query;
outputting a fourth sound composed of a performance sound of one instrument through the speaker, displaying a third query requesting to select an instrument corresponding to the fourth sound from among a plurality of instruments, through the display, and the third query performing a confirmation training step of receiving a third user input for selecting at least one of the plurality of instruments in response to;
outputting a fifth sound including performance sounds of a plurality of instruments through the speaker, displaying a fourth query related to at least one of the plurality of instruments corresponding to the fifth sound through the display, and the fourth performing a comprehension training step of receiving a fourth user input in response to a query;
At least one of a detection training step, a discrimination training step, a confirmation training step, and an understanding training step is added based on at least one of the first user input, the second user input, the third user input, and the fourth user input An electronic device that performs a retraining step. In the computer program stored in a computer-readable medium,
A computer program executed by a processor of an electronic device to cause the electronic device to perform the auditory training method of claim 1 .

Images