KR102429108B1 - Electronic apparatus for performing auditory training based on singing, method, and computer program - Google Patents

Abstract

Disclosed is a method for training an hearing ability of an electronic device. The present auditory training method includes the steps of selecting the difficulty level of training based on user information, outputting a song corresponding to the selected difficulty level, displaying a UI (User Interface) guiding the user to sing along the output song, and receiving the user's voice. and determining whether to additionally perform the training step based on a matching rate between the input received, the training step, and the output audio signal of the song and the audio signal of the user's voice.

Description

ELECTRONIC APPARATUS FOR PERFORMING AUDITORY TRAINING BASED ON SINGING, METHOD, AND COMPUTER PROGRAM }

The present disclosure relates to an electronic device for performing auditory training by singing along to a song, and to an auditory training method.

Various hearing training has been developed and performed as the effectiveness of hearing training for hearing impaired people who have received a cochlear implant or wear a hearing aid has been proven.

In particular, as it is known that the part of the brain, which plays a major role in understanding spoken language, also plays a major role in processing music, a method of performing auditory training by having them sing along to a flowing song was devised.

However, the conventional hearing training was inconvenient in that it was performed under the direct participation of experts for each training.

Registered Patent Publication No. 10-19849910000 (user characteristic-based hearing training device)

The present disclosure provides an electronic device that provides a user interface for singing training and an auditory training method.

The present disclosure provides an electronic device and an auditory training method for setting the difficulty and/or number of hearing training for a user according to user information about singing along to a song.

The objects of the present disclosure are not limited to the above-mentioned objects, and other objects and advantages of the present invention not mentioned may be understood by the following description, and will be more clearly understood by the embodiments of the present disclosure. Moreover, it will be readily apparent that the objects and advantages of the present disclosure may be realized by the means and combinations thereof indicated in the claims.

The auditory perception training method of an electronic device according to an embodiment of the present disclosure includes selecting a difficulty level of training based on user information, outputting a song corresponding to the selected difficulty level, and inducing the user to sing along with the output song A training step of displaying a user interface (UI) and receiving a user voice, the step of determining whether to additionally perform the training step based on a matching rate between the outputted song audio signal and the user voice audio signal includes

In this case, the user information may include information on at least one of the user's age, whether or not the cochlear implant is implanted, the timing of the cochlear implantation, whether the hearing aid is used, and the hearing ability training history.

In addition, the determining whether to additionally perform the training step may include performing the training step additionally when the matching rate is lower than a value obtained by subtracting a threshold value from a preset matching rate. Here, the preset matching rate may be a matching rate between the audio signal of the output song and the audio signal of a voice of a normal person sang along with the output song.

The determining whether to additionally perform the training step may include determining not to additionally perform the training step when the matching rate is greater than or equal to a value obtained by subtracting the threshold from the preset matching rate.

On the other hand, in the hearing ability training method of the electronic device, when the number of consecutively performed training steps is equal to or greater than a predetermined number of times, the concordance rate of the last training step among the continuously performed training steps is calculated as the successively performed training. The method may further include comparing with an average matching rate of the steps, and determining not to perform the training step further when the matching rate of the last training step is lower than the average matching rate.

Also, the method for training the hearing ability of the electronic device may further include measuring the level of ambient noise. In this case, the selecting of the difficulty may include selecting the difficulty of the training based on the user information and the level of the ambient noise.

Also, the method for training the hearing ability of the electronic device may further include identifying a pitch of ambient noise. In this case, the training step may output a song corresponding to a pitch in which the difference between the identified pitch and the frequency is a predetermined number or more, display a UI inducing the user to sing along the output song, and receive a user's voice. .

An electronic device according to an embodiment of the present disclosure includes a speaker, a display, a microphone, and a processor. The processor selects a difficulty level of training based on user information, outputs a song corresponding to the selected difficulty level through the speaker, and induces a user to sing along the output song through the display (User Interface) displays, performs a training step of receiving a user's voice through the microphone, and determines whether to further perform the training step based on a matching rate between the audio signal of the user's voice and the audio signal of the output song do.

The electronic device and hearing ability training method according to the present disclosure are effective in providing singing training with a difficulty level suitable for user information and providing repeated training suitable according to the user's training performance.

The electronic device and the auditory training method according to some embodiments of the present disclosure have an effect of determining the optimal number of repetitions by objectively real-time evaluation of performance according to repetition of training.

1 is a block diagram illustrating a configuration of an electronic device according to an embodiment of the present disclosure;
2 is a flowchart illustrating a method for training an auditory ability of an electronic device according to an embodiment of the present disclosure;
3 is a view for explaining a training process of an electronic device according to an embodiment of the present disclosure;
4 is a diagram for explaining an operation of comparing a degree of matching between a user's voice and a target voice (song) by an electronic device according to an embodiment of the present disclosure;
5 is an algorithm for explaining an operation in which an electronic device repeats training depending on whether additional training is performed according to an embodiment of the present disclosure;
6 is an algorithm for explaining an operation in which an electronic device determines whether training is terminated according to the number of training sessions, according to an embodiment of the present disclosure;
7 is a block diagram illustrating a configuration and operation of an electronic device according to various embodiments of the present disclosure.

Prior to describing the present disclosure in detail, a description will be given of the description of the present specification and drawings.

First, terms used in the present specification and claims have been selected in consideration of functions in various embodiments of the present disclosure. However, these terms may vary depending on the intention or legal or technical interpretation of a person skilled in the art, and the emergence of new technology. Also, some terms are arbitrarily selected by the applicant. These terms may be interpreted in the meaning defined in the present specification, and if there is no specific term definition, it may be interpreted based on the general content of the present specification and common technical knowledge in the art.

Also, the same reference numerals or reference numerals in each drawing attached to this specification indicate parts or components that perform substantially the same functions. For convenience of description and understanding, the same reference numbers or reference numerals are used in different embodiments. That is, even though all components having the same reference number are illustrated in a plurality of drawings, the plurality of drawings do not mean one embodiment.

In addition, in this specification and claims, terms including an ordinal number, such as "first" and "second", may be used to distinguish between elements. This ordinal number is used to distinguish the same or similar elements from each other, and the meaning of the term should not be construed as limited due to the use of the ordinal number. For example, the components combined with such an ordinal number should not be limited in the order of use or arrangement by the number. If necessary, each ordinal number may be used interchangeably.

In this specification, the singular expression includes the plural expression unless the context clearly dictates otherwise. In the present application, terms such as "comprises" or "consisting of" are intended to designate that the features, numbers, steps, operations, components, parts, or combinations thereof described in the specification exist, and are intended to indicate that one or more other It is to be understood that this does not preclude the possibility of addition or presence of features or numbers, steps, operations, components, parts, or combinations thereof.

In an embodiment of the present disclosure, terms such as “module”, “unit”, “part”, etc. are terms for designating a component that performs at least one function or operation, and such component is hardware or software. It may be implemented or implemented as a combination of hardware and software. In addition, a plurality of "modules", "units", "parts", etc. are integrated into at least one module or chip, except when each needs to be implemented as individual specific hardware, and thus at least one processor. can be implemented as

In addition, in an embodiment of the present disclosure, when a part is connected to another part, this includes not only direct connection but also indirect connection through another medium. In addition, the meaning that a certain part includes a certain component means that other components may be further included without excluding other components unless otherwise stated.

1 is a block diagram illustrating a configuration of an electronic device according to an embodiment of the present disclosure.

Referring to FIG. 1 , the electronic device 100 may include a speaker 110 , a display 120 , a microphone 130 , and a processor 140 .

The electronic device 100 may be implemented as various devices such as a smart phone, a tablet PC, a notebook PC, a desktop PC, a kiosk, and a TV. However, unlike FIG. 1 , the electronic device 100 may be implemented as a server or as a control device such as a set-top box. In this case, the electronic device 100 may be connected to an external device including at least one of a speaker, a display, and a microphone.

The speaker 110 is a configuration for providing a variety of information audibly, and may include various conventional configurations for converting an electrical audio signal to output a sound.

The display 120 is configured to visually provide various information.

The display 120 may be implemented as a liquid crystal display (LCD), a plasma display panel (PDP), an organic light emitting diode (OLED), a transparent OLED (TOLED), a micro LED, etc., but is not limited thereto. It may include various known types of displays.

The display 120 may be implemented in the form of a touch screen capable of sensing a user's touch manipulation, or may be implemented as a flexible display that can be folded or bent.

The microphone 130 may include at least one circuit for converting an input sound into an electrical audio signal.

The processor 140 is a configuration for controlling the electronic device 100 as a whole by being connected to components included in the electronic device 100 .

The processor 140 may perform operations according to various embodiments to be described later by executing at least one instruction stored in the memory of the electronic device 100 .

The processor 140 is a general-purpose processor such as a central processing unit (CPU), an application processor (AP), etc., a graphics-only processor such as a graphic processing unit (GPU), a vision processing unit (VPU), or the like, or artificial It may be implemented as an intelligence-only processor or the like. In addition, the processor 140 may include a volatile memory such as SRAM.

Hereinafter, operations of the electronic device 100 that provide auditory training (sing along to a song) to a user will be described with reference to the drawings.

2 is a flowchart illustrating a method for training an auditory ability of an electronic device according to an embodiment of the present disclosure.

Referring to FIG. 2 , the processor 140 may select a difficulty level of hearing training based on user information ( S210 ).

The user information may include various information about the user, such as the age of the user, whether or not the cochlear implant has been implanted, the timing of the cochlear implantation, whether the hearing aid is used, and the hearing ability training history. In addition, the user information may further include information on an optimal loudness for the user (Most Comfortable Loudness) obtained according to a result of a hearing test for the user.

Here, the user is a subject of hearing training, and may correspond to a hearing impaired person who has undergone a cochlear implant surgery or wears a hearing aid, but is not limited thereto.

The difficulty level may be preset to a plurality of stages (eg, stages 1 to 5), and songs (melody and rhythm) that are the target of auditory training may be preset for each difficulty stage.

In this case, the melody or rhythm of the song may be changed according to the difficulty. For example, the higher the difficulty, the faster the rhythm of the song or the more complex the melody.

In an embodiment, the processor 140 may select a lower difficulty level as the age of the user is younger.

Also, according to an embodiment, the processor 140 may select a lower difficulty level as the difference between the time when the user received a cochlear implant (or when the user started using the hearing aid) and the current time is shorter.

Also, according to an embodiment, the processor 140 may select a difficulty level of a song to be used for auditory training (singing a song) based on a (past) auditory training history.

The auditory training history may include information on the number, period, frequency, etc. of the user's auditory training (singing). In addition, the auditory training history may include information on the difficulty of the auditory training performed by the user in the past (the difficulty of the song).

Auditory training history, in addition to the history of auditory training performed through the electronic device 100, may further include a history of auditory training performed through the at least one external device 100 or with the help of an expert. have.

When the auditory performance histories of a plurality of users are stored in the electronic device 100 , the processor 140 may select the difficulty of a song through the auditory training histories of users who have currently logged in, etc. among the plurality of users.

As a specific example, the processor 140 may select the difficulty level of auditory training according to the number of times the auditory training is performed.

Here, the processor 140 may set the difficulty of the auditory training to be lower as the number or frequency (eg, the number of times of hearing training during a recent certain period) is less.

In addition, the processor 140 may select the difficulty level of the auditory training to be performed currently by using the difficulty level of the auditory training performed by the user in the past.

For example, the processor 140 may identify auditory training of the highest difficulty (ie, difficulty of a song) among auditory training performed by the user in the past, and select a difficulty level higher than the corresponding difficulty level.

As another example, the processor 140 may identify a difficulty level of a successful auditory training exercise with the highest difficulty among auditory training exercises performed by a user in the past, and select a difficulty level higher than the corresponding difficulty level.

Here, the successful auditory training may mean a case in which the user successfully sang along to a song provided in the auditory training.

For example, when the agreement rate between the audio signal corresponding to the song and the user's voice (audio signal) sung along with the song is equal to or greater than a certain ratio, it may be determined that the auditory training has been successful. However, the present invention is not limited thereto.

Whether or not the auditory training is successful may be determined through the electronic device 100 whenever each auditory training is performed, and a specific embodiment of determining whether the auditory training is successful will be described later in step S230.

Meanwhile, the auditory training history may further include, in addition to auditory training corresponding to singing, a history of auditory training in other methods such as listening to a performance sound of a musical instrument.

The auditory training consisting of listening to the playing sound of the instrument may correspond to, for example, auditory training consisting of the steps of detecting, discriminating, confirming, and understanding the playing sound of the musical instrument. Here, detection is training to check whether or not the playing sound of the instrument is heard, discrimination is training to distinguish the sameness/difference in melody and/or rhythm of the played sounds, and checking is the specific characteristic of the played sound (instrument, rhythm of This is a training to check the level of understanding of speed, melody, etc.), and understanding may be training to check whether more in-depth information (ex. harmony) is recognized about the played sound.

As a specific example, the detection step may include outputting a sound composed of a performance sound of a musical instrument, displaying a query asking whether a performance sound is heard, and receiving a user input in response to the query. In the discriminating step, sequentially outputting sounds composed of a performance sound of each instrument, displaying a query asking whether at least one of a rhythm and a melody of the sounds sounds different from each other, and receiving a user input in response to the query can In the confirmation step, the user outputs a sound composed of a performance sound of one instrument, displays a query requesting to select an instrument corresponding to the corresponding sound from among a plurality of instruments, and selects at least one of the plurality of instruments in response to the query input can be received. In the understanding step, outputting a sound corresponding to one performance sound including performance sounds of a plurality of musical instruments, displaying a query asking for the number of (played) instruments constituting the sound, and a user input responding to the query can receive

At this time, the hearing ability training history stored in the electronic device 100 includes information on whether the user input in response to the query of each step in the detection/discrimination/check/understanding step described above is correct, and the user's hearing determined according to the correct answer. Information on cognitive abilities may be included.

In this case, the processor 140 may select the difficulty level of the auditory training (singing a song) according to the user's auditory cognitive ability acquired through the above-described other auditory training (listening to the sound of a musical instrument).

That is, the processor 140 may perform auditory training (singing) of difficulty matching the auditory cognitive ability of the user, and may increase the difficulty of the auditory training (singing) as the user's auditory cognitive ability increases.

On the other hand, the processor 140 may select the difficulty of hearing ability training in addition to the above-described user information in consideration of the level of ambient noise.

In this case, the processor 140 may measure the level of ambient noise through the microphone 130 , and may select a lower difficulty level as the measured noise level increases.

As a specific example, if it is identified that the level of ambient noise is greater than or equal to the threshold while the level of difficulty is selected as three levels according to user information, the processor 140 may reduce the level of difficulty by one level to select the level of difficulty of the second level.

When the difficulty level is selected according to the above-described various embodiments, the processor 140 may perform auditory training of the corresponding difficulty level (S220).

Specifically, the processor 140 outputs a song corresponding to the selected difficulty level through the speaker 110, displays a UI (User Interface) guiding the user to sing along the output song on the display 120, and a microphone ( 130) can receive the user's voice input.

In this case, the processor 140 may output the song at the most comfortable loudness determined according to the user's hearing test result.

In addition, the processor 140 may output the selected song according to the pitch of the ambient noise.

Specifically, the processor 140 may identify the pitch (frequency) of the ambient noise received through the microphone 130 . In this case, the processor 140 may identify the pitch of the ambient noise or the pitch range of the ambient noise by using the frequency with the highest weight among the audio signals constituting the ambient noise.

In addition, the processor 140 may output a song corresponding to a pitch in which a difference between a pitch (or a pitch range) and a frequency of the identified ambient noise is a predetermined number or more.

Specifically, the processor 140 may output a song having as the main pitch a pitch in which a difference in frequency is greater than or equal to a predetermined number or more from the main pitch of the ambient noise (eg, the frequency with the highest specific gravity).

In this case, since ambient noise and singing are more likely to be clearly distinguished, the effect of ambient noise on training can be relatively reduced.

3 is a diagram for explaining a training process of an electronic device according to an embodiment of the present disclosure.

Referring to FIG. 3 , the electronic device 100 may output a song 310 and display a UI 320 induced to sing along to the song 310 .

Here, the UI 320 may include guide information such as the sheet music 325 of the song 310 to be output.

As a specific example, referring to FIG. 3 , the electronic device 100 highlights the lyrics in the sheet music 325 of the UI 320 in order to inform in real time the progress of a song that is currently being sung along (output). Or, an arrow for intuitively indicating a change in pitch (pitch) in height may be displayed in the sheet music 325 .

Referring to FIG. 3 , the UI 320 restarts the 'pause' item for pausing the playback of the currently output song 310 and the playback (sing along) of the currently output song 310 from the beginning. It can include a 'from scratch' item to start with. When the 'pause' item is selected through a user's touch or the like, the 'pause' item may be changed to a 'play' item.

On the other hand, although FIG. 3 shows only a situation in which the UI 320 for inducing singing along to the song 310 is displayed at the same time as the song 310 is output, the song is to be sung after the output of the song is completed (the song ends). Of course, it is also possible that the inducing UI is provided separately.

Referring to FIG. 3 , a user voice 330 singing along to a song according to the UI 320 may be input through the microphone 130 .

Meanwhile, as shown in FIG. 3 , when the electronic device 100 outputs a song and a user voice is input at the same time, the processor 140 may extract only an audio signal corresponding to the user voice from among the audio signals received through the microphone. .

Specifically, the processor 140 may extract only the audio signal corresponding to the user's voice by removing a portion matching the audio signal corresponding to the output song among the audio signals received through the microphone 130 .

Alternatively, the processor 140 may use an artificial intelligence model trained to recognize only the user's voice.

Here, when an audio signal is input, the artificial intelligence model may be trained to extract only an audio signal corresponding to a specific user's voice. Specifically, the artificial intelligence model may be trained using audio signals corresponding to various utterances/songs of the corresponding user as training data.

In this case, the processor 140 may input the audio signal received through the microphone 130 into the artificial intelligence model to obtain only the audio signal corresponding to the user's voice.

Meanwhile, in order to extract only the audio signal corresponding to the user's voice, the processor 140 may remove noise from the audio signal received through the microphone 130 using a conventional module or artificial intelligence model.

When the audio signal corresponding to the user's voice is extracted, the processor 140 may determine whether to additionally perform training (S220) based on the user's voice (S230).

In relation to this, FIG. 4 is a diagram for describing an operation in which an electronic device compares a degree of matching between a user's voice and a target voice (song) according to an embodiment of the present disclosure.

Referring to FIG. 4 , based on the matching rate between the audio signal 420 of the song output through the speaker 110 and the audio signal 410 of the user's voice, the processor 140 additionally performs a training step S220 . You can decide whether to

Specifically, the processor 140 may determine whether singing in the training step S220 is successful according to the matching rate. Here, the processor 140 determines not to perform additional training additionally when it is determined that singing in the training step S220 is successful, and when it is determined that singing in the training step S220 is not successful, add It may be decided to conduct additional training.

The coincidence rate may correspond to the coincidence rate of at least one of factors such as amplitude, frequency, and fashion for each time of the audio signal.

In this case, the processor 140 may use at least one artificial intelligence model trained to output a matching rate by comparing amplitudes, frequencies, etc. between audio signals. Specifically, the processor 140 may input the audio signal of the user's voice and the audio signal of the output song to the AI model to identify the matching rate.

Alternatively, the coincidence rate may correspond to the coincidence rate of elements such as melody and time signature of a sound represented by the audio signal. Here, since the melody corresponds to the pitch, it can be identified through the frequency, and the beat can be identified through the points in time when a sudden change in amplitude, etc. is sensed.

Meanwhile, as an example, the coincidence rate may mean whether the pitch coincides with the change direction (rising or falling) of the pitch.

Specifically, when the pitch of the output song is “sol-mi-mi-pa-re-re” within the same octave, the processor 140 sets the direction of change in the pitch of the output song as “fall-maintain- It can be identified as “rising-falling-holding”.

Here, if the pitch of the user's voice singing along with the song is "sol-mi (sharp)-mi (sharp)-pa-do-do" within the same octave, the processor 140 determines the direction of change in the pitch of the user's voice. It can be identified as “fall-hold-rise-fall-hold”.

In this case, although the absolute pitch of the pitch is different, the processor 140 may identify that the pitch change directions coincide with each other.

In general, since there are many cases where even people who are not deaf cannot clearly match the absolute tone of a song, it is also possible to determine whether or not the song has been sung successfully by identifying the coincidence rate through the direction of pitch change as in the above-described embodiment. do.

Meanwhile, according to an embodiment, when the matching rate is lower than a value obtained by subtracting the threshold from the preset matching rate, the processor 140 may identify that the training S220 has not been successful and may determine to perform additional training. .

On the other hand, when the matching rate is higher than or equal to the value obtained by subtracting the threshold from the preset matching rate, the processor 140 may identify that the training S220 has been successful and determine that training is not additionally performed.

Here, the preset matching rate may correspond to a matching rate between the output song and the voice of a normal person singing along with the output song.

A normal person may be a person who has not suffered a hearing impairment, has not received a cochlear implant, and does not wear a hearing aid. Also, the normal person may be a person whose age difference with the user is less than a certain age.

Specifically, the preset matching rate may correspond to a matching rate between the audio signal of the voice of at least one normal person who sang along (listening to) the same song as the target song in the training (S220) step and the audio signal of the corresponding song.

In this case, the preset coincidence rate may correspond to an average value of the coincidence rates obtained from each normal voice, but is not limited thereto.

Meanwhile, FIG. 5 is an algorithm for explaining an operation in which the electronic device repeats training according to whether or not additional training is performed according to an embodiment of the present disclosure.

Referring to FIG. 5 , when it is determined that training is additionally performed according to the coincidence rate in step S230 described above, the processor 140 may additionally perform the training in step S220 described above.

In this case, the same song as the song previously used in step S220 may be used, or a song having the same difficulty as the song used in step S220 may be used, but is not limited thereto.

In addition, the processor 140 may perform step S230 to identify whether additional training is required again after step S220 performed again.

In this case, the training may be repeated one or more times until the user successfully sings along to the song.

However, in general, since it has been revealed that auditory training can only be effective if auditory training is performed over a mid- to long-term (ex. several months), it may not be effective to focus only on short-term repetitive training.

In relation to this, the processor 140 according to an embodiment of the present disclosure may measure a training performance for each continuously performed training process, and limit the number of training sessions according to the measured performance.

6 is an algorithm for explaining an operation in which the electronic device determines whether training is terminated according to the number of training sessions according to an embodiment of the present disclosure. The algorithm of FIG. 6 may be performed while the algorithm of FIG. 5 is in progress.

Referring to FIG. 6 , the processor 140 may identify whether the number of consecutive training steps ( S220 ) is greater than or equal to a preset number (eg, 5 times) ( S610 ).

If the number of consecutively performed training steps S220 is greater than or equal to the preset number (S610 - Y), the processor 140 continuously performs the matching rate of the last training step among the continuously performed training steps. It can be compared with the average coincidence rate of the trained training steps (S620).

Here, the coincidence rate corresponds to the coincidence rate between the song sung by the user and the output song as described above.

Referring to FIG. 6 , if the matching rate of the last training step is lower than the average matching ratio (S630 - Y), the processor 140 may determine not to additionally perform the training step (S640). In this case, the processor 140 may stop the process of FIG. 5 irrespective of the determination result of S230 .

As such, the electronic device 100 according to the above-described embodiment of FIG. 6 can objectively identify a point in time when the efficiency of repetitive training is low and stop training, and as a result, it is possible to perform repetitive training only within the range where the efficiency is guaranteed. It has the effect that it can.

Meanwhile, FIG. 7 is a block diagram for explaining the configuration and operation of an electronic device according to various embodiments of the present disclosure.

Referring to FIG. 7 , the electronic device 100 further includes a memory 150 , a user input unit 160 , a communication unit 170 , etc. in addition to the speaker 110 , the display 120 , the microphone 130 , and the processor 140 . may include

The memory 150 is a configuration for storing an operating system (OS) for controlling overall operations of the components of the electronic device 100 and at least one instruction or data related to the components of the electronic device 100 . .

The processor 140 may perform operations according to various embodiments of the present disclosure by executing at least one instruction stored in the memory 150 .

For example, the memory 150 may include information on at least one application for performing the above-described auditory training method according to the present disclosure. In this case, the processor 140 may perform the hearing ability training method according to various embodiments of the present disclosure by executing the corresponding application.

The user input unit 160 is configured to receive a user's command for the electronic device 100 and/or user information required for the operation of the electronic device 100 .

The user input unit 160 may include various components such as one or more buttons, a touch screen, a microphone, and a camera.

For example, when a user command for executing the above-described application is received through the user input unit 160 , the processor 140 may perform the hearing training method according to the present disclosure.

The communication unit 170 may include a circuit as a configuration for the electronic device 100 to communicate with at least one external device.

The communication unit 170 may be connected to an external device based on a network implemented through wired communication and/or wireless communication. In this case, the communication unit 170 may be directly connected to an external device, or may be connected to an external electronic device through one or more external servers (eg, Internet Service Providers (ISPs)) that provide a network.

The network may be a personal area network (PAN), a local area network (LAN), a wide area network (WAN), etc. depending on the area or size, and depending on the openness of the network, an intranet, It may be an extranet or the Internet.

Wireless communication includes long-term evolution (LTE), LTE Advance (LTE-A), 5th generation (5G) mobile communication, code division multiple access (CDMA), wideband CDMA (WCDMA), universal mobile telecommunications system (UMTS), WiBro (Wireless Broadband), GSM (Global System for Mobile Communications), DMA (Time Division Multiple Access), WiFi (Wi-Fi), WiFi Direct, Bluetooth, NFC (near field communication), at least one of the communication methods such as Zigbee may include

Wired communication may include at least one of communication methods such as Ethernet, optical network, USB (Universal Serial Bus), and ThunderBolt. Here, the communication unit 150 may include a network interface or a network chip according to the above-described wired/wireless communication method. Meanwhile, the communication method is not limited to the above-described example, and may include a communication method newly appearing according to the development of technology.

For example, the processor 140 may be connected to at least one external server through the communication unit 170 by executing an application for performing the hearing ability training method according to the present disclosure.

In this case, the processor 140 may receive application-related information (eg, a web page, sound information (song), UI information, etc.) from an external server through the communication unit 170 .

And, in the process of performing the training step (S220), the processor 140 may output various songs/UIs according to the received information.

Meanwhile, the processor 140 may transmit information on the UI provided in the training step S220 to at least one display device through the communication unit 170 .

In this case, the above-described UI 320 and the like may be provided through a display device.

Meanwhile, the processor 140 may transmit an audio signal for a song (sound) output in the training step S220 to at least one speaker device through the communication unit 170 .

As a specific example, the processor 140 may transmit a corresponding audio signal to a speaker device such as a wireless earphone, a wireless headphone, or a wireless speaker through a Bluetooth communication method.

As a result, the song to be sung by the user can be output through the speaker device described above.

Meanwhile, although not shown, the electronic device 100 may further include a (wired) audio output terminal in addition to the speaker.

In this case, the electronic device 100 may transmit the audio signal for the song output in the training step S220 to at least one speaker device (eg, earphone/headphone, speaker, etc.) through the audio output terminal. As a result, a song may be output from the corresponding speaker device.

Meanwhile, in the auditory training method of the electronic device 100 according to the present disclosure, before the process of FIG. 2 , another method of auditory training based on the playing sound of a musical instrument may be performed. In addition, the process of FIG. 2 may be performed only when it is determined that the auditory cognitive ability of the user determined in the auditory training method based on the playing sound of the instrument is equal to or greater than a predetermined value or score. That is, after auditory training based on the playing sound of the instrument is successfully performed, auditory training based on singing may be performed.

Another auditory training method based on the playing sound of a musical instrument may include a retraining step of additionally performing at least one of the above-described training in addition to detection, discrimination, confirmation, and understanding training.

Specifically, the processor 140 may additionally perform at least one of detection, discrimination, confirmation, and understanding training in the retraining step according to a user input received in the detection, discrimination, confirmation, and understanding training process.

In this case, the difficulty of the detection, discrimination, confirmation, and understanding training performed initially may be different from the difficulty of the training additionally performed in the retraining stage.

Specifically, the processor 140 is The difficulty of training included in the retraining step may be adjusted according to the time taken for at least one of the user inputs received in the initially detecting, discriminating, confirming, and understanding steps to be received.

As an example, if the user input received according to the query provided in the first discrimination training stage is the correct answer, the processor 140 receives the user input from the time taken to receive the corresponding user input (eg, the time the query is displayed). time) can be compared with the average time required to receive user input.

The average time may be obtained based on a training history of one or more discrimination training steps performed in the past. As an example, the processor 140 determines that the user input in the past discrimination training stage through the (same) user's past discrimination training history (eg, time taken to receive user input, correct answer rate, number of training times/frequency, etc.) You can calculate the average of the time it took to be received.

Here, if the time taken to receive the user input in the first discrimination training step is longer than or equal to the average time plus the threshold time, in the retraining step, the processor 140 determines the difficulty of the initially performed discrimination training step and The discrimination training step may be additionally performed according to the same difficulty level.

On the other hand, if the time taken to receive the user input in the first discrimination training step is shorter than the average time plus the threshold time, in the retraining step, the processor 140 sets a level of difficulty higher than that of the initially performed discrimination step. According to this, a discrimination training step may be additionally performed.

As such, by performing customized retraining in consideration of not only the correct answer but also the time taken to select the correct answer, there is an effect that optimal hearing training for an individual user can be performed.

Meanwhile, the processor 130 may adjust the number of times of training performed in the retraining step according to at least one of user inputs received in the detection, discrimination, confirmation, and understanding steps.

According to an embodiment, it is assumed that the user inputs received in the detection, discrimination, and understanding steps are correct answers, but the user input received in the confirmation step is an incorrect answer.

In this case, in the retraining step, the processor 140 may additionally perform the detection training, the discrimination training, and the understanding training the same number of times, but may additionally perform the confirmation training more times than the corresponding number.

Meanwhile, according to an embodiment, in the checking step, it is assumed that the electronic device 100 outputs a sound composed of a performance sound of one instrument among a plurality of instruments corresponding to a plurality of groups.

The plurality of groups may include, for example, at least one of a keyboard instrument, a wind instrument, a percussion instrument, and a string instrument by being classified according to a sound generation principle, but in addition to various criteria (ex. traditional instrument/modern instrument, oriental instrument/western instrument) may include groups separated by .

In this case, in the confirmation step, the electronic device 100 displays a query requesting to select an instrument that played the output sound (played sound) from among a plurality of musical instruments corresponding to a plurality of groups, and responds to the query (user input) ) can be received.

Here, if the received user input is an incorrect answer (: select another instrument), the processor 140 identifies whether the instrument selected according to the user input is included in the same group as the instrument (correct answer) that played the output sound. can

If the selected instrument is included in the same group as the instrument that played the corresponding sound, in the retraining step, the processor 140 may additionally perform the confirmation training step a first number of times.

For example, it is assumed that the performance sound output in the confirmation step is the performance sound of the clarinet, and the instrument selected by the user is the saxophone included in the same group (string instrument) as the clarinet. In this case, the processor 140 may perform confirmation training three times in the retraining step.

On the other hand, if the selected instrument is not included in the same group as the instrument that played the corresponding sound, in the retraining step, the processor 140 may additionally perform the confirmation training step a second number of times greater than the first number.

For example, it is assumed that the performance sound output in the confirmation step is the performance sound of the clarinet, and the instrument selected by the user is a piano belonging to a group different from the clarinet. In this case, the processor 140 may perform the confirmation training 6 times more than 3 times in the retraining step.

As such, when the instrument selected by the user in the confirmation training step is not the correct answer, the electronic device 100 according to the present embodiment recognizes the user's recognition according to whether the selected instrument is similar to the played instrument (eg, included in the same group). You can measure your ability differently and perform your retraining process differently. As a result, the optimal customized retraining process for individual users (audience training subjects) can be automatically and delicately constructed.

Meanwhile, according to an embodiment, it is assumed that in the understanding step, the electronic device 100 outputs a sound composed of playing sounds of a plurality of musical instruments corresponding to the same group.

In the understanding stage, it is assumed that a query is provided to inquire about the number of musical instruments that have played the played note.

At this time, if the user input in response to the above-mentioned query is the correct answer, in the understanding training process of the retraining step, the processor 140 outputs a sound composed of a sound composed of a plurality of musical instruments corresponding to a plurality of groups, and related You can provide a query.

On the other hand, if the user input in response to the above-mentioned question is an incorrect answer, in the (additional) understanding training process in the retraining phase, the processor 130 still outputs a sound composed of the playing sounds of a plurality of instruments corresponding to the same group and , and can provide related queries.

Meanwhile, according to an embodiment, it is assumed that in the understanding step, the electronic device 100 outputs sounds composed of playing sounds of a plurality of musical instruments corresponding to a plurality of groups (eg, two groups).

In the understanding stage, it is assumed that a query is provided to inquire about the number of musical instruments that have played the played note.

At this time, if the user input in response to the above-mentioned query is the correct answer, in the (additional) understanding training process of the retraining step, the processor 140 may use a larger number than the above-described plurality of groups (eg, two groups). It is possible to output a sound composed of sound played by a plurality of musical instruments corresponding to a plurality of groups (eg, three groups) and provide a related query.

On the other hand, if the user input in response to the above-mentioned question is an incorrect answer, in the (additional) understanding training process of the retraining step, the processor 140 outputs a sound composed of the playing sounds of a plurality of instruments corresponding to the same group, You can provide related queries.

Since the difficulty of comprehension training increases as musical instruments corresponding to various groups are played together, retraining with a difficulty level suitable for the user's cognitive ability may be performed according to the above-described embodiment.

Meanwhile, the various embodiments described above may be implemented in a recording medium readable by a computer or a similar device using software, hardware, or a combination thereof.

According to the hardware implementation, the embodiments described in the present disclosure are ASICs (Application Specific Integrated Circuits), DSPs (digital signal processors), DSPDs (digital signal processing devices), PLDs (programmable logic devices), FPGAs (field programmable gate arrays) ), processors, controllers, micro-controllers, microprocessors, and other electrical units for performing other functions may be implemented using at least one.

In some cases, the embodiments described herein may be implemented by the processor itself. According to the software implementation, embodiments such as the procedures and functions described in this specification may be implemented as separate software modules. Each of the above-described software modules may perform one or more functions and operations described herein.

On the other hand, the computer instructions or computer program for performing the processing operation in the electronic device 100 according to various embodiments of the present disclosure described above is a non-transitory computer-readable medium. can be stored in When the computer instructions or computer program stored in such a non-transitory computer-readable medium are executed by the processor of the specific device, the specific device performs the processing operation in the electronic device 100 according to the various embodiments described above.

The non-transitory computer-readable medium refers to a medium that stores data semi-permanently, rather than a medium that stores data for a short moment, such as a register, cache, memory, etc., and can be read by a device. Specific examples of the non-transitory computer-readable medium may include a CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM, and the like.

In the above, preferred embodiments of the present disclosure have been illustrated and described, but the present disclosure is not limited to the specific embodiments described above, and is commonly used in the technical field pertaining to the present disclosure without departing from the gist of the present disclosure as claimed in the claims. Various modifications may be made by those having the knowledge of

100: electronic device 110: speaker
120: display 130: microphone

Claims (9)

In the hearing training method of an electronic device,
selecting a difficulty level of training based on user information;
a training step of outputting a song corresponding to the selected difficulty level, displaying a UI (User Interface) guiding to sing along the output song, and receiving a user voice;
determining whether to additionally perform the training step based on a matching rate between the output audio signal of the song and the audio signal of the user's voice;
if the number of consecutively performed training steps is equal to or greater than a preset number, comparing the matching rate of the last training step among the continuously performed training steps with the average matching rate of the continuously performed training steps; and
When the matching rate of the last training step is lower than the average matching rate, determining that the training step is not additionally performed;
The step of determining whether to additionally perform the training step is,
For each of the output audio signal of the song and the audio signal of the user's voice, determining whether a pitch change direction with respect to at least one of rising and falling coincides with each other. According to claim 1,
The user information is
A hearing ability training method for an electronic device, comprising information on at least one of a user's age, whether or not a cochlear implant has been implanted, when a cochlear implant is performed, whether a hearing aid is used, and an auditory training history. According to claim 1,
The step of determining whether to additionally perform the training step is,
If the matching rate is lower than a value obtained by subtracting a threshold from a preset matching rate, it is determined that the training step is additionally performed;
The preset matching rate is,
and a matching rate between the audio signal of the output song and the audio signal of a voice of a normal person sang along with the output song. According to claim 1,
The step of determining whether to additionally perform the training step is,
and determining not to additionally perform the training step when the matching rate is greater than or equal to a value obtained by subtracting a threshold from the preset matching rate. delete According to claim 1,
The hearing ability training method of the electronic device,
Measuring the level of ambient noise; further comprising,
The step of selecting the difficulty level is
and selecting the difficulty level of the training based on the user information and the level of the ambient noise. According to claim 1,
The hearing ability training method of the electronic device,
further comprising identifying a pitch of ambient noise;
The training phase is
Outputting a song corresponding to a pitch in which a difference between the identified pitch and a frequency is a predetermined number or more, displaying a UI guiding the user to sing along the output song, and receiving a user's voice. delete A computer program stored in a computer readable medium,
A computer program stored in a computer-readable medium, which is executed by a processor of an electronic device to cause the electronic device to perform the auditory training method of claim 1 .

Images