KR102630356B1 - Electronic device, control method, and computer program for auditory language and cognitive rehabilitation of infants and children - Google Patents

Abstract

The method of controlling the present electronic device includes a pre-test step of analyzing a video recorded by a test subject to identify the test subject's ability level for at least one of vocalization, listening, development, and communication, and determining the identified ability level. Based on this, providing rehabilitation content including a plurality of vocalization samples, a vocal rehabilitation stage and after the rehabilitation stage, analyzing a video recorded by the test subject to determine at least one of the test subject's vocalization, listening, development, and communication Includes a post-test step, identifying competency levels.

Description

Electronic device, control method, and computer program for auditory language and cognitive rehabilitation of infants and children {Electronic device, control method, and computer program for auditory language and cognitive rehabilitation of infants and children}

This disclosure relates to a method of controlling an electronic device for auditory speech cognitive rehabilitation of infants and young children.

The American Association on Infant Hearing (Joining Committee on Infant Hearing, JCIH) provides guidelines for the diagnosis and rehabilitation of hearing loss in infants and toddlers: a hearing screening test is performed within 1 month of age, hearing loss is confirmed at 3 months, and early intervention is performed before 6 months of age. It suggests that

The timing of starting rehabilitation for hearing-impaired infants not only has a significant impact on each child's linguistic *?* social *?* emotional development, but can also require large social costs in the long term, so timely implementation of early rehabilitation is necessary.

This auditory rehabilitation method for the hearing impaired is described in Korean Patent No. 10-2019-0098663.

However, the subject of the prior patent application is limited to cochlear implant users, and its scope is limited to use for auditory language and cognitive rehabilitation of general infants and young children, and in practice, it is difficult to use it for the purpose of auditory language and cognitive rehabilitation of hearing impaired infants and toddlers.

Additionally, JCIH's early intervention before 6 months actively recommends wearing hearing aids.

Therefore, before 6 months of age, wear hearing aids suitable for the hearing of infants and toddlers, and after fitting them, apply the following techniques.

Based on the vocalization patterns and communication behavior standards for each stage of development of Korean children from 1 to 24 months, we comprehensively analyze infants and toddlers' vocalizations and behaviors with communicative intent, induce vocalizations in prelingual children, and improve communication. There is a need to develop a structured auditory rehabilitation program that can promote.

In other words, the hearing rehabilitation program described above combines Korea's traditional play method and attachment parenting method in addition to rehabilitation methods widely used abroad, and reflects Korea's unique culture by considering the unique vocal development stage of infants and toddlers that fits the Korean language structure and improves the phonological quality of Korean. It must be possible to specifically present an approach to early auditory language cognitive rehabilitation of hearing-impaired infants and toddlers, including their characteristics.

Public Patent Publication No. 10-2021-0019712 (Auditory training system and auditory training method for hearing loss patients)

This disclosure provides a Korean Aural Rehabilitation for Infants (KARI) program based on an evaluation tool that systematizes the vocal development stages of infants and toddlers.

The objects of the present disclosure are not limited to the objects mentioned above, and other objects and advantages of the present invention that are not mentioned can be understood by the following description and will be more clearly understood by the examples of the present disclosure. Additionally, it will be readily apparent that the objects and advantages of the present disclosure can be realized by the means and combinations thereof indicated in the patent claims.

A method of controlling an electronic device according to an embodiment of the present disclosure includes a pre-test step of analyzing a video captured by a test subject to identify the test subject's ability level for at least one of vocalization, listening, development, and communication; Based on the identified ability level, a vocal rehabilitation step, providing rehabilitation content including a plurality of vocalization samples, and after the rehabilitation step, analyzing the video recorded by the test subject to determine the test subject's vocalization, listening, and development , and a post-test phase that identifies the level of competency in at least one of the following: communication.

The vocal rehabilitation step includes providing a first vocalization sample, identifying whether the test subject's voice imitating the first vocalization sample is input within a certain time from when the first vocalization sample is provided, and the first vocalization sample. 1. If the test subject's voice imitating the first vocalization sample is identified as input, providing a second vocalization sample, and if the test subject's voice imitating the first vocalization sample is not identified as input, providing the first vocalization sample. and providing the vocalization sample again.

The step of providing the first speech sample includes displaying a plurality of image frames representing the mouth shape of a character uttering the first text through a display, and including a voice in which the first text is uttered through a speaker. Target sound can be output.

The step of identifying whether the test subject's voice imitating the first vocalization sample is input includes comparing a plurality of image frames obtained by photographing the test subject through a camera with a plurality of image frames representing the mouth shape of the character. When sound including the voice of the test subject is input through the microphone, audio data constituting the input sound can be converted into at least one text, and the converted text can be compared with the first text.

The step of providing the first speech sample again includes identifying the direction of the examinee based on the image acquired through the camera, and uttering the first text while looking in the identified direction through the display. Displaying a plurality of image frames representing the mouth shape of the character, and outputting a first target sound including a voice in which the first text was uttered through the speaker at a volume greater than the step of providing the first speech sample. can do.

The control method of the electronic device may include a stimulation rehabilitation step of controlling the toy terminal based on communication with the toy terminal and providing sound through the toy terminal.

The stimulation rehabilitation step includes providing a first sound through the toy terminal; And when the voice of the test subject is input after the first sound is provided, it may include providing a second sound that imitates the voice of the test subject.

The control method of the electronic device may include a voice control step of controlling the wearable terminal based on communication with the wearable terminal and providing vibration through the wearable terminal.

The voice adjustment step is to set a reference range of the appropriate voice level in the wearable terminal, and when the level of the test subject's voice is input after being set, the reference range of the appropriate voice level and the level of the test subject's voice are compared. step; and providing vibration through the wearable device when the level of the test subject's voice is not included in the reference range.

The step of providing vibration through the wearable device may provide vibration intensity proportional to the difference between the level of the test subject's voice and the reference range.

The control method of the electronic device may include acquiring at least one external sound related to the surrounding environment through a microphone, and generating a third sound based on the acquired external sound.

In the stimulation rehabilitation step, the third sound may be provided together with the second sound.

The pre-test step includes analyzing one or more image frames included in the video to identify the test subject and the guardian, recognizing the mouth shape of each of the test subject and the guardian over time, each of the test subject and the guardian Based on the shape of the mouth at each time, dividing the audio data included in the video into a time section including the voice of the test subject and a time section including the voice of the guardian, the guardian input into the first time section The first text into which the voice of the person has been converted can be compared with the second text into which the voice of the test subject has been converted, input in a second time period following the first time period.

An electronic device according to an embodiment of the present disclosure includes a microphone, a camera, a display, a speaker, and a processor. The processor generates a video by filming the test subject through the microphone and the camera, and analyzes the video to identify the test subject's ability level for at least one of vocalization, listening, development, and communication. Performing a vocal rehabilitation step, providing rehabilitation content including a plurality of vocalization samples based on the identified ability level, and after the rehabilitation step, analyzing the video captured by the test subject to perform the vocal rehabilitation step. Perform a post-test step, identifying the test subject's level of competency in at least one of speech, listening, development, and communication;

In performing the vocal rehabilitation step, the processor provides a first vocalization sample through the speaker and the display, and the test subject imitates the first vocalization sample within a certain time from when the first vocalization sample is provided. Identifying whether the voice of the subject is input through the microphone, and if the voice of the test subject imitating the first vocalization sample is identified as being input, providing a second vocalization sample, and imitating the first vocalization sample If the test subject's voice is not identified as input, the first speech sample is provided again.

The control method of an electronic device according to the present disclosure provides a Korean Aural Rehabilitation for Infants (KARI) program based on an evaluation tool that systematizes the vocal development stages of infants and toddlers, providing a must-have for hearing-impaired children. Receiving necessary and appropriate auditory rehabilitation early has the effect of maximizing communication skills.

The electronic device and auditory training method according to some embodiments of the present disclosure has the effect of determining the optimal number of repetitions by objectively evaluating performance according to repetition of training in real time.

1 is a block diagram for explaining the configuration of an electronic device according to an embodiment of the present disclosure;
2 is a flowchart illustrating a method for controlling an electronic device according to an embodiment of the present disclosure;
FIG. 3 is a diagram illustrating an operation of outputting the mouth shape of a character uttered through a display and a voice uttered through a speaker as sound in a method for controlling an electronic device according to an embodiment of the present disclosure;
4 is an algorithm for explaining an operation of determining whether to end an electronic device control method according to an embodiment of the present disclosure;
5 is a flowchart illustrating a method for controlling an electronic device according to an embodiment of the present disclosure;
FIG. 6 is a diagram illustrating an operation of providing sound through a toy terminal based on communication with the toy terminal in the electronic device control method according to an embodiment of the present disclosure;
Figure 7 is a block diagram for explaining the configuration and operation of an electronic device according to various embodiments of the present disclosure.

Before explaining the present disclosure in detail, the description method of the present specification and drawings will be explained.

First, the terms used in the specification and claims are general terms selected in consideration of their functions in various embodiments of the present disclosure. However, these terms may vary depending on the intention of technicians working in the relevant technical field, legal or technical interpretation, and the emergence of new technologies. Additionally, some terms are arbitrarily selected by the applicant. These terms may be interpreted as defined in this specification, and if there is no specific term definition, they may be interpreted based on the overall content of this specification and common technical knowledge in the relevant technical field.

In addition, the same reference numbers or symbols in each drawing attached to this specification indicate parts or components that perform substantially the same function. For convenience of explanation and understanding, the same reference numerals or symbols are used in different embodiments. That is, even if all components having the same reference number are shown in multiple drawings, the multiple drawings do not represent one embodiment.

Additionally, in this specification and claims, terms including ordinal numbers such as “first”, “second”, etc. may be used to distinguish between components. These ordinal numbers are used to distinguish identical or similar components from each other, and the meaning of the term should not be interpreted limitedly due to the use of these ordinal numbers. For example, the order of use or arrangement of components combined with these ordinal numbers should not be limited by the number. If necessary, each ordinal number may be used interchangeably.

In this specification, singular expressions include plural expressions, unless the context clearly dictates otherwise. In this application, terms such as “comprise” or “consist of” are intended to designate the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, but are intended to indicate the presence of one or more other It should be understood that this does not exclude in advance the presence or addition of features, numbers, steps, operations, components, parts, or combinations thereof.

In embodiments of the present disclosure, terms such as “module”, “unit”, “part”, etc. are terms to refer to components that perform at least one function or operation, and these components are either hardware or software. It may be implemented or may be implemented through a combination of hardware and software. In addition, a plurality of "modules", "units", "parts", etc. are integrated into at least one module or chip, except in cases where each needs to be implemented with individual specific hardware, and is integrated into at least one processor. It can be implemented as:

Additionally, in an embodiment of the present disclosure, when a part is connected to another part, this includes not only direct connection but also indirect connection through other media. In addition, the meaning that a part includes a certain component does not mean that other components are excluded, but that it may further include other components, unless specifically stated to the contrary.

1 is a block diagram for explaining the configuration of an electronic device according to an embodiment of the present disclosure.

Referring to FIG. 1 , the electronic device 100 may include a speaker 110, a display 120, a microphone 130, a camera 140, and a processor 150.

The electronic device 100 may be implemented as a variety of devices, such as a smartphone, tablet PC, laptop PC, desktop PC, kiosk, or TV. The electronic device 100 may be implemented as a server or a control device such as a set-top box. In this case, the electronic device 100 may be connected to an external device including at least one of the speaker 110, display 120, microphone 130, and camera 140.

The speaker 110 is a component for providing various information audibly, and may include various conventional components for converting electrical audio signals to output sound.

The display 120 is configured to visually provide various information.

The display 120 may be implemented as a Liquid Crystal Display (LCD), Plasma Display Panel (PDP), Organic Light Emitting Diodes (OLED), Transparent OLED (TOLED), Micro LED, etc., but is not limited thereto and may be implemented as a conventional display panel. It may include various known types of displays.

The display 120 may be implemented in the form of a touch screen capable of detecting a user's touch operation, or may be implemented as a flexible display that can be folded or bent.

The microphone 130 may include at least one circuit for converting input sound into an electrical audio signal.

The camera 140 is configured to photograph at least one subject, and the processor 150 can recognize at least one object or identify the distance to the object through the camera 140.

The processor 150 is connected to components included in the electronic device 100 to generally control the electronic device 100.

The processor 150 may perform operations according to various embodiments, which will be described later, by executing at least one instruction stored in the memory of the electronic device 100.

The processor 150 may be a general-purpose processor such as a Central Processing Unit (CPU), an Application Processor (AP), a graphics-specific processor such as a Graphic Processing Unit (GPU), a Vision Processing Unit (VPU), or an artificial processor such as a Neural Processing Unit (NPU). It can be implemented with an intelligent processor, etc. Additionally, the processor 150 may include volatile memory such as SRAM.

Through the following drawings, a method of controlling an electronic device provided to a test subject will be described.

FIG. 2 is a flowchart illustrating a method for controlling an electronic device according to an embodiment of the present disclosure.

Referring to FIG. 2, the processor 150 may perform steps S210 to S230 using the speaker 110, display 120, microphone 130, and camera 140.

Specifically, the processor 150 may sequentially perform pre-test (S210), vocal rehabilitation (S220), and post-test (S230).

Performing the pre-test (S210) is a step to identify the test subject's ability level for at least one of vocalization, listening, development, and communication by analyzing the video recorded by the test subject.

Additionally, while performing a preliminary inspection (S210), the electronic device 100 may provide a questionnaire to the guardian of the inspected person through the display 120. For example, a separate checklist may be provided to check the test subject's vocalization, listening, development, and behavioral development for communication.

At this time, the ability level of the test subject can be identified according to the guardian's user input for each item.

Additionally, in the pre-test performance step (S210), the electronic device 100 may analyze a video captured by the test subject.

The video may be obtained as a result of the subject being photographed through the camera 140 of the electronic device 100 or an external imaging device, and may include a plurality of image frames and audio data that follow sequentially.

Here, the video may be filmed while the guardian and the test subject are conducting a test or training related to at least one of vocalization, listening, development, and communication.

Specifically, the electronic device 100 may identify the test subject and the guardian by analyzing one or more image frames included in the video.

At this time, at least one artificial intelligence model (e.g. CNN ; Convolution Neural Network) trained to recognize objects (e.g. people, human faces) by analyzing images may be used.

Additionally, the electronic device 100 can recognize the mouth shape of each test subject and guardian over time. For example, through an artificial intelligence model trained to identify mouth shapes in the order in which they were acquired at regular time intervals, audio data that matches the point in time when the mouth shape was acquired can be called.

At this time, based on the mouth shapes of the test subject and the guardian at each time, the electronic device 100 may divide the audio data included in the video into a time section containing the test subject's voice and a time section including the guardian's voice. For example, the electronic device 100 converts the audio data to a frequency for each time section to extract a feature vector (ex. MFCC ; Mel Frequency Cepstrum Coefficients), while the speaker (the test subject or guardian) can be judged.

For example, assume that the first time section includes the guardian's voice, and the second time section following the first time section includes the test subject's voice.

In this case, the electronic device 100 may compare the first text converted from the guardian's voice input in the first time interval with the second text converted from the test subject's voice inputted in the second time interval.

At this time, the test subject's ability level can be calculated according to the degree of agreement between the first text and the second text. The higher the match, the higher the ability level can be identified.

The degree of correspondence between texts may be calculated according to meaning-related vectors identified according to natural language understanding technology, or may be calculated according to the degree of correspondence between characters constituting each text.

The vocal rehabilitation step (S220) is a step for providing rehabilitation content including a plurality of vocalization samples based on the identified ability level.

Speech samples include specific mouth shapes and voices that produce vowels, consonants, phonemes, words, sentences, etc. of various languages. Specifically, a speech sample may include an image of a mouth uttering a specific text and a sound (audio data) including a voice uttering the text.

The post-test step (S230) is a step for identifying the test subject's ability level for at least one of vocalization, listening, development, and communication by analyzing the video recorded by the test subject after the vocal rehabilitation step (S220).

Additionally, the post-test step (S230) may be conducted in a process and format similar to the pre-test step (S210) described above.

FIG. 3 is a diagram illustrating an operation of outputting a mouth shape of a character uttered through a display and a voice uttered through a speaker as sound in a method for controlling an electronic device according to an embodiment of the present disclosure.

FIG. 3 is an example showing an operation in which an electronic device 100 implemented as a smartphone performs a vocal rehabilitation step (S220), in which the electronic device 100 displays the shape of a character's mouth through the display 120 and , This shows the operation of outputting the voice uttered through the speaker 110 as sound.

Figure 3 illustrates a case where a vocalization sample for the word “hello” is provided as an example. The electronic device 100 may display a character having the shape of a mouth uttering the word “hello” through a display, and simultaneously output a voice sound including the word “hello” through a speaker.

Additionally, characters displayed through the display 120 may appear as various characters, mouth shapes, etc. in addition to the characters shown.

FIG. 4 is an algorithm for explaining an operation of determining whether a method for controlling an electronic device has ended according to an embodiment of the present disclosure.

Referring to FIG. 4, the algorithm through the vocal rehabilitation step (S220) will be described by way of example.

In the vocal rehabilitation step (S220), a first vocalization sample may be provided (S221) in order to perform the vocal rehabilitation step (S220) described above.

Specifically, the first speech sample may include an image of a mouth uttering the first text and a sound (first target sound) including a voice uttering the first text.

At this time, the electronic device 100 may display/output various emoticons, letters, phrases, etc. through the display 120 to encourage and encourage the test subject to imitate the first vocalization sample.

Additionally, the electronic device 100 may identify (S222) whether the test subject's voice imitating the first vocalization sample is input within a certain period of time from the time the first vocalization sample is provided (S221).

Specifically, the electronic device 100 may compare a plurality of image frames obtained by photographing the examinee through the camera 140 with a plurality of image frames representing the mouth shape of the character uttering the first text. At this time, if the mouth shape of the test subject and the mouth shape of the character do not match for a certain number of image frames or more, it may be identified that the test subject's voice imitating the first speech sample was not input.

In addition, when a sound including the voice of the test subject is input through the microphone 130, the electronic device 100 converts the audio data constituting the input sound into at least one text (: voice recognition) and converts the converted text into at least one text. can be compared with the first text described above.

At this time, if the converted text does not match the above-described first text, it may be identified that the test subject's voice imitating the first speech sample was not input. The degree of correspondence between texts may be calculated according to meaning-related vectors identified according to natural language understanding technology, or may be calculated according to the degree of correspondence between characters constituting each text.

In this way, the electronic device 100 can identify that the test subject's voice imitating the first speech sample has been input, assuming that both the mouth shape and voice of the test subject that match the first speech sample are recognized.

And when the test subject's voice imitating the first vocalization sample is identified as being input, the electronic device may provide a second vocalization sample (S224).

At this time, the second speech sample, unlike the first speech sample, may be a speech sample for a second text, and, like the first speech sample, may include an image of a mouth shape and audio data. Specifically, the second speech sample may include audio data including an image of the mouth of a character speaking the second text and a voice speaking the second text.

Meanwhile, if it is identified that the test subject's voice imitating the first vocalization sample has not been input after the first vocalization sample is provided, the electronic device may provide the first vocalization sample again (S223).

In this case, the electronic device 100 may output the first target sound including the voice in which the first text was spoken at a louder volume than the step of providing the first speech sample.

Additionally, the electronic device 100 can acquire an image of the subject being inspected through a camera or an external imaging device, and can identify the direction in which the subject is located based on the camera 140 based on the acquired image.

At this time, the electronic device 100 may display a plurality of image frames representing the mouth shape of the character uttering the first text while looking in the identified direction through the display 120. For example, when the test subject is located to the left of the camera 140, the electronic device 100 may display a character speaking while looking to the left.

Additionally, the electronic device 100 may output the first target sound including the voice in which the first text was uttered through the speaker 110 at a louder volume than in the previous step of providing the first speech sample.

Additionally, the electronic device 100 may provide a speech sample on the premise that the test subject is looking at the camera 140.

The electronic device 100 can identify whether the examinee is looking toward the camera 140 through the acquired image.

Specifically, when the examinee is facing the camera 140, the examinee's pupils can be identified, and whether the examinee is looking at the camera 140 can be identified based on the shape or angle of the identified pupil.

As an example, when the direction of the examinee is identified, the electronic device 100 may provide a vocalization sample implemented as a character speaking while looking in the identified direction, assuming that the examinee is looking at the camera 140.

FIG. 5 is a flowchart for explaining a method of controlling an electronic device according to an embodiment of the present disclosure, and FIG. 6 is a flowchart of a method of controlling an electronic device according to an embodiment of the present disclosure, based on communication with a toy terminal through a toy terminal. This is a diagram to explain the operation of providing sound.

Referring to FIGS. 5 and 6, the control method of the electronic device is additionally performed in step S220 described above.

This is a stimulation rehabilitation step (S610), and the toy terminal 400 is controlled based on communication with the toy terminal 400 to provide sound through the toy terminal 400.

Here, the toy terminal 400 is a character toy and can communicate with a dedicated application installed on the electronic device 100, such as a smart phone or tab, which is a portable terminal.

At this time, the toy terminal 400 externally includes a camera for taking external images, a speaker for outputting voice, and a microphone for inputting voice, and all of these components are mounted at appropriate locations within the character toy body. It can be.

And the toy terminal 400 can be connected to the electronic device 100 through a wireless communication network.

Here, the wireless communication network is Bluetooth, a short range communication technology, and other standard technologies such as RFID (Radio Frequency Identification), Infrared Data Association (IDA), UWD (Ultra Wideband), and ZigBee. It can be used.

Additionally, a wireless communication network can communicate with a smartphone application located at a distance through a communication network.

For example, a home where a toy terminal is located can connect to the network by connecting to WiFi, and the application user can communicate using a smartphone carried outside the home.

In this case, wireless Internet technologies such as WLAN (Wireless LAN), Wi-Fi, Wibro (Wireless broadband), Wimax (World Interoperability for Microwave Access), and HSDPA (High Speed Downlink Packet Access) can be used as wireless communication modules. .

In addition, the toy terminal 400 is configured to have a power module stored inside the body of the toy and supply power to internal modules of the toy terminal 400.

Additionally, in the stimulation rehabilitation step (S610), the first sound may be provided through the toy terminal 400. Here, the first sound is a general mechanical sound (beep, jing, etc.), speech sound (peekaboo, rumble, etc.), which is input to the audio data of the electronic device 100 and is transmitted as a sound through a speaker provided in the toy terminal 400. can be provided. Therefore, by providing the first sound through the toy terminal 400, the test subject can be induced to respond to the first sound and make the same sound as the first sound.

Additionally, after the first sound is provided from the toy terminal 400, a voice produced by the test subject similar to the first sound is input to the electronic device. Additionally, the voice input to the electronic device 100 may provide a voice imitated by the test subject as a second sound through the toy terminal 400.

Additionally, the electronic device 100 may acquire at least one external sound related to the surrounding environment through the microphone 130.

Here, the external sound may be a voice or environmental sound that is different from the first sound.

Additionally, the electronic device 100 may generate a third sound based on the acquired external sound. Additionally, the third sound generated through the electronic device 100 can be provided through the speaker of the toy terminal 400.

Additionally, the method of controlling an electronic device may further include a voice control step.

In the voice control step, the electronic device 100 may control the wearable terminal based on communication with the wearable terminal and provide vibration through the wearable terminal.

Here, the wearable terminal is, for example, a smartwatch, and when the electronic device 100 is implemented as a portable terminal such as a smart phone or tablet PC, it can communicate with a dedicated application installed on the electronic device 100.

Additionally, the wearable terminal can connect to the electronic device 100 through a wireless communication network.

Additionally, in the voice adjustment step, the electronic device 100 sets a reference range corresponding to the appropriate voice level, and then, when the test subject's voice level is input, the reference range of the appropriate voice level and the test subject's voice level are compared. You can.

In this case, the electronic device 100 can identify whether the level of the test subject's voice is within the reference range.

If the level of the test subject's voice is not within the reference range, the electronic device 100 may provide vibration through the wearable terminal.

As an example, if the wearable terminal is implemented in the form of a bracelet or watch, it is assumed that it is worn on the wrist of the subject. Here, while the reference range is set to 30 to 60 decibels, when the input level of the test subject's voice is 15 decibels, the electronic device 100 sends “Jingjingjing-“, or “Jing-“ through the wearable terminal. The same vibration may be provided.

Additionally, the electronic device 100 may calculate the difference between the reference range and the loudness of the test subject's voice.

In this case, as the difference value between the level of the test subject's voice and the reference range increases, the intensity of the vibration may gradually increase. On the other hand, as the difference between the level of the test subject's voice and the reference range becomes smaller, the intensity of vibration may gradually decrease.

As another example, multiple wearable terminals may be used.

For example, it can be assumed that the wearable terminal is in the form of a bracelet or watch, and is worn on the test subject's left wrist (: first wearable device) and right wrist (: second wearable device), respectively.

Here, when the level of the test subject's voice is outside the reference range, the electronic device 100 may provide vibration through the first wearable terminal or the second wearable terminal. Specifically, if the level of the test subject's voice is greater than the reference range, vibration may be provided through the first wearable terminal, and if the level of the test subject's voice is smaller than the reference range, vibration may be provided through the second wearable terminal. .

Figure 7 is a block diagram for explaining the configuration and operation of an electronic device according to various embodiments of the present disclosure.

Referring to FIG. 7, the electronic device 100 includes a speaker 110, a display 120, a microphone 130, a camera 140, a processor 150, a memory 160, a communication unit 170, and a user input unit. (180) may further be included.

The speaker 110, display 120, microphone 130, and camera 140 may have the same configuration as described above with reference to FIG. 1.

The memory 160 is configured to store an operating system (OS) for controlling the overall operation of the components of the electronic device 100 and at least one instruction or data related to the components of the electronic device 100. .

The processor 150 may perform operations according to various embodiments of the present disclosure by executing at least one instruction stored in the memory 160.

As an example, the memory 160 may include information about at least one application for performing the above-described electronic device control method according to the present disclosure. At this time, the processor 150 may perform the electronic device control method according to various embodiments of the present disclosure by executing the corresponding application.

The user input unit 180 is configured to receive user commands for the electronic device 100 and/or user information required for operation of the electronic device 100.

The user input unit 180 may include various components such as one or more buttons, a touch screen, a microphone, and a camera.

For example, when a user command for executing the above-described application is received through the user input unit 180, the processor 150 may perform the electronic device control method according to the present disclosure.

The communication unit 170 may include a circuit that allows the electronic device 100 to communicate with at least one external device.

The communication unit 170 may be connected to an external device based on a network implemented through wired communication and/or wireless communication. At this time, the communication unit 170 may be connected directly to an external device, but may also be connected to an external electronic device through one or more external servers (ex. Internet Service Provider (ISP)) that provide a network.

Depending on the area or size, the network may be a personal area network (PAN), a local area network (LAN), or a wide area network (WAN). Depending on the openness of the network, it may be an intranet, It may be an extranet, or the Internet.

Wireless communications include LTE (long-term evolution), LTE-A (LTE Advance), 5G (5th Generation) mobile communications, CDMA (code division multiple access), WCDMA (wideband CDMA), UMTS (universal mobile telecommunications system), and WiBro. At least one of the following communication methods: (Wireless Broadband), GSM (Global System for Mobile Communications), DMA (Time Division Multiple Access), WiFi (Wi-Fi), WiFi Direct, Bluetooth, NFC (near field communication), and Zigbee. It can be included.

Wired communication may include at least one of communication methods such as Ethernet, optical network, USB (Universal Serial Bus), and Thunderbolt. Here, the communication unit may include a network interface or network chip according to the wired or wireless communication method described above. Meanwhile, the communication method is not limited to the examples described above and may include newly emerging communication methods as technology develops.

As an example, the processor may be connected to at least one external server through a communication unit by executing an application for performing the electronic device control method according to the present disclosure.

At this time, the processor 150 may receive application-related information (ex. web page, etc.) from an external server through the communication unit 170.

Meanwhile, although not shown, the electronic device 100 may further include a (wired) audio output terminal in addition to the speaker.

In this case, the electronic device 100 may transmit an audio signal including sound such as voice to at least one speaker device (e.g., earphone/headphone, speaker, etc.) through an audio output terminal. As a result, sound can be output from the corresponding speaker device.

In some cases, embodiments described herein may be implemented in the processor itself. According to software implementation, embodiments such as procedures and functions described in this specification may be implemented as separate software modules. Each of the software modules described above may perform one or more functions and operations described herein.

Meanwhile, computer instructions or computer programs for performing processing operations in the electronic device 100 according to various embodiments of the present disclosure described above are non-transitory computer-readable medium. It can be saved in . Computer instructions or computer programs stored in such non-transitory computer-readable media, when executed by a processor of a specific device, cause the specific device to perform processing operations in the electronic device 100 according to the various embodiments described above.

A non-transitory computer-readable medium refers to a medium that stores data semi-permanently and can be read by a device, rather than a medium that stores data for a short period of time, such as registers, caches, and memories. Specific examples of non-transitory computer-readable media may include CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM, etc.

Meanwhile, in the control method according to the various embodiments described above, the following various guidelines related to inspection and rehabilitation may be used. Each text constituting the following guidelines may be provided visually/audibly through the electronic device 100, and may be applied to the operation of the electronic device 100 during the rehabilitation process.

1. Guardian consultation

A. Before starting the rehabilitation program, guardian consultation is conducted with questions in two areas met.

i. Information consultation (Thinking mind)

ii. Emotional directive counseling (Feeling mind)

2. Conduct rehabilitation baseline confirmation test

A. Voice and speech area

i. Use the Korean Infant Vocal and Articulatory Development List (KIVAD) to determine the current vocal development stage of infants and toddlers of all ages.

ii. Inspection and analysis is conducted by analyzing video recordings.

B. Listening and Communication Area

i. The Infant-Toddler Auditory & Communicative Behavioral Checklist (IT-ACBA) is used to check the development of communicative behavior in infants and toddlers aged 0 to 26 months using the communication behavior checklist, a parent questionnaire.

ii. The development of listening skills in infants and toddlers aged 0 to 24 months was checked using the Infant-Toddler Meaningful Auditory Integration Scale (IT-MAIS), a parent questionnaire.

1. This test is conducted after confirming hearing loss in infants and wearing assistive devices (hearing aids, cochlear implants).

C. General areas of development

i. The Parent Questionnaire, Developmental assessment for the Early intervention Program Planning (DEP), is used to check the overall development of infants and toddlers aged 0 to 36 months in terms of gross motor skills, fine motor skills, cognition, and social emotions.

D. Language area

i. The vocabulary development of infants and toddlers aged 8 to 36 months was checked using the Korean-MacArthur Bates Communicative Development Inventories (K-MBCDI), a parent questionnaire.

ii. The overall language development of infants and toddlers aged 4 to 36 months is checked using the Sequenced Language Scale for Infants (SELSI), a parent interview.

iii. The CSBS DP, which is conducted by analyzing video recordings, is used to check the development of communication skills in infants and toddlers aged 6 to 24 months.

3. Specific progress of auditory language cognitive rehabilitation KARI

A. Conceive a rehabilitation plan and prepare a session plan by applying KIVAD and IT-ACBC

B. Implementing KARI’s detailed process

i. For hearing-impaired infants and toddlers, listen to the early vocalizations intensively and have them imitate them based on the list of vocalization development of normal infants and toddlers.

ii. Short fun input for modeling (SFIM)

1. Present speech or vocal modeling for 1 second and encourage the child to imitate by looking into the child's eyes for 4 seconds and waiting for imitation.

2. Present 1 minute of modeling, consisting of 12 stimuli of 5 seconds each and an imitation induction period, repeatedly presented briefly as above for about 1 second, at least 5 times a day when the child is in a good mood and in a modeling position. box

iii. Attachment parenting method implemented

1. Create an environment similar to the womb by allowing the baby to feel the parent’s body temperature and heartbeat using swaddles.

2. Naturally applying skinship, which is the optimal way to convey gestures or language, within the Ondol lifestyle.

3. Create a stable environment when sleeping through the habit of hugging.

iv. Implementing traditional parenting methods

1. Movement play using our people’s traditional parenting method, Dandong Siphun (Bul-a-Bul-a, Sang-sang-sang, Dori-dori, Jiam-jiam, Gonji-gonji, Seomma-seomma, Upbi-eopbi, Aham-aham, Clap Clap Clap Clap Gung, Zillaabi Wheel Wheel) carried out

2. The movements according to the meaning of each hun are applied and performed differently according to the development of the appropriate monthly age.

v. Application of overseas infant rehabilitation techniques such as Auditory Verbal Therapy (AVT)

1. An early auditory language rehabilitation program is implemented in which the child's parents or primary caregivers actively participate in mediation so that the child can naturally hear, speak, and develop.

2. SKI-HI, one of the examples of overseas infant rehabilitation techniques, covers not only pure hearing impairment but also overlapping disabilities, and is referenced for its composition and some contents.

3. It is possible to check the function of a hearing aid or cochlear implant by measuring Ling 6, which consists of the 6 basic vowels (um), (right), (a), (ii), (sh), and (s) by frequency.

4. When measuring Ling 6, a reconstructed measurement method using toys, sounds, games, songs, etc. appropriate for domestic infants is used.

vi. Modeling using various games and toys for each age group

1. 0~2 months

A. Lie down in a quiet place with your baby, make eye contact, and call your baby’s name.

B. Give the child a toy that makes a sound, wait for the child to respond to the sound, and then make the same sound as the baby.

C. Stimulate the baby with sound through physical play

2. 3 months

A. Prepare an unbreakable mirror and look at the mirror together and call the baby’s name.

B. Play bye-bye and peek-a-boo while looking at a mirror and encourage a positive vocalization model by immediately praising the baby's response.

3. For 4 months, imitate the baby's vocalizations and induce vocalizations by playing different environmental sounds.

4. At 5 months, the baby learns the communication custom of taking turns by giving and receiving his or her favorite toy using the ‘please’ motion.

5. 6 months

A. Use paper or sound toys to find the direction of sound at the ear level and horizontal level.

B. Using groups to encourage vocalization through interaction with other babies

6. At 7 months, improve your baby’s listening and vocal skills with toys or tools that make sounds when struck.

7. For 8 months, communication is encouraged by rolling a sounding ball.

8. At 9 months, place the baby on a dining chair and play toss to stimulate interest and language.

9. For 10 months, peek-a-boo, seek-and-find games, songs and movements are performed.

10. At 11 months, the easiest language is induced using toys related to animal sounds and onomatopoeia.

11. Combine movement and language, add comments to symbolic language, and complete short sentences.

4. Provision of parental counseling and educational materials

A. We provide ‘How should we talk to my child every day?’, a parent education material that compiles ways to present auditory language stimulation to infants and toddlers according to situations in daily life.

i. A total of 16 situations are ‘when waking up in the morning’, ‘when changing diapers’, ‘before and after bathing’, ‘when massaging’, ‘when breastfeeding’, ‘when putting on and taking off clothes’, and ‘when feeding the child’. , 'Reading a book', 'Playing with a ball', 'Playing with a child's doll or teddy bear', 'Playing a puzzle', 'Playing with a cow', 'Playing at the playground (slide, swing)', 'Playing with colored clay', 'Doing laundry. ', includes 'taking a walk'

B. Through a questionnaire, parents or primary caregivers are asked to write down all the questions they have during the hearing rehabilitation process at home, as well as questions about hearing loss and parenting in general, in real time every day, and then the rehabilitation staff member answers them in an appropriate manner for the child's development process and time. Professional counseling is provided continuously to ensure rehabilitation.

i. Use 'I'm curious' to receive questions about anything you have questions about during the child rehabilitation process every week.

C. KARI daily checklist is provided so that parents can record the situations and contents of intensive language stimulation to infants and toddlers, spontaneous vocalizations, and vocalizations during modeling every day.

D. KARI monthly checklist provided

i. Monthly age divided into 1~3, 4~5, 6~7, 8~9, 10~11, 12~13, 14~15, 16~17, 18~19, 20~21, and 22~24 months. Speech goals are presented and hearing rehabilitation methods at home are presented, including 'Dandong Siphun' and 'Attachment Parenting Method'.

E. Provide video education materials

i. The eight stages of listening signature behavior are understanding hearing loss and orthotics, using the orthotics during all waking hours, managing the orthotics, focusing on hearing sounds, providing meaningful experiences with sounds, and inducing concentration on sounds. Includes learning, teaching sound discrimination, and connecting the sounds and meanings of language.

5. Check and prepare professional materials

A. Session Plan and Report

i. It is designed to display the infant's basic information, short-term and long-term plans, pre-session checklist and parent education materials, status of orthotic devices, and Ling 6 Sound results at each session.

ii. A notes section is included to record the goals, tools, and results of the session, as well as infant and toddler characteristics seen in the session and parent reports.

B. Guidelines for hearing/speech rehabilitation of infants and toddlers

i. Contents that are essential to be aware of when performing auditory rehabilitation for hearing-impaired infants are divided into four areas.

1. Orthotics

2. Area development assessment

3. Rehabilitation progress and environment

4. Types of sound stimulation

6. Conduct tests to check progress after rehabilitation

A. Vocalization and speech area

i. Use the Korean Infant Vocal and Articulatory Development List (KIVAD) to determine the current vocal development stage of infants and toddlers of all ages.

ii. Inspection and analysis is conducted by analyzing video recordings.

B. Listening and Communication Area

i. The Infant-Toddler Auditory & Communicative Behavioral Checklist (IT-ACBA) is used to check the development of communicative behavior in infants and toddlers aged 0 to 26 months using the communication behavior checklist, a parent questionnaire.

ii. The development of listening skills in infants and toddlers aged 0 to 24 months was checked using the Infant-Toddler Meaningful Auditory Integration Scale (IT-MAIS), a parent questionnaire.

1. This test is conducted after confirming hearing loss in infants and wearing assistive devices (hearing aids, cochlear implants).

C. Global Development Area

i. The Parent Questionnaire, Developmental assessment for the Early intervention Program Planning (DEP), is used to check the overall development of infants and toddlers aged 0 to 36 months in terms of gross motor skills, fine motor skills, cognition, and social emotions.

D. Language area

i. The vocabulary development of infants and toddlers aged 8 to 36 months was checked using the Korean-MacArthur Bates Communicative Development Inventories (K-MBCDI), a parent questionnaire.

ii. The overall language development of infants and toddlers aged 4 to 36 months is checked using the Sequenced Language Scale for Infants (SELSI), a parent interview.

iii. The CSBS DP, which is conducted by analyzing video recordings, is used to check the development of communication skills in infants and toddlers aged 6 to 24 months.

In the above, preferred embodiments of the present disclosure have been shown and described, but the present disclosure is not limited to the specific embodiments described above, and may be used in the technical field pertaining to the disclosure without departing from the gist of the disclosure as claimed in the claims. Of course, various modifications can be made by those skilled in the art, and these modifications should not be understood individually from the technical ideas or perspectives of the present disclosure.

100: electronic device 110: speaker
120: Display 130: Microphone
140: Camera 150: Processor

Claims (10)

In a method of controlling an electronic device for auditory language cognitive rehabilitation of infants and young children,
A pre-test step of analyzing the video recorded by the test subject to identify the test subject's ability level for at least one of vocalization, listening, development, and communication;
A vocal rehabilitation step, providing rehabilitation content including a plurality of vocalization samples based on the identified ability level; and
After the vocal rehabilitation step, a post-test step of analyzing the video recorded by the test subject to identify the test subject's ability level for at least one of vocalization, listening, development, and communication,
The vocal rehabilitation stage is,
providing a first speech sample;
Identifying whether the test subject's voice imitating the first vocalization sample is input within a certain time from when the first vocalization sample is provided;
providing a second speech sample when the test subject's voice imitating the first speech sample is identified as being input; and
If the test subject's voice imitating the first speech sample is not identified as being input, providing the first speech sample again. According to paragraph 1,
The step of providing the first vocalization sample includes:
Displaying a plurality of image frames representing the mouth shape of the character speaking the first text through the display,
Outputting a first target sound including a voice in which the first text was uttered through a speaker,
The step of identifying whether the test subject's voice imitating the first speech sample is input,
Comparing a plurality of image frames obtained by photographing the subject through a camera with a plurality of image frames representing the mouth shape of the character,
When a sound including the voice of the test subject is input through a microphone, control of an electronic device that converts the audio data constituting the input sound into at least one text and compares the converted text with the first text. method. According to paragraph 2,
The step of providing the first vocalization sample again includes:
Identifying the direction of the examinee based on the image acquired through the camera,
displaying, through the display, a plurality of image frames representing the shape of the mouth of the character uttering the first text while looking in the identified direction;
A method of controlling an electronic device, outputting a first target sound including a voice in which the first text was uttered through the speaker at a louder volume than the step of providing the first speech sample. According to paragraph 1,
The control method of the electronic device is,
After the vocal rehabilitation step, a stimulation rehabilitation step of controlling the toy terminal based on communication with the toy terminal and providing sound through the toy terminal,
The stimulation rehabilitation stage is,
providing a first sound through the toy terminal; and
When the voice of the test subject is input after the first sound is provided, providing a second sound that imitates the voice of the test subject. delete According to paragraph 1,
The pre-test step is,
Analyzing one or more image frames included in the video to identify the subject and the guardian;
Recognizing the mouth shape of each of the test subject and the guardian over time;
Based on the mouth shape of each of the test subject and the guardian at each time, dividing the audio data included in the video into a time section including the voice of the test subject and a time section including the voice of the guardian;
An electronic device that compares a first text converted from the guardian's voice input in a first time interval with a second text converted from the test subject's voice inputted in a second time interval following the first time interval. Control method. delete delete In electronic devices,
mike;
camera;
display;
speaker; and
Including a processor connected to the microphone, camera, display, and speaker,
The processor,
Perform a pre-test step of capturing the test subject through the microphone and the camera to generate a video, and analyzing the video to identify the test subject's ability level for at least one of vocalization, listening, development, and communication; ,
Based on the identified competency level, performing vocal rehabilitation steps, providing rehabilitation content including a plurality of vocal samples,
After the rehabilitation step, performing a post-test step in which the test subject analyzes the recorded video to identify the test subject's ability level for at least one of vocalization, listening, development, and communication,
In performing the vocal rehabilitation step, the processor
providing a first speech sample through the speaker and the display,
Identifying whether the test subject's voice imitating the first vocalization sample is input through the microphone within a certain time from when the first vocalization sample is provided,
If the test subject's voice imitating the first speech sample is identified as input, providing a second speech sample,
If the test subject's voice imitating the first speech sample is not identified as input, the electronic device provides the first speech sample again. In a computer program stored on a computer-readable medium,
A computer program that is executed by a processor of an electronic device and causes the electronic device to perform the control method of claim 1.