KR20180052952A - Service Appratus and System for training Communication based voice recogintion - Google Patents

Abstract

The present invention relates to a service apparatus for training communication based on voice recognition. The service apparatus for training communication based on voice recognition of the present invention comprises: an acoustic adaptation model, in which an acoustic model, modeled by using small-scale speech data generated by late-talking children and characteristics of speech and articulation errors of the late-talking children analyzed by the speech data, is adapted based on a retained acoustic model; a voice recognition unit, in which a voice signal of a user is inputted in response to a service for training communication, and then the voice signal of a user is compared with the acoustic adaption model to recognize the inputted voice signal; and a learning providing unit, in which a service for the next training course is provided when the recognized voice signal matches with predetermined information. Accordingly, it is possible to enhance communication skills of the late-talking children.

Description

Technical Field [0001] The present invention relates to a voice recognition-based communication training service apparatus and system,

The present invention relates to a speech recognition-based communication training service apparatus and a system for providing services for communication training of a child having a slow speech and language.

More specifically, based on a retention acoustic model, a small-scale speech data generated by a child with a late speech and an acoustic adaptation adapted to an acoustic model modeled using a speech characteristic and an articulatory error characteristic of a late- A speech recognition unit for recognizing the input speech signal in comparison with the acoustic adaptation model when a user's speech signal is input in response to a model and a communication training service; And a learning providing unit for providing a service for the speech recognition based communication training apparatus.

The language is learned through books and objects expressing vocabulary in early childhood by means of voices and letters used to express or convey ideas and feelings. However, language development may vary depending on the child (child).

Approximately 8% of the child population is surveyed to have speech and language disorders (Nelson et al. 2006). These children can improve communication skills by receiving language therapy before the age of four.

Therefore, although children with language disabilities and children with no language disabilities do not have language development, children with language development tend to receive language treatment through specialists such as language rehabilitation institutions. However, There is a problem that it is required.

In addition, various services for language learning are provided through the Internet and the like, but most of them use tools, information services or non-professional contents.

Conventional services that provide early childhood learning provide learning through simple knowledge transfer using agitation, children animation, and word cards. These learning services are difficult to use for children with speech disorders such as hearing impairment, or for children whose language development is slow but not impaired.

Therefore, it is necessary to provide a learning service for children who have language disabilities such as hearing impairment, or children who have no disability but have slow language development.

1. Korea Open Patent No. 2012-000032 (2012.03.03. Open)

It is an object of the present invention to provide a communication training service through a story that can be experienced in everyday life based on speech recognition using an acoustic adaptive model that reflects a speech characteristic and an articulatory error characteristic of a slow language child, And to provide a communication training service apparatus and system based on speech recognition capable of improving communication.

In addition, it provides communication training services using authenticated language development expert pool, multimedia library and story scenario, and communication based on speech recognition that can improve service by collecting and analyzing usability evaluation. Training service devices and systems.

According to an aspect of the present invention, there is provided a speech recognition-based communication training service apparatus, comprising: a speech recognition module for generating speech data based on a retention acoustic model, When the user's speech signal is input in correspondence with the communication training service and the acoustic adaptation model adapted to the acoustic model modeled using the speech characteristic and the articulation error characteristic of the late child, And a learning and providing unit for providing a service for the next training process if the recognized voice signal matches the preset information.

In addition, the learning and providing unit can provide a story-based communication training service that can be experienced in daily life using a pool-based multimedia library.

Also, the learning and providing unit may provide a story-based communication training service in which at least one of recognition and vocalization through lexical formation, user action corresponding to the presented voice, voice feedback and voice interaction is induced, If the user action is consistent with the predetermined information, the sound effect and the screen corresponding to the story can be provided.

In addition, the speech recognition-based communication training service is provided by an app installed and received from the service providing server, and the learning providing unit provides the learning information generated at the time of using the communication training service to the service providing server, The providing server analyzes the usability evaluation collected through the questionnaire and provides it to the administrator terminal, and can manage the user learning situation using the transmitted learning information.

Meanwhile, the speech recognition-based communication training service system according to an embodiment of the present invention is a communication training service system based on the speech recognition, in which a user's voice signal is input in response to a story-based communication training service, Based communication that recognizes the input speech signal in comparison with the acoustic adaptation model reflecting the difference in speech characteristics and provides the next training learning or iterative learning according to the degree of matching of the recognized speech signal with the preset information, And a service providing server for receiving the learning information according to the story-based communication training service and managing the user learning situation when the training service apparatus and the speech recognition-based communication training service apparatus are connected to the communication network.

As described above, according to the present invention, the speech recognition-based communication training service apparatus and system of the present invention can be applied to communication training based on speech recognition using an acoustic adaptive model that reflects the vocal characteristic and articulatory error characteristic of a slow- By providing the service, communication of a child with a late language can be improved.

In addition, by providing communication training services based on stories that can be experienced in everyday life, it provides familiarity with children and trains communication of daily life. have.

The present invention also provides a story-based communication training service in which at least one of cognition and utterance through lexical shaping, user action corresponding to a presented voice, voice feedback and voice interaction is induced, so that a child can listen and speak naturally.

In addition, expertise and reliability are verified by participating in certified language development experts, providing communication training services using generated vocabulary pools, multimedia libraries, and story scenarios.

In addition, usability evaluation according to service utilization can be reflected in service improvement by collecting and analyzing.

FIG. 1 is a block diagram schematically illustrating the configuration of a speech recognition-based communication training service system according to an embodiment of the present invention.
2 is a block diagram schematically illustrating the configuration of a speech recognition-based communication training service apparatus according to an embodiment of the present invention.
3 to 8 are views for explaining a communication training service according to an embodiment of the present invention.

The terms and words used in the present specification and claims should not be construed as limited to ordinary or dictionary terms and the inventor may properly define the concept of the term in order to best describe its invention It should be construed as meaning and concept consistent with the technical idea of the present invention.

Therefore, the embodiments described in the present specification and the configurations shown in the drawings are merely the most preferred embodiments of the present invention and are not intended to represent all of the technical ideas of the present invention. Therefore, various equivalents It should be understood that water and variations may be present.

Before describing the present invention with reference to the accompanying drawings, it should be noted that the present invention is not described or specifically described with respect to a known configuration that can be easily added by a person skilled in the art, Let the sound be revealed.

FIG. 1 is a block diagram schematically illustrating the configuration of a speech recognition-based communication training service system according to an embodiment of the present invention.

1, a communication training service system 1000 according to an embodiment of the present invention may include a speech recognition-based communication training service device 100, a service providing server 200, and a DB 300 .

The speech recognition-based communication training service apparatus 100 according to an embodiment of the present invention includes a wireless Internet, wifi  A speech recognition-based communication training service for communication training of a child with a slow language from the service providing server 200 through the network 10 App (APP) type It can be received and installed.

At this time, the speech recognition-based communication training service apparatus 100 Your app  Once installed, without connection to the service providing server 200 via the network, Based on speech recognition We can provide communication training service.

Accordingly, the voice recognition-based communication training service apparatus 100 can receive a voice recognition-based communication training service app from the service providing server 200 and install the same on a smart phone, Tapes, PCs, notebooks, etc., as well as toys and learning devices that can provide app services according to an embodiment of the present invention.

The speech recognition-based communication training service according to an embodiment of the present invention is a communication training service for a child who is late in language due to a hearing impairment, a language development delay, or the like, through a multimedia image provided based on a story that can be experienced in everyday life It can be a learning service that can induce children to listen and speak naturally.

The service providing server 200 provides a speech recognition based communication training service app. At this time, the application can provide a voice recognition-based communication training service using the service DB 113 without connecting to the network 10 with the service providing server 200.

To this end, the service providing server 200 can provide an application by applying an acoustic model compression technique using State tying, Mixture tying, and codebook.

The DB 300 can store and manage various information generated according to the voice recognition-based communication training service application and the voice recognition-based communication training service using the user. At this time, the information to be stored may be user-specific learning information, usability evaluation information, and the like.

2 is a block diagram schematically illustrating the configuration of a speech recognition-based communication training service apparatus according to an embodiment of the present invention.

Referring to FIG. 2, the speech recognition-based communication training service apparatus 100 includes a communication training Service Apps (110), an input unit (120), a microphone (130) Display portion (140) and a sound output unit (150). In addition, The app 110 Providing learning The portions 111, The voice recognition unit (112) and a service DB (113). Also, the service DB 113 Vocabulary pool An audio adaptation model 114, an acoustic adaptation model 115, and a multimedia library 116.

The communication training service app 110 of the communication training service apparatus 100 based on the speech recognition may be an app installed and received from the service providing server 200 through the network 10. [

At this time, the speech recognition-based communication training service apparatus 100 includes an input unit 120 for receiving a user selection through a touch or the like for operating an application, a microphone 130 for receiving a user voice, A display unit 140 for displaying the service and a sound output unit 150 for outputting the sound provided by the app.

The speech recognition-based communication training service apparatus 100 can provide a communication training service based on the information of the service DB 113.

Here, the lexical pool (Pool: 114) is a list of training vocabulary proved by experts based on language rehabilitation clinics. It is composed of vocabulary learning vocabulary based on sounds, animals, ride and vocabulary for learning Korean List or the like.

The acoustic adaptation model 115 analyzes voice characteristics and articulatory error characteristics of children who are late in speech using voice data of a small voice data DB and extracts phoneme sets and voice features suitable for the child, And language developmental level.

The acoustic adaptation model 115 can be constructed as an acoustic adaptation model for speech recognition by reflecting the analyzed speech characteristic difference in the small voice data DB and adapting the voice data of the small voice data DB based on the retention acoustic model.

Here, the retention acoustic model may be an acoustic model applied to a conventional Korean speech recognition technology, such as a Korean adult acoustic model previously stored and a Korean child acoustic model. At this time, the stored acoustic models may be acoustic data based on the vocabulary pool 114. [

In addition, the small-scale voice data DB can be a DB constructed by voice data generated by children whose language is late, based on the vocabulary pool 114. At this time, it can be constructed by varying the number of vocal repetition depending on the type of the child's disorder and the degree of language development.

Here, the retention acoustic model and the small voice data DB are stored in the DB 300 managed by the service providing server 200, and only the acoustic adaptation model modeled using the acoustic adaptation model is configured in the voice recognition-based communication training service apparatus 100).

At this time, the acoustic adaptation model can be modeled according to adaptive techniques such as Maximum Likelihood Linear Regression (MLLR) and Maximum a Posteriori (MAP). Accordingly, the acoustic adaptation model 340 can be a speaker adaptation model that reflects the difference in the utterance characteristics depending on the type of disorder and the degree of language development.

The multimedia library 116 can store and manage templates, images, sound effects, voice data, moving pictures, and the like according to story scenarios based on the vocabulary pool 114.

At this time, the story scenario can be designed by child language development experts based on the process of the interaction of the language therapist with the child in the clinic, and the story that the child can directly or indirectly experience in everyday life.

The learning providing unit 111 may provide a communication training service based on a story scenario corresponding to a related target set according to the speaking area and the listening area to the speech recognition based communication training service apparatus 100. [

Here, the related target according to the speaking area and the listening area can be set as shown in Table 1 below.

domain Related Goals




Listening Detect and find directions Finding and responding to a speaker Respond to music Perform conditional play audiometry Respond appropriately to meaningful sounds Sound - Development of the target connection Development of auditory memory Development of auditory sequencing Listen to the description and check the items Hearing finish (Closing) Follow instructions Recall the central element of the story The development of listening-cognitive skills


Words Development of supersymmetry characteristics of speech The development of specific vowels and consonants Omission of speech / elimination of confrontation Adjust / Enhance voice quality Participation of voice play Perform phonemic identification / recognition task Output of words and boilerplate Reduction and elimination of certain phonological variations

In addition, the learning providing unit 111 can construct a story scenario using the service DB 113 constructed according to the related target according to the speaking area and the listening area. At this time, the story scenario may be configured such that at least one of recognition and utterance through lexical formation, user action corresponding to the presented voice, voice feedback and voice interaction may be induced.

Here, recognition and vocalization through lexical shaping can be a method of visually recognizing a word that is shaped by a user, that is, a user having a slow language, by voicing certain words, and voicing the word.

In addition, the user action corresponding to the presented voice may be a method of providing a plurality of shaped words and selecting a word corresponding to the presented voice.

In addition, the voice feedback may be a method in which a user utters a specific word according to a utterance request for a specific word.

In addition, the voice interaction may be a method of communicating with the user about the provided image.

3 to 8 are views for explaining a communication training service according to an embodiment of the present invention. As an example of the communication training service according to an embodiment of the present invention, five story scenarios can be described with reference to FIG. 3 to FIG.

[ story  Scenario 1] View the image on the screen, speak the word

The learning unit 111 provides various images of a specific place, space, and the like in an everyday life, and provides a visual image at random to recognize the child and speak the specific word Let's do it.

Story scenario 1 can be speech training. Referring to FIG. 3, when the main character recalls a specific food material (fish or fish) to visualize and provide food materials in the refrigerator, the child visually recognizes the fish and can speak the word .

At this time, when a voice uttered through the microphone 130 is input, the voice recognition unit 112 receives the voice signal and compares the voice signal with the acoustic adaptation model 340 to recognize the voice signal.

The learning providing unit 111 can determine whether the learning is correct according to the voice signal recognized by the voice recognition unit 112. [ At this time, when the degree of coincidence between the predetermined voice information and the voice signal recognized by the voice recognition server 220 is within the predetermined range, it is determined that the answer is correct, and the degree of language development of the child is determined according to the degree of coincidence .

At this time, if the learning providing unit 111 is correct, it is possible to provide a specific next random image. On the other hand, if the answer is not correct, the image may be repeatedly learned by the preset number of times.

When the learning providing unit 111 determines that the answer is correct, the learning providing unit 111 can provide effects such as a highlight and an effect sound according to the correct answer. In addition, the learning providing unit 111 may provide a specific effect image providing a hint, a sound effect according to an incorrect answer, and the like, when it is determined that the answer is incorrect.

On the other hand, in the story scenarios 2 and 5, it is assumed that the speech recognition unit 112 recognizes the speech signal, determines whether or not the answer to the learning aid 111 is correct, determines the degree of language development of the child, Can be applied. In addition, the effect image and the effect sound provided according to the correct answer can be applied to all story scenarios.

Meanwhile, FIG. 3 provides a word with the nested image as a noun, but may express the verb as shown in FIG. As shown in Fig. 4, images of a to d can be sequentially displayed to form a verb "melt". At this time, the child can recognize and "speak" through the provided image.

[ story  Scenario 2] Speaking based on the image on the screen and the voice presented

Story Scenario 2 can be a long long speech training. Referring to FIG. 5 (a), the learning part 111 selects one of long sounds (ㅃ, ㄸ, ㅉ, ㄲ), and referring to FIG. 5 (b) And a voice corresponding to the image is provided so that the child can utter the voice. At this time, it is possible to provide an iterative learning such that the number of words predetermined in one activity, for example, five words can be spoken.

[ story  Scenario 3] Understanding and acting on the behaviors of the presented voice

Story Scenario 3 can be a training to improve structured listening skills. Referring to FIG. 6, the learning aid 111 selects one of the verbs to be activated, for example, ta, ta, fur, bo, eat, etc. and provides a voice for the selected vocabulary, The user can touch the image and execute the action corresponding to the voice in the image.

For example, if the user selects the word "eat ", the user can provide the image in which the rice and side dishes are prepared and provide the main character in which the user can perform the action. In addition, a voice for an action such as "eat" or "eat rice" is presented so that the user can touch the food image and move the food image to the main character's mouth through dragging or the like. At this time, the activity for the selected vocabulary can be provided so that it can be repeatedly learned a predetermined number of times.

Here, the story scenarios 3 and 4 are learning for improving the auditory memory, and it is possible to judge whether or not an answer is made by the user action according to the provided voice rather than the correct answer according to the speech recognition.

In other words, the learning providing unit 111 can determine the correct answer of the learning by judging the position of the child to touch on the screen, the position to move, etc. according to the selected vocabulary and the provided voice.

At this time, the learning providing unit 111 may provide the next learning through selecting the next vocabulary after providing iterative learning for the corresponding vocabulary by the preset number of times.

[ story  Scenario 4] Understanding what the provided speech means and choosing

Story Scenario 4 can be a training to listen to and understand the description. Referring to FIG. 7, an environment in which an activity can be experienced in daily life, for example, a home, a subway, a garden, a kindergarten, and the like is selected and selected and a description corresponding to the selected environment is provided, I will.

For example, in FIG. 7, when a subway is selected, a corresponding screen is provided, and a character of a specific person among the various people sitting on the subway is provided as a description text so that the user can select the specific person.

According to the selected environment and the provided description text, the learning providing unit 111 can determine the position of the child to touch on the screen and determine whether or not the learning is correct.

At this time, the learning providing unit 111 can provide the preset number of the description text activities in the selected environment so that the learning can be repeatedly performed. When the repeated learning is completed, the learning can be provided through the next environment selection.

[ story  Scenario 5] Answering the question of the child's voice

Story Scenario 5 can be a listening and memorizing speaking exercise. Referring to FIG. 8, it is possible to provide an environment that can be experienced in daily life, for example, by providing a window, a front yard, etc., and provide a voice in a form of a question depending on the selected environment. At this time, the user can recognize the situation of the provided screen and provide an answer to the presented question.

Taking FIG. 8 as an example, if a voice of "Who is on the tree?" Is provided along with the screen of FIG. 8, the child can answer "New" through the provided screen.

At this time, when a voice response generated by the child is input through the microphone 130, the voice recognition unit 112 receives the voice signal and compares the voice response with the acoustic adaptation model 115 to recognize the voice signal have.

The learning providing unit 111 can determine whether the learning is correct according to the voice signal recognized by the voice recognition unit 112. [ At this time, when the degree of coincidence between the speech information preset by the correct answer and the speech signal recognized by the speech recognition unit 112 is within the predetermined range, it is determined that the answer is correct, and the degree of language development of the child is determined according to the degree of coincidence .

After the use of the communication training service, the learning providing unit 111 may provide a questionnaire for the usability evaluation through the display unit 140 when the learning ends. At this time, the questionnaire can be classified according to the user, the child, the guardian, the speech therapist and the special teacher. At this time, the questionnaire may be satisfaction of service utilization, difficulty, inconvenience, additional content demand, and so on.

The learning providing unit 111 collects usability evaluations created by the user and analyzes the collected usability evaluations and provides them to a display unit (not shown) included in the administrator terminal (not shown) and the service providing server 200 . Therefore, the usability evaluation result can be reflected in the service and updated.

In addition, the input unit 120 may set a network connection option. The speech recognition-based communication training service apparatus 100 includes a user learning recording file, a learning progress status, a learning correction rate, and the like generated by the learning providing unit 111 when the apparatus 100 is connected to the network 10 with the service providing server 200 And information on the usability evaluation prepared through the questionnaire can be transmitted to the service providing server 200.

The service providing server 200 can manage the learning status of the user using the received information. In addition, the service providing server 200 may provide the learning status of the user to an associated terminal (not shown) such as a language therapist or parents.

As a result, by providing communication training services based on stories that can be experienced in everyday life, it is possible to train the communication of everyday life while providing familiarity with the child and allow the child to listen and speak naturally.

1 to 8 described above only describe the main matters of the present invention. As various designs can be made within the technical scope thereof, the present invention is not limited to the configurations of Figs. 1 to 8 It is self-evident.

1000: Speech recognition based communication training service system
100: Speech recognition based communication training service device
110: Communication Training Apps
111: learning aid 112: speech recognition unit
113: service DB 114: vocabulary pool
115: Acoustic adaptation model 116: Multimedia library
120: input unit 130: microphone
140: display unit 150: sound output unit
200: service providing server 300: DB

Claims (5)

An acoustic adaptation model that adapts an acoustic model modeled using vocal characteristics and articulatory error characteristics of a child with a late speech analyzed by the small voice data generated by a child with a late language based on a possessed acoustic model.
A speech recognition unit for recognizing the input speech signal in comparison with the acoustic adaptation model when a speech signal of a user is input in response to a communication training service; And
And a learning and providing unit for providing a service for a next training process if the recognized voice signal matches the preset information.
The method according to claim 1,
The learning /
Based communication training service using a vocabulary pool based multimedia library to provide a story-based communication training service that can be experienced in daily life.
The method according to claim 1,
The learning /
A story-based communication training service in which at least one of recognition and vocalization through lexical shaping, user action corresponding to a presented voice, voice feedback, and voice interaction is induced,
And provides a sound effect and a screen corresponding to the story if the user action corresponding to the presented voice coincides with predetermined information.
The method according to claim 1,
The speech recognition-based communication training service is provided by an application received from a service providing server,
Wherein the learning and providing unit provides the learning information generated at the time of using the communication training service to the service providing server,
Wherein the service providing server manages a user learning situation using the transmitted learning information.
When the user's speech signal is input in response to the story-based communication training service, the input speech signal is recognized by comparing with the acoustic adaptation model that reflects the difference in the speech characteristics according to the type of the obstacle and the degree of language development of the child, A speech recognition-based communication training service device for providing the next training learning or iterative learning according to the degree of matching of the recognized voice signal with the preset information; And
Based communication training service apparatus and a communication network are connected to each other, learning information according to the use of the story-based communication training service is received and the user learning situation is managed.

Images