KR102568994B1 - Apparatus and method for providing feedback for improving a user's speech ability - Google Patents

Abstract

According to an embodiment of the present disclosure, a method of operating a server for improving a speech ability of a car user includes obtaining visual and auditory inputs in real time based on the speech of the user, and data corresponding to the inputs. It may include a process of generating a process, a process of generating feedback information corresponding to the user's speech in response to generating the data, and a process of providing a video or audio output according to the feedback information.

Description

Feedback device and method for improving a user's speech ability {Apparatus and method for providing feedback for improving a user's speech ability}

Various embodiments of the present disclosure relate to a technique for providing feedback to improve a user's speech ability, for example, providing speech feedback using virtual reality (VR) content.

Speech in front of a large audience lacks opportunities to practice in advance due to environmental factors, and even if such an opportunity is given, it is difficult to receive immediate feedback, so it is difficult to find clear technical measures to improve speech ability. .

Recently, the VR content industry has entered a rapid growth trend, and although attempts are continuously being made to provide realistic experiences and feedback by applying it to various areas of life, solving technical challenges to improve speech ability using VR technology still far away

The present disclosure provides an apparatus and method for generating adaptive feedback for improving a user's speech ability based on VR content technology.

According to an embodiment of the present disclosure, a method of operating a server for improving a speech ability of a car user includes obtaining visual and auditory inputs in real time based on the speech of the user, and data corresponding to the inputs. It may include a process of generating a process, a process of generating feedback information corresponding to the user's speech in response to generating the data, and a process of providing a video or audio output according to the feedback information.

According to an embodiment of the present disclosure, an apparatus of a server for improving a user's speech ability includes a transceiver; and at least one control unit operatively connected to the transceiver, wherein the at least one control unit performs a process of acquiring visual and auditory inputs in real time based on the user's speech, and data corresponding to the inputs. It may include a process of generating a process, a process of generating feedback information corresponding to the user's speech in response to generating the data, and a process of providing a video or audio output according to the feedback information.

The apparatus and method according to various embodiments of the present disclosure can effectively improve speech ability of a user by reflecting speech input in real time and adaptively providing VR content-type feedback.

Effects obtainable in the present disclosure are not limited to the effects mentioned above, and other effects not mentioned may be clearly understood by those skilled in the art from the description below. will be.

1 illustrates the configuration of a system for improving a user's speech ability according to an embodiment of the present disclosure.
2 is a flowchart illustrating an operation of a server for improving user speech capability according to an embodiment of the present disclosure.
3 is a flowchart illustrating operations of a server for improving a user's speech capability according to another embodiment of the present disclosure.
4 illustrates classification of feedback information output by a server for improving a user's speech ability according to an embodiment of the present disclosure.

Before entering into the detailed description below, it may be helpful to understand the present invention to set forth definitions of certain words and phrases used throughout this patent document. The term “couple” and its derivatives may refer to any direct or indirect communication between two or more elements, or whether or not these elements are in physical contact with each other. The terms "transmit", "receive" and "communicate", as well as their derivatives, include direct and indirect communication thereof. The terms "include" and "comprise", as well as their derivatives, mean inclusion without limitation. The term "or" is inclusive and means 'and/or'. The phrase "associated with", as well as its derivatives, also include, be included within, interconnect with, contain , be contained within, connect to or with, couple to or with, be communicable with, cooperate with , interleave, juxtapose, be proximate to, be bound to or with, have, have properties It can mean a property of), have a relationship to or with. The term “controller” means any device, system, or part thereof that controls at least one operation. Such a controller may be implemented in hardware or a combination of hardware and software and/or firmware. Functions related to a particular controller may be centralized or distributed, either locally or remotely. When the phrase “at least one” is used with a listed item, it means that different combinations of one or more of the listed items may be used. For example, "at least one of A, B, C" includes any one of A, B, C, A and B, A and C, B and C, and A and B and C combinations.

In addition, various functions described below may be implemented or supported by one or more computer programs, and each computer program is formed as computer readable program code and implemented in a computer readable medium. The terms "application" and "program" refer to one or more computer programs, software components, instruction sets, procedures, functions, objects, classes, instances, related data, or portions thereof adapted for implementation in an appropriate computer readable program. . The phrase "computer readable program code" includes all types of computer code, including source code, object code and executable code. The phrase "computer readable medium" means a medium stored by a computer, including read only memory (ROM), random access memory (RAM), hard disk drive, compact disc (CD), digital It includes any type of medium that can be accessed. It may be a digital video disc (DVD), or any other type of memory. A “non-transitory” computer readable medium excludes wired, wireless, optical or other communication links that transmit transitory electrical or other signals. Non-transitory computer readable media include media on which data can be permanently stored and media on which data can be stored and later overwritten, such as rewritable optical discs or removable memory devices.

Definitions of certain words and phrases are provided throughout this disclosure, and skilled artisans should understand that, in most cases, these definitions may apply to prior as well as future uses of the defined words and phrases. do.

1 through 4 discussed below and the various embodiments used to explain the principles of the present disclosure are for illustrative purposes only and should not be construed as limiting the scope of the present disclosure in any way. Those skilled in the art will appreciate that the principles of this disclosure may be implemented in any suitably arranged wireless communication system.

1 illustrates the configuration of a system for improving a user's speech ability according to an embodiment of the present disclosure.

Referring to FIG. 1 , a system according to an embodiment of the present disclosure may include a client device, a voice recognition device, and an external electronic device.

Although not shown in FIG. 1, the system according to an embodiment of the present disclosure may further include a server. The server may control data transmission or reception between the client device, the voice recognition device, and an external electronic device, and is electrically or operably connected to the corresponding devices included in the system, or exchanges signals and data through a wireless network. can be configured to

The client device may be an electronic device configured to be carried or worn by a user. By way of example, the client device may be a head mounted display (HMD) worn on the head by a user. In this case, the client device may provide VR contents, voice messages, and the like to the user through a display or speaker included in the client device itself. The client device may output feedback information generated and transmitted in response to the user's speech so that the user may experience it.

The voice recognition device may be an electronic device configured to be carried or worn by a user. For example, the voice recognition device may be an electronic device in the form of a microphone or recorder that generates a digital signal based on a user's speech voice and transmits the digital signal to an external server or computing device. Voice content generated by the user while the user is speaking may be converted into data through a voice recognition device, and may be referred to as voice data. Voice data may be transmitted through an external server or an external device, or may be stored in a storage provided inside the voice recognition device. The client device and the voice recognition device may exchange voice and video data obtained in real time or synchronize them to be compatible with each other.

The external electronic device may be operatively connected to the client device and/or the voice recognition device to receive audio data or video data. Reception of audio data or video data may be performed based on a pre-established wireless network. The external electronic device may include, for example, a mobile phone terminal, a smart phone, a notebook computer, and a personal computer. More preferably, it may be an electronic device equipped with a camera and configured to acquire or store a motion of a user performing a speech as a dynamic image. Thus, it can be a laptop computer, smart phone or tablet PC equipped with a webcam.

The external electronic device may obtain a static or dynamic image corresponding to a user's behavior, gesture, gait, complexion, facial expression, etc. performing speech, and may process and convert it into data.

In the system according to an embodiment of the present disclosure, when a user carrying a client device speaks, visual information such as movement, expression, and gesture of the user is obtained through an input unit (eg, camera) of an external electronic device and can be processed In the present disclosure, only an embodiment in which an external electronic device acquires behavioral information about a user's movement has been described, but acquisition of the user's behavioral information through an input unit provided in the client device itself is not excluded.

In the system according to an embodiment of the present disclosure, when a user carrying a voice recognition device performs speech, auditory information such as tone, voice, intonation, text included in speech, volume, tempo, etc. is used for voice recognition. It may be obtained and processed through the device, and the acquired and processed auditory information may be transmitted to an external electronic device or server.

Voice and video data acquired or received by the client device, voice recognition device, and external electronic device may be transmitted to the server. Although the server is disclosed in the present disclosure, other than the server, various known means may be used as long as the device operates to collect and process acquired audio and video data.

The server may generate feedback information according to a pre-stored database based on the acquired audio and video data. The generated feedback information may be output in real time through an external electronic device or a client device.

2 is a flowchart illustrating an operation of a server for improving user speech capability according to an embodiment of the present disclosure.

Referring to FIG. 2, the server may perform a series of operations in a situation where a user's speech is made, and the operations described below may be performed step by step, but the sequence of the described steps limits the time-sequential order of the operations to be performed. it may not be

In step 201, the server obtains visual and auditory input based on the user's speech.

Visual and auditory input obtained by the server may be obtained through an external input device operatively connected to the server. For example, an external input device equipped with a microphone or camera first acquires voice input or video input, and the external input device transmits it to the server or processes it into audio or video data and transmits it to the server.

Visual and auditory input may be based on the user's speech. Accordingly, the visual and auditory inputs may include information related to sound and motion information generated in the course of the user's speech.

In step 203, the server generates data corresponding to the input.

Data corresponding to the input may refer to data related to or associated with visual and auditory inputs obtained by the server. In this case, data may mean not only converting analog signals corresponding to visual and auditory inputs into digital signals, but also classifying inputs according to preset criteria to generate speech feedback. For example, an operation in which a server obtains auditory input and generates voice data based thereon may mean generating voice data processed according to predetermined classification criteria such as sentence structure, tone, intonation, and keywords. can Similarly, an operation by which the server obtains a visual input and generates image data based thereon may mean generating image data processed according to a predetermined classification criterion, such as a gesture, posture, facial expression, complexion, and the like.

In step 205, the server generates feedback information corresponding to the user's speech in response to generating the data.

Generation of the feedback information by the server may be performed based on pre-stored criteria. The pre-stored criterion may correspond to an evaluation index for generating feedback. For example, when the server determines through voice data that the speech speed is too fast, the server may generate feedback corresponding to 'speech speed is fast'. In this case, whether the speech speed is fast may be determined according to a preset criterion, for example, a critical tempo.

In step 207, the server provides video or audio output according to the feedback information.

The server may provide feedback information through video or audio so that the user performing the speech can check the feedback information.

At this time, the provision of feedback information may be performed by a speaker or display device operatively connected to the server. For example, when a user is wearing an HMD capable of experiencing VR content, the server may output feedback such as audience cheers and applause to the HMD through a 360-degree packaging image. At this time, voice feedback corresponding to ?T, applause, etc. of the audience may be output through a speaker connected to the server.

An operation of providing feedback information by the server may be performed in real time in response to an operation of obtaining visual and auditory inputs from the user.

3 is a flowchart illustrating operations of a server for improving a user's speech capability according to another embodiment of the present disclosure.

At step 301, the server obtains speech input.

Speech input can be obtained in the form of auditory and visual input. This step may be similar to step 201 of FIG. 2 .

In step 303, the server determines parameters related to the feedback information.

A parameter related to feedback information may be a variable that determines the type of feedback information. The type of feedback information may be determined based on an evaluation index for evaluating a user's speech. For example, the type of feedback information may include at least one of basic speech feedback, content delivery capability feedback, real-time response emotion feedback, and speech behavior feedback. In addition to the feedback types listed above, types related to basic criteria for evaluating speech may be further included.

In step 305, the server determines the item of speech input corresponding to the parameter.

Items of the speech input corresponding to the parameters may refer to items to be considered in order to evaluate speech according to the parameters. These items may be determined according to pre-stored information or may be added by a user's input. For example, when the parameter, that is, the type of feedback information is real-time emotional response feedback, the speech input item may be information such as the user's electrocardiogram, iris, and eye changes.

Items of speech input may be determined based on acquired audio and video data. Accordingly, when the parameter is real-time emotional response feedback, the speech input item may be determined mainly based on video data.

Items of the speech input may overlap each other even if they correspond to different parameters. For example, in both the real-time emotional response feedback and the speech behavior feedback, whether or not a facial expression has changed may be a criterion for determining feedback generation.

At step 307, the server determines a measurement value corresponding to the breakdown of the speech input.

The server may determine a numerical value of a region corresponding to the item part of the determined speech input in the audio and video data determined based on the speech. For example, when the item of the speech input is the degree of change in facial expression, the server may determine the degree of change in the measured facial expression based on the image data. In this case, the change in facial expression may be digitized or leveled based on a preset table.

In step 309, the server determines whether the measured value is greater than or equal to a threshold value.

The threshold value may be determined based on predetermined data. At this time, the threshold value may be a criterion for determining whether the feedback generated by the server is positive or negative. Accordingly, predetermined data for determining the threshold value may be continuously updated according to changes in the environment or a user performing a speech.

In response to the server determining that the measurement value is greater than or equal to the threshold value, the server may perform step 313. In response to the server determining that the measurement value is less than the threshold value, the server may perform step 311 .

At step 311, the server outputs positive feedback.

The server may output a positive feedback to the external electronic device in response to the determination that the measurement value does not fall short of the threshold value. For example, a cheering image, a clapping image, a gesture of interest, and the like of the audience may be output through a display that can be viewed by the user.

At step 313, the server outputs negative feedback.

The server may output negative feedback to an external electronic device in response to determining that the measured value is greater than or equal to the threshold value. For example, it is possible to output images of the audience booing, indifference, and leaving the room through a display that the user can view.

The server may provide positive or negative feedback to the user in real time while the user is performing the speech. In this case, the user may be in an environment where feedback can be checked while performing speech.

4 illustrates classification of feedback information output by a server for improving a user's speech ability according to an embodiment of the present disclosure.

The server may acquire speech input from the outside.

The server may generate feedback information based on the acquired speech input.

Specifically, the server can provide a situation adaptation service that adapts to and gets used to various situations that occur during speech by creating realistic VR situations based on VR/AI technology.

The server includes a control unit, and the control unit of the server may be driven by an application. The application can set specific conditions for generating feedback information, and check and record the user's behavior and emotions in the field through the client device. In addition, the application can provide a realistic speech ability improvement training service that judges the accuracy and fluency of speech with AI's voice recognition and allows the feedback to check the audience's reaction in the virtual world in real time.

Through the application, you can input data that requires feedback, such as specific situation conditions, place, number of audience members, and time settings. Even the appearance is recorded so that it can be stored and checked.

The client device can know the facial expression by changing the eyes, and by enabling iris recognition and electrocardiogram measurement, situations and times that cause tension are analyzed and recorded. AI's voice recognition measures the presenter's prepared information delivery ability and situational proficiency by checking the vibration, tone, speed, and pronunciation of the presenter's voice, and can reflect it as default behavioral feedback from the audience.

At this time, the audience feedback may be realistic actions and expressions of the audience that change through the actions and voices of the presenter, and through this, the user can prepare in advance for causing tension due to the audience.

When giving a speech, the use of appropriate language is also required through data of various speech situations such as an auditorium with many people, business situations, classes, and interviews. This part that responds through voice recognition is not only the speaker's confidence that can be known by voice, but also the part that can prevent the use of language necessary in the field, such as habitual interjections and slang. In addition to the native language, it is possible to set a second foreign language, so you can receive feedback on correct grammar, word order, and pronunciation.

When the set time is over, you can receive feedback through the application, and you can organize the script you prepared or the key keywords you need to deliver. The feedback is dataized and organized, and through the recorded video, you can see and correct your appearance when you are habitually nervous.

In the system according to an embodiment of the present disclosure, feedback information may be classified into two types according to the nature of the feedback information.

Audience feedback may be information that allows an immediate view of the audience's reaction as the speech progresses. At this time, the reaction of the audience can be changed according to the priority of the desired feedback, and the unexpected behavior of the audience can be varied according to the level progressed due to the basic test, and even emotions of panicked or nervous moments while coping with the situation can be recorded.

Situational feedback includes unexpected situations, environmental situations (noise, weather, people, etc.)

It can be a situational feedback that appears as an environment change, and can occur according to the level.

Audience feedback can be categorized into positive and negative feedback. Positive feedback can be, for example, the audience clapping, nodding their heads, taking notes, asking questions, laughing, or making eye contact.

Negative feedback can include actions such as lowering your head, using your phone, lying down, leaving the room, answering the phone, chatting, eating, making noise, or not sitting properly in your chair. there is.

The environmental feedback may be noise during presentation or feedback corresponding to a case where an exceptional situation occurs. The situation is based on the set data, and all digitized data can be averaged and applied. Audience feedback can show changes according to the set priority of feedback, the number of people in the audience, speech level, and environment.

Referring to FIG. 4 , according to an embodiment of the present disclosure, feedback information may be classified into four criteria.

The first feedback information may correspond to basic speech feedback.

For basic speech feedback, speed, pronunciation, intonation, stress, tone of voice, size, proper use of language, business language, humor, neologisms, slang, and interjections can be determined through voice recognition, and keywords delivered in comparison to the entire content is automatically recognized, and the obtained information can be used to catch the recorded video subdivided in units of seconds.

(1) Pronunciation and intonation: Accuracy ~ 100% (based on standard language)

(2) Speed, loudness, stress and tone of voice: Graphing the video in seconds (measured based on the data of each person's voice tested at the beginning)

(3) Use of business language, neologisms, slang, and interjections: Conformity percentage ~ 100% (The application automatically recognizes speech and subtitles the language, and inappropriate data from that time is filtered out and incorrect language usage is checked)

It is possible to check the recorded video by cutting it off in seconds, and the basic speech feedback at that moment is digitized and graphed and recorded in the application, and pressing the corresponding feedback leads to more detailed feedback.

In addition to being individually quantified or graphed, it is possible to train in a more professional manner when doing business as feedback by modifying the script or converting to a high-level language used in social economy, such as a similar language or lion idiom.

It is connected to the corresponding speech education program so that pronunciation or intonation training can be performed, or the corresponding link is connected and can be compared (dialect, foreign language, pronunciation, etc.).

The second feedback information may correspond to content delivery capability feedback.

Content delivery ability feedback can be generated by comparing keywords delivered through voice recognition based on the basic content of registered files, and based on data of similar languages even if the set keywords are properly spoken or not accurate keywords. It can be broken down into percentages.

Similar to the basic speech feedback, if more advanced content is desired, additional articles or news content to be known may be included.

The third feedback information may correspond to real-time reaction emotion feedback.

Real-time reaction emotion feedback can be generated based on audience feedback from the field and information obtained at the same time as the start, and can be generated based on changes in the electrocardiogram, iris, and eye.

This is also graphed, so you can remember the moment you were embarrassed and improve your ability to cope with each situation while contrasting the video and graph to see what kind of moment you were embarrassed and how you should respond. Changes in facial expression, electrocardiogram, and iris recognition can be graphed and output.

The fourth feedback information may correspond to speech action feedback.

Speech behavioral feedback may be generated based on a behavioral change in units of seconds based on video data and a habitual presentation habit. For example, it can be created based on habitual gestures made when standing with one leg, pointing with a finger, using presentation materials when presenting, or being embarrassed.

According to an embodiment of the present disclosure, a method of operating a server for improving a speech ability of a car user includes obtaining visual and auditory inputs in real time based on the speech of the user, and data corresponding to the inputs. It may include a process of generating a process, a process of generating feedback information corresponding to the user's speech in response to generating the data, and a process of providing a video or audio output according to the feedback information.

Preferably, the data includes audio data and video data, and the feedback information includes: first feedback information on basic speech ability of the user; second feedback information about the user's speech content delivery ability; third feedback information about emotion control in the user's speech; and fourth feedback information about the user's behavior during speech.

Preferably, the audio or video output is provided to the user's client device operatively connected to the server, and the client device outputs VR content.

Preferably, the method further comprises determining a priority based on a preset criterion among the first feedback information to the fourth feedback information, and the video or audio output is based on the priority can be provided.

Preferably, the method further comprises determining the feedback information based on comparing pre-stored reference information with data corresponding to the input, wherein the pre-stored reference information is information about a specific threshold , The process of determining the feedback information may further include a process of determining whether data corresponding to the input exceeds a specific threshold.

According to an embodiment of the present disclosure, an apparatus of a server for improving a user's speech ability includes a transceiver; and at least one control unit operatively connected to the transceiver, wherein the at least one control unit performs a process of acquiring visual and auditory inputs in real time based on the user's speech, and data corresponding to the inputs. It may include a process of generating a process, a process of generating feedback information corresponding to the user's speech in response to generating the data, and a process of providing a video or audio output according to the feedback information.

Preferably, the data includes audio data and video data, and the feedback information includes: first feedback information on basic speech ability of the user; second feedback information about the user's speech content delivery ability; third feedback information about emotion control in the user's speech; and fourth feedback information about the user's behavior during speech.

Preferably, the audio or video output is provided to the user's client device operatively connected to the server, and the client device outputs VR content.

Preferably, the at least one control unit is further configured to determine a priority based on a preset criterion among the first feedback information to the fourth feedback information, and the video or audio output is based on the priority. can be provided.

Preferably, the at least one control unit is further configured to determine the feedback information based on comparing pre-stored reference information with data corresponding to the input, wherein the pre-stored reference information includes information about a specific threshold. And, the at least one controller may be further configured to determine whether data corresponding to the input exceeds a specific threshold in order to determine the feedback information.

According to various embodiments, each component (eg, module or program) of the described components may include a singular object or a plurality of entities. According to various embodiments, one or more components or operations among the corresponding components described above may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (eg modules or programs) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to integration. According to various embodiments, operations performed by a module, program, or other component are executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations are executed in a different order, omitted, or Or one or more other actions may be added.

Claims (10)

A method of operating a server for improving a user's speech ability,
The method,
(A) obtaining visual and auditory inputs in real time based on the user's speech;
(B) a process of generating data corresponding to the input;
(C) generating feedback information corresponding to the user's speech in response to generating the data;
(D) providing a video or audio output according to the feedback information;
(E) determining a priority based on a preset criterion among first feedback information to fourth feedback information;
(F) determining the feedback information based on comparing previously stored reference information with data corresponding to the input; including,
The pre-stored reference information is information about a specific threshold,
The process of (F) above; silver
determining whether data corresponding to the input exceeds a specific threshold; contains
The data includes audio data and video data,
The feedback information,
first feedback information about the basic speech ability of the user;
second feedback information about the user's speech content delivery ability;
third feedback information about emotion control in the user's speech; and
fourth feedback information about the user's behavior during speech; and
The process of (A) above,
acquiring the visual and auditory inputs through an external input device operably connected to a server;
A process in which an external input device having a microphone or a camera first obtains a voice input or an image input; and
A process in which an external input device transmits data to a server or processes audio or video data and transmits the data to the server; including,
The visual and auditory inputs are
It is based on the user's speech and includes information related to sound and motion information generated in the course of the user's speech;
The process of (B) above; silver
(B-1) classifying an input according to a preset criterion to generate feedback on speech;
(B-2) the server acquiring an auditory input and generating voice data based on the acquired auditory input;
(B-3) generating voice data processed according to predetermined classification criteria including sentence structure, tone, intonation, and keywords;
(B-4) a process in which the server acquires a visual input and generates image data based on the visual input; and
(B-5) generating processed image data according to predetermined classification criteria including gestures, postures, facial expressions, and complexion; contains
Step (B-1) above; Is
determining a numerical value of an area corresponding to an item part of the determined speech input in the audio and video data determined based on the speech in the server;
When the item of the speech input is the degree of change in facial expression, the server determines the degree of change in the measured facial expression based on the image data, and the change in facial expression is digitized or leveled based on a preset table. How to characterize.
delete The method of claim 1,
The audio or video output,
Provided to the user's client device operably connected with the server;
Wherein the client device outputs VR content.
delete delete delete delete delete delete delete