KR20210074649A - Speech Recognition Method Determining the Subject of Response using Multi-Modal Analysis in Natural Language Sentences - Google Patents

Abstract

The present invention relates to a speech recognition method for determining whether or not to respond using text and sound information extracted from a natural utterance without inputting a call language. As a two-pass cascade type using an intention classifier and a topic classifier, the speech recognition method for determining whether or not to respond in a natural language sentence of the present invention is capable of configuring a unique language model for each topic through the topic classifier, and at the same time, is capable of more accurately determining a response and a non-response by using not only the text information but also the sound information in order to overcome a problem of difficulty in classifying utterances as a declarative sentence or an interrogative sentence in accordance with an intonation in the intention classifier.

Description

Speech Recognition Method Determining the Subject of Response using Multi-Modal Analysis in Natural Language Sentences

The present invention relates to an interactive voice recognition method, and more particularly, to a voice recognition method for determining whether to respond or not by using text and sound information extracted from natural speech without inputting a call word.

Speech Recognition is a process in which a computer interprets the speech language spoken by a person and converts the contents into text data. It is a technology distinct from speaker recognition used for authentication purposes by comparing it with a pre-recorded voice pattern of a specific person. It is widely used to control devices with voice and search for information in intelligent machines such as telematics and robots, where information communication and the automobile industry are fused. In order to broaden the range of users, the voices uttered by various speakers are statistically modeled to construct an acoustic model and a pronunciation model, and a language model is constructed through corpus collection.

In order for human-machine conversation to be possible using 'speech', the input/output interface of an intelligent machine must be voice, and such a machine is also called a voice recognition device. As the accuracy of the voice recognition rate of voice recognition devices increases, the application services of voice recognition technology have also been expanded, and from the assistant-type voice recognition system of the smartphone to the speaker-type artificial intelligence assistant and the input technology of the Internet of Things (Internet of Things). is expanding

In a conventional voice recognition device, a voice recognition mode is activated using a calling word such as “Siri” or “OK Google”, or a predetermined calling word is activated by expanding the calling word range. When the call word is determined as described above, the device cannot be controlled by recognizing the spoken word during a natural conversation, so the versatility of the device is lowered.

Korean Patent Application Laid-Open No. 2014-0073889 relates to 'a call word buffering and filling interface for interactive voice recognition', and the invention relates to a user's natural conversation even if the user does not repeatedly input the call word every time the user inputs a voice. Disclosed is a technology for processing by performing speech recognition of command phrases from type speech input. However, the invention is called natural language speech recognition technology because the user goes through the process of directly inputting the calling word with his or her voice, recognizing the voice waveform input with the calling word, and re-recognizing the waveform in the answer when the device asks a question. hard to see Therefore, it is required to develop a technology for a system that allows the voice recognizer to determine whether to respond without using a call word.

Republic of Korea Patent Publication No. 2014-0073889

An object of the present invention is to provide a voice recognition method that analyzes a user's utterance and sound information through voice identification without a call word to determine what a voice recognition device should respond to and what it does not.

The present invention provides a voice recognition method for determining whether to respond in a natural language sentence, the method comprising the steps of: converting a voice uttered by a user into a digital voice signal in a voice input device; extracting sound information from the converted digital voice signal using OpenSmile Toolkit, which is a toolkit for extracting sound information, and recognizing text by word using an embedded voice recognizer; sending the extracted sound information and the recognized word to an intention classifier, and classifying it into a request sentence, a question sentence, and a declarative sentence; determining non-response for the voice classified as a declarative sentence in the intention classifier, and sending the request sentence and the interrogative sentence to a topic classifier; and classifying the request and interrogative sentences into topics and others of a predetermined class in a topic classifier, determining others as non-responsive, and determining the topic of the predetermined class as a response target, wherein the topic classifier and the intent classifier provides a speech recognition method for determining whether to respond in natural language sentences using Linear Bag of Words Classifier, a sentence classification algorithm of Fasttext, a natural language processing toolkit.

In another aspect of the present invention, the intention classifier includes a sentence database for recognizing words including pitch and formant information as sound information, and the sentence database converts the input sentence into a request sentence and a plain statement. and request sentences including pitch and formant information for classifying into interrogative sentences, and sentence data for each declarative sentence and interrogative sentence, wherein the request sentence, declarative sentence, and interrogative sentence data including the pitch and formant information are stored for a predetermined period Provided is a voice recognition method for determining whether to respond in a natural language sentence, which is updated and stored in units.

The present invention also further includes, in the sentence database, an answer by topic according to the determining step as declarative sentence data corresponding to a request and a question sentence including a topic of the predetermined class, wherein the determining step includes: It provides a voice recognition method for determining whether to respond in a natural language sentence, further comprising the step of uttering the corresponding plaintext sentence with a speaker.

In the present invention, the topics of the predetermined class are e-mail, house control, weather, and schedule, and the topic classifier is a new topic in the topic of the predetermined class. It provides a voice recognition method for determining whether to respond in a natural language sentence, further comprising a topic adding unit for adding a.

The present invention also provides that the topic classifier includes a word database, the word database includes embedding data for each topic, and the words and similar word data for each topic are updated and stored in units of a predetermined period. A voice recognition method for determining whether to respond in a sentence is provided.

The present invention also provides a code part programmed to convert the voice uttered by the user into a digital voice signal in the voice input device; a code part programmed to extract sound information from the converted digital voice signal using OpenSmile Toolkit, which is a sound information extraction toolkit, and to recognize text by word using an embedded voice recognizer; a code part programmed to send the extracted sound information and the recognized word to an intention classifier and classify it into a request sentence, a question sentence, and a declarative sentence; a code part programmed to determine non-response for the voice classified as a declarative sentence by the intention classifier, and to send the request sentence and the interrogative sentence to the topic classifier; and a code part programmed to classify the request and interrogative sentences into topics and others of a predetermined class in a topic classifier, determine others as non-response, and determine a topic of the predetermined class as a response target, the topic classifier and The intention classifier provides a computer-readable storage medium storing a speech recognition computer program programmed to determine whether to respond in a natural language sentence using a Linear Bag of Words Classifier, a sentence classification algorithm of the natural language processing toolkit Fasttext.

Since the speech recognition method for determining whether to respond in a natural language sentence of the present invention is a two-pass cascade type using an intention classifier and a topic classifier, it is possible to construct a unique language model for each topic through the topic classifier, and at the same time, the utterance in the intention classifier is possible. In order to overcome the problem that it is difficult to classify whether it is a declarative sentence or an interrogative sentence depending on intonation, it is possible to more accurately determine response and non-response by using sound information as well as text information.

1 shows an exemplary structure of a speech recognition method for diagnosing answered and non-responsive sentences, which sequentially passes through a speech recognizer and a sound information extractor, an intention classifier, and a topic classifier according to the present invention.
2 is a conceptual flowchart of a method for discriminating response and non-corresponding sentences using an intention classifier and a topic classifier according to the present invention.

Various aspects are disclosed with reference to the drawings. In the following description, for purposes of explanation, numerous specific details are set forth to provide a thorough understanding of one or more aspects. It will be appreciated, however, that these aspects may be practiced without each specific detail. The following description and the accompanying drawings set forth in detail certain illustrative aspects of one or more aspects. These aspects are illustrative, however, and some of the various methods may be employed in principle in various aspects and the description is intended to include all such aspects and their equivalents.

Various aspects and features will be presented by a system that may include a number of devices, modules, and the like. It should also be understood and appreciated that the various systems may include additional devices, parts, components, etc. and/or may not include all of the devices, parts, components, etc. discussed with respect to the drawings.

As used herein, "embodiment", "example", "aspect", "exemplary", etc. should not be construed as an advantage or advantage over any aspect or design described herein. The terms 'system', 'server' and the like used below generally refer to a computer-related entity, and may refer to, for example, hardware, a combination of hardware and software, or software.

In addition, the term “or” is intended to mean an inclusive “or” rather than an exclusive “or.” That is, unless otherwise specified or clear from context, "X employs A or B" is intended to mean one of the natural implicit substitutions. That is, X employs A; X employs B; Or when X employs both A and B, "X employs A or B" can be applied to any of the above cases. It is also to be understood that the term “and/or” as used herein refers to and includes all possible combinations of one or more of the listed related items.

Also, the terms "comprises" and/or "comprising" mean that the feature, step, action, module, and/or component is present, but one or more other features, steps, actions, modules, configurations. It is to be understood that this does not exclude the presence or addition of elements, and/or groups thereof. In addition, although terms such as first and second may be used in this specification to describe various components, these components are not limited by these terms. That is, these terms are only used to distinguish between two or more components, and should not be construed as meaning an order or priority. Also, unless otherwise specified or unless the context is clear as to designating a singular form, the singular in the specification and claims should generally be construed to mean "one or more." Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.

Unlike a conventional voice recognition interface using a call word, the robot voice recognition technology without a call word can provide a more natural conversational interface between a human and a robot by judging the content of the user's utterance. In addition, if the existing system performed (call word recognition -> voice recognition -> task), the present invention can immediately reflect the user's request by performing (voice recognition -> task). The present invention will be described in detail below with reference to the drawings.

1 shows an exemplary structure of a speech recognition method for responsive and non-responsive sentence discrimination that sequentially passes through a speech recognizer and a sound information extractor according to the present invention, an intention classifier and a topic classifier, and FIG. 2 is an intention classifier and a topic according to the present invention. A conceptual flowchart of a method for discriminating response and non-corresponding sentences using a classifier is shown. In one embodiment of the present invention, a voice recognition method includes the steps of: converting a voice uttered by a user into a digital voice signal in a voice input device; extracting sound information from the converted digital voice signal using OpenSmile Toolkit, which is a toolkit for extracting sound information, and recognizing text by word using an embedded voice recognizer; sending the extracted sound information and the recognized word to an intention classifier and classifying it into a request sentence, a question sentence, and a declarative sentence; determining non-response for the voice classified as a declarative sentence by the intention classifier, and sending the request sentence and the interrogative sentence to the topic classifier; and classifying the request and interrogative sentences into topics and others of a predetermined class by a topic classifier, determining others as non-responsive, and determining the topic of the predetermined class as a response target.

In an embodiment of the present invention, the topic classifier and the intention classifier may use a Linear Bag of Words Classifier, a sentence classification algorithm of the natural language processing toolkit Fasttext. Through the above configuration, the present invention can improve the accuracy of determining whether to respond in a natural language sentence compared to the prior art. In one embodiment of the present invention, the intention classifier includes a sentence database for word recognition including pitch and formant information as sound information. In one embodiment of the present invention, it is possible to reflect information such as intonation, stress, and tempo due to the sound information. In addition, in an embodiment of the present invention, the sentence database includes sentence data for each request, declarative sentence, and interrogative sentence including pitch and formant information for classifying the input sentence into a request sentence, a declarative sentence, and a question sentence, and the pitch It is possible to update and store the text data for each request, declarative text, and interrogative text including the formant information and the formant information in units of a predetermined period.

In addition, in one embodiment of the present invention, the sentence database further includes, as declarative sentence data corresponding to a request sentence and a question sentence including a topic of the predetermined class, an answer for each topic according to the determining step, and the judgment The doing may further include uttering the corresponding declarative sentence through a speaker.

In one embodiment of the present invention, the topics of the predetermined class are e-mail, house control, weather, and schedule, and the topic classifier is the topic of the predetermined class. It may further include a topic adding unit for adding a new topic. Through these functions, topic expansion can be implemented naturally. In one embodiment of the present invention, the topic classifier includes a word database, the word database includes embedding data for each topic, and the word and similar word data for each topic can be updated and stored in units of a predetermined period. have.

The voice recognition method of the present invention may be implemented in the form of a computer-readable storage medium, and the storage medium may store a voice recognition computer program programmed to determine whether to respond in a natural language sentence. The storage medium may include a code part programmed to convert the voice uttered by the user into a digital voice signal in the voice input device; a code part programmed to extract sound information using the LIBROSA python library, which is a sound information extraction toolkit, from the converted digital voice signal, and to recognize text by word using an embedded voice recognizer; a code part programmed to send the extracted sound information and the recognized word to an intention classifier and classify it into a request sentence, a question sentence, and a declarative sentence; a code part programmed to determine non-response for the voice classified as a declarative sentence by the intention classifier, and to send the request sentence and the interrogative sentence to the topic classifier; and a code part programmed to classify the request and interrogative sentences into topics and others of a predetermined class in a topic classifier, determine others as non-responsive, and determine topics of the predetermined class as response targets.

In an embodiment of the present invention, the topic classifier and the intention classifier use a Linear Bag of Words Classifier, a sentence classification algorithm of the natural language processing toolkit Fasttext.

As described above, the present invention relates to a speech recognition method for determining whether to respond in a natural language sentence. This invention can be applied to, for example, immediately reflecting the user's needs while driving in the automobile field, and can be applied to increase user interface convenience in the Internet of Things environment in the home automation field, and artificial intelligence assistant application can be used to increase the convenience of the user interface of smart speakers or robots.

Various embodiments described herein may be implemented in a computer-readable medium using, for example, software, hardware, or a combination thereof.

According to the hardware implementation, the embodiments described herein are ASICs (application specific integrated circuits), DSPs (digital signal processors), DSPDs (digital signal processing devices), PLDs (programmable logic devices), FPGAs (field programmable gate arrays, It may be implemented using at least one of processors, controllers, micro-controllers, microprocessors, and other electrical units for performing functions. The described embodiments may be implemented with the management server and/or the system itself.

According to the software implementation, embodiments such as the procedures and functions described in this specification may be implemented as separate software modules. Each of the software modules may perform one or more functions and operations described herein. The software code may be implemented as a software application written in a suitable programming language. The software code may be stored in the management server and/or database, and executed by the app.

Meanwhile, various embodiments presented herein may be implemented as methods, apparatus, or articles of manufacture using standard programming and/or engineering techniques. The term “article of manufacture” includes a computer program, carrier, or media accessible from any computer-readable device. For example, computer-readable media include magnetic storage devices (eg, hard disks, floppy disks, magnetic strips, etc.), optical disks (eg, CDs, DVDs, etc.), smart cards, and flash memory devices. (eg, EEPROM, card, stick, key drive, etc.). Also, various storage media presented herein include one or more devices and/or other machine-readable media for storing information. The term “machine-readable medium” includes, but is not limited to, wireless channels and various other media capable of storing, holding, and/or carrying instruction(s) and/or data.

The description of the presented embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the scope of the invention. Thus, the present invention is not to be limited to the embodiments presented herein, but is to be construed in the widest scope consistent with the principles and novel features presented herein.

Claims (6)

As a speech recognition method for determining whether to respond in a natural language sentence:
The method includes the steps of: converting a voice uttered by a user into a digital voice signal in a voice input device;
extracting sound information from the converted digital voice signal using OpenSmile Toolkit, which is a sound information extraction toolkit, and recognizing text by word using an embedded voice recognizer;
sending the extracted sound information and the recognized word to an intention classifier and classifying it into a request sentence, a question sentence, and a declarative sentence;
determining non-response for the voice classified as a declarative sentence by the intention classifier, and sending the request sentence and the interrogative sentence to the topic classifier; and
Classifying the request and interrogative sentences into topics and others of a predetermined class in a topic classifier, determining others as non-response, and determining the topic of the predetermined class as a response target,
The topic classifier and the intention classifier use Linear Bag of Words Classifier, a sentence classification algorithm of the natural language processing toolkit Fasttext,
A speech recognition method for determining whether to respond in a natural language sentence.
The method of claim 1,
The intention classifier includes a sentence database for word recognition including pitch and formant information as sound information,
The sentence database includes sentence data for each request, declarative sentence, and interrogative sentence including pitch and formant information for classifying the input sentence into a request sentence, a declarative sentence, and a question sentence, and the request including the pitch and formant information Sentence data for each sentence, declarative sentence, and interrogative sentence is updated and stored in units of a predetermined period
A speech recognition method for determining whether to respond in a natural language sentence.
3. The method of claim 2,
The sentence database further includes an answer by topic according to the step of determining as declarative sentence data corresponding to a request sentence and a question sentence including a topic of the predetermined class,
The determining step further comprises the step of uttering the corresponding declarative sentence through a speaker,
A speech recognition method for determining whether to respond in a natural language sentence.
The method of claim 1,
The topics of the predetermined class are email, house control, weather, and schedule,
The topic classifier further comprises a topic adding unit for adding a new topic to the topic of the predetermined class,
A speech recognition method for determining whether to respond in a natural language sentence.
5. The method of claim 4,
The topic classifier includes a word database,
The word database includes embedding data for each topic, and the word and similar word data for each topic are updated and stored in units of a predetermined period.
A speech recognition method for determining whether to respond in a natural language sentence.
a code part programmed to convert the voice uttered by the user into a digital voice signal in the voice input device;
a code part programmed to extract sound information from the converted digital voice signal using the LIBROSA python library, which is a sound information extraction toolkit, and to recognize text by word using an embedded voice recognizer;
a code part programmed to send the extracted sound information and the recognized word to an intention classifier and classify it into a request sentence, a question sentence, and a declarative sentence;
a code part programmed to determine non-response for the voice classified as a declarative sentence in the intention classifier, and to send the request sentence and the interrogative sentence to the topic classifier; and
Classifying the request and interrogative sentences into topics and others of a predetermined class in a topic classifier and determining others as non-responsive, including a code part programmed to determine the topic of the predetermined class as a response target,
The topic classifier and the intention classifier use Linear Bag of Words Classifier, a sentence classification algorithm of the natural language processing toolkit Fasttext,
A computer-readable storage medium storing a speech recognition computer program programmed to determine whether to respond in a natural language sentence.

Images