KR20210019924A - System and method for modifying voice recognition result - Google Patents

Abstract

Provided are a system and method for correcting a speech recognition result. The method for correcting a speech recognition result provided from a device by a server comprises the operations of: receiving a text outputted from an automatic speech recognition (ASR) model of the device from the device; identifying at least one domain related to the received text; selecting at least one text correction model corresponding to the identified domain among a plurality of text correction models included in the server; and correcting the received text using the selected text correction model.

Description

System and method for modifying speech recognition results {SYSTEM AND METHOD FOR MODIFYING VOICE RECOGNITION RESULT}

The present disclosure relates to a system and method for modifying a speech recognition result, and more particularly, to a system and method for modifying a speech recognition result by interworking with a device and a server.

Automatic Speech Recognition is a technology that receives a human voice, recognizes it, and converts it into text. Voice recognition is being used in various electronic devices such as smartphones, air conditioners, refrigerators, and AI speakers. First, the device receives a human voice as an input, and uses a voice recognition model that is already trained inside the device to recognize the input voice and convert it into text. You will have the converted text as the final output. Recently, a deep neural network (DNN) algorithm has been used in various machine learning fields, and performance has been improved. In the field of speech recognition, performance has been greatly improved using neural networks, and recently, an Automatic Speech Recognition Model for speech recognition has been studied. As artificial intelligence systems are used, the recognition rate improves and users' tastes can be more accurately understood, and the existing rule-based smart systems are gradually being replaced by deep learning-based artificial intelligence systems.

An embodiment of the present disclosure provides a system and method capable of providing an output value of an ASR (Automatic Speech Recognition) model of a device to a server, and modifying an output value of an ASR model using an artificial intelligence model of the server. have.

In addition, an embodiment of the present disclosure may provide a system and method capable of correcting a speech recognition result by using a text correction model corresponding to a domain related to an output value of an ASR model of a device.

In addition, an embodiment of the present disclosure may provide a system and method for enabling a server to effectively apply text received from a device to text modification models related to a plurality of domains.

In addition, an embodiment of the present disclosure may provide a system and method for enabling a server to effectively identify a domain related to text using a plurality of domain identification modules related to a plurality of domains.

As a technical means for achieving the above-described technical problem, an aspect of the present disclosure includes an operation of receiving, from the device, a text output from an Automatic Speech Recognition (ASR) model of the device; Identifying at least one domain related to the received text; Selecting at least one text correction model corresponding to the identified at least one domain from among a plurality of text correction models included in the server; Modifying the received text using the selected at least one text modification model; And providing the modified text to the device. Including, wherein the text correction model is an artificial intelligence model that receives the received text for speech recognition and outputs the corrected text, wherein the server provides a method of modifying the speech recognition result provided from the device. I can.

In addition, another aspect of the present disclosure, a communication interface; A storage unit for storing a program including one or more instructions; And a processor that executes one or more instructions of the program stored in the storage unit. Including, wherein the processor receives the text output from the ASR (Automatic Speech Recognition) model of the device from the device, identifies at least one domain related to the received text, and a plurality of Among text modification models, select at least one text modification model corresponding to the identified at least one domain, modify the received text using the selected at least one text modification model, and select the modified text. Provided to the device, wherein the text correction model is an artificial intelligence model that receives the received text and outputs the corrected text for speech recognition, providing a server for modifying the speech recognition result provided from the device. I can.

1 is a schematic diagram of a speech recognition system according to an embodiment of the present disclosure.
2 is a schematic diagram of a speech recognition system including text correction models related to a plurality of domains according to an embodiment of the present disclosure.
3 is a flowchart of a method for acquiring a modified text by recognizing a voice input by a device and a server in a voice recognition system according to an embodiment of the present disclosure.
4 is a diagram illustrating an example in which a server according to an embodiment of the present disclosure identifies a domain related to text and selects a text correction model of a domain related to text.
5 is a diagram illustrating an example in which a device identifies a domain related to text and a server selects a text modification model of a domain related to text according to an embodiment of the present disclosure.
6 is a diagram illustrating an example in which a server and a device according to an embodiment of the present disclosure identify domains related to text, and the server selects a text modification model of a domain related to text.
7 is a flowchart of a method for selecting a domain related to text by using a domain reliability obtained from a device and a domain reliability obtained from a server according to an embodiment of the present disclosure.
8 is a diagram illustrating an example in which a server according to an embodiment of the present disclosure selects a text correction model by using a domain identification module selected from among a plurality of domain identification modules in the server.
9 is a flowchart of a method for selecting a domain for text correction by using a domain identification module selected from among a plurality of domain identification modules by a server according to an embodiment of the present disclosure.
10 is a diagram illustrating an example of a first domain identification module, a second domain identification module, and a text correction model related to hierarchically classified domains according to an embodiment of the present disclosure.
11 is a diagram illustrating an example in which a server corrects text using a plurality of text correction models according to an embodiment of the present disclosure.
12 is a flowchart of a method of accumulating and calculating domain reliability for texts in a plurality of sections by a server according to an embodiment of the present disclosure.
13 is a diagram illustrating an example in which a server obtains domain reliability for a text stream accumulated in word units according to an embodiment of the present disclosure.
14 is a flowchart illustrating a method of dividing text into a plurality of sections by a server and selecting a domain for each text of each section according to an embodiment of the present disclosure.
FIG. 15 is a diagram illustrating an example in which a server compares the domain reliability of text for each of a plurality of domains, and selects and corrects a text correction model for each text in each section.
16 is a diagram illustrating an example in which a server according to an embodiment of the present disclosure modifies text received from a device by using text modified from a plurality of text correction models.
17 is a block diagram of a server according to an embodiment of the present disclosure.
18 is a block diagram of a device according to an embodiment of the present disclosure.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art may easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In the drawings, parts irrelevant to the description are omitted in order to clearly describe the present disclosure, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a part is said to be "connected" to another part, this includes not only "directly connected" but also "electrically connected" with another element interposed therebetween. . In addition, when a part "includes" a certain component, it means that other components may be further included rather than excluding other components unless specifically stated to the contrary.

Hereinafter, the present disclosure will be described in detail with reference to the accompanying drawings.

1 is a schematic diagram of a speech recognition system according to an embodiment of the present disclosure.

Referring to FIG. 1, a voice recognition system according to an embodiment of the present disclosure includes a device 1000 and a server 2000.

The device 1000 may include an Automatic Speech Recognition (ASR) model, and the server 2000 may include a text correction model. The device 1000 may recognize a user's voice input and output text using the ASR model, and the server 2000 may modify the text generated by the device 1000.

The ASR model is a speech recognition model that recognizes speech using an integrated neural network, and can output text from a user's speech input. The ASR model may be, for example, an artificial intelligence model including an acoustic model, a pronunciation dictionary, and a language model. Alternatively, the ASR model may be, for example, an end-to-end speech recognition model having a structure including an integrated neural network without separately including an acoustic model, a pronunciation dictionary, and a language model. The end-to-end ASR model uses an integrated neural network, so after recognizing a phoneme from a speech, it can convert speech into text without the process of converting the phoneme into text. The text may include at least one character. Letters refer to symbols used to express human language in a visible form and write it down. For example, characters may include Korean characters, alphabets, Chinese characters, numbers, pronunciation marks, punctuation marks, and other symbols. Also, for example, the text may include a character string. String refers to the sequence of characters. For example, the text may include at least one character element. Grapheme is the smallest unit of sound, composed of at least one character. For example, in the case of the alphabet notation system, one character may be a character element, and a character string may mean an arrangement of character elements. For example, the text may include morphemes or words. A morpheme is the smallest unit that has a meaning composed of at least one letter. A word is a basic unit of language that can be used independently or represents a grammatical function, composed of at least one morpheme.

The device 1000 may receive a user's voice input, recognize the received voice input using an ASR model, and provide text, which is an output value of the ASR model, to the server 2000. Also, the server 2000 may receive text, which is an output value of the ASR model, from the device 1000 and may modify the received text. The server 2000 may identify the degree to which text, which is an output value of the ASR model, is related to a specific domain registered in the server 2000, and correct the text using the text correction model of the identified domain. Also, the server 2000 may provide the modified text to the device 1000.

The text correction model is an artificial intelligence model that has been trained to correct at least a portion of text resulting from speech recognition, and may include, for example, a sequence-to-sequence mapper. The text correction model may be an artificial intelligence model trained using text output from the ASR model and a preset ground truth text. The text correction model may be an artificial intelligence model trained for each domain. For example, the text correction model of the first domain may be trained using the text that is an output value of the ASR model and the correct answer text specialized in the first domain. In addition, for example, the text correction model of the second domain may be trained using a text that is an output value of the ASR model and a correct answer text specialized for the second domain.

Further, the text correction model may be an artificial intelligence model trained using texts output from various types of ASR models and preset correct answer texts. In this case, since the text correction model is trained with text output from several types of ASR models, the correct output value (for example, regardless of what kind of ASR model the text input to the text correction model is) Revised text) can be provided.

For example, the revised text may include at least one of revised characters, revised letters, revised morphemes, and revised words. For example, if the text output from the ASR model contains a typo, the corrected text may include a character corrected from the typo. In addition, for example, when the text output from the ASR model includes a word with an incorrect meaning inappropriate to the context, the corrected text may include a word with a correct meaning replacing the word with the wrong meaning. Further, for example, a specific word in the text output from the ASR model may be replaced with a similar word, thereby generating a modified text.

The device 1000 includes a smartphone, a tablet PC, a PC, a smart TV, a mobile phone, a personal digital assistant (PDA), a laptop, a media player, a micro server, a global positioning system (GPS) device, an e-book terminal, a digital broadcasting terminal, Navigation, kiosk, MP3 player, digital camera, home appliances, and other mobile or non-mobile computing devices may be, but are not limited thereto. In addition, the device 1000 may be a wearable device such as a watch, glasses, hair band, and ring having a communication function and a data processing function. However, the present invention is not limited thereto, and the device 1000 may include all kinds of devices capable of transmitting and receiving data through a network with the server 2000 for voice recognition.

The network 200 includes a local area network (LAN), a wide area network (WAN), a value added network (VAN), a mobile radio communication network, a satellite communication network, and a mutual It includes a combination, and is a data communication network in a comprehensive meaning that enables each network member shown in FIG. 1 to communicate with each other smoothly, and includes a wired Internet, a wireless Internet, and a mobile wireless communication network.

2 is a schematic diagram of a speech recognition system including text correction models related to a plurality of domains according to an embodiment of the present disclosure.

Referring to FIG. 2, the device 1000 may include an ASR model, and the server 2000 may include text correction models for correcting text output from the ASR model. For example, the server 2000 may include a first text correction model corresponding to the first domain and a second text correction model corresponding to the second domain.

The device 1000 may obtain a feature vector by extracting a feature from the voice input, and may input the obtained feature vector into the ASR model. The device 1000 may provide an output value output from the ASR model to the server 2000. The output value output from the ASR model can include various types of text. For example, the device 1000 may provide text in sentence units to the server 2000 or a text stream to the server 2000, but is not limited thereto.

The server 2000 may receive text from the device 1000 and select a domain related to the received text. The domain represents a field related to the input voice, and may be preset according to, for example, the meaning of the input voice and the properties of the input voice. The domain may be classified according to, for example, a service related to an input voice. In addition, the text correction model may be trained for each domain. In this case, the text correction model trained for each domain may be a model trained using an input text related to a corresponding domain and a correct answer text corresponding thereto. The server 2000 may select at least one of a plurality of preset domains, and select at least one of text correction models corresponding to the selected domain. Also, the server 2000 may obtain the corrected text by inputting the text received from the device 1000 into the text correction model of the selected domain. The server 2000 may provide the modified text to the device 1000.

Meanwhile, the server 2000 may provide various types of voice assistant services to the device 1000 by using the modified text. The voice assistant service may be a service that provides a conversation with a user. In the voice assistant service, a response message can be provided to the user as if a person had a direct conversation with the user in consideration of the situation of the user and the device. In addition, in the voice assistant service, information required by the user, such as a personal assistant of the user, may be appropriately generated and provided to the user. The voice assistant service includes, for example, a broadcasting service, a content sharing service, a content providing service, a power management service, a game providing service, a chat service, a document writing service, a search service, a call service, a photo shooting service, a transportation recommendation service, and In connection with various services such as a video playback service, information or functions required by the user may be provided to the user.

3 is a flowchart of a method for obtaining a modified text by recognizing a voice input by a device and a server in a voice recognition system according to an embodiment of the present disclosure.

The device 1000 may execute the operation of the device 1000 in FIG. 3 by executing instructions stored in the memory of the device 1000. For example, the device 1000 includes a voice recognition evaluation module 1430, a domain identification module 1440, an NLU determination module 1450, a domain registration module 1460, an ASR model 1410, and an NLU of FIG. 18 to be described later. The operation of the device 1000 in FIG. 3 may be performed by executing at least one of the models 1420. However, the present invention is not limited thereto, and the device 1000 may execute another program stored in the memory to perform a predetermined operation of the device 1000.

In addition, the server 2000 can execute the operation of the server 2000 in FIG. 3 by executing instructions stored in the storage unit of the server 2000. For example, the server 2000 in FIG. 3 by executing at least one of the domain management module 2310, the speech analysis management module 2340, the text modification module 2320, and the NLU module 2330 of FIG. 17 to be described later. The operation of the server 2000 of may be performed. However, the present invention is not limited thereto, and the server 2000 may execute another program stored in the storage to perform a predetermined operation of the server 2000.

In S300, the device 1000 may obtain a feature vector from the voice signal. The device 1000 may receive a user's voice input (eg, speech) through a microphone, and may generate a feature vector representing characteristics of the voice signal by using the voice signal obtained through the microphone. When noise is included in the voice signal, the device 1000 may remove noise in the voice signal and obtain a feature vector from the noise-removed voice signal. Also, for example, the device 1000 may extract a feature vector representing a characteristic of the speech signal from the speech signal. For example, the device 1000 may receive data representing a feature vector of a voice signal from an external device (not shown).

In S305, the device 1000 may obtain text from the feature vector using the ASR model. The device 1000 may input a feature vector into the ASR model in the device 1000 in order to recognize the user's voice. When a plurality of ASR models are included in the device 1000, the device 1000 may select one of the plurality of ASR models and convert the feature vector into a format suitable for the selected ASR model. The ASR model of the device 1000 may be, for example, an artificial intelligence model including an acoustic model, a pronunciation dictionary, and a language model. Alternatively, the ASR model of the device 1000 may be, for example, an end-to-end speech recognition model having a structure including an integrated neural network without separately including an acoustic model, a pronunciation dictionary, and a language model.

In S310, the device 1000 may acquire the reliability of the text. The reliability of the text may be a numerical value indicating the degree to which the text output from the ASR model matches the input voice, and may include, for example, a confidence score, but is not limited thereto. Also, the reliability of the text may be related to the probability that the text will match the input speech. For example, the reliability of the text is calculated based on at least one of the likelihood of a plurality of estimated texts output from the ASR model of the device 1000 and posterior probabilities in which at least one character in the text is replaced with another character. Can be. For example, the device 1000 may calculate the reliability based on the likelihood output as a result of Viterbi decoding. Alternatively, for example, the device 1000 may calculate the reliability based on posterior probabilities output from the softmax layer in the end-to-end ASR model. Alternatively, for example, the device 1000 determines a plurality of estimated texts estimated in the speech recognition process of the ASR model of the device 1000, and based on the correlation of characters in the plurality of estimated texts, the reliability of the text Can be calculated. Also, for example, the device 1000 may acquire the reliability of the text using the speech recognition evaluation module 1430 of FIG. 18.

In S315, the device 1000 may determine whether to transmit the text to the server 2000. The device 1000 may determine whether to transmit the text to the server 2000 by comparing the reliability of the text with a preset threshold. The device 1000 may determine not to transmit the text to the server 2000 when the reliability of the text is greater than or equal to a preset threshold. Also, if the reliability of the text is less than a preset threshold, the device 1000 may determine to transmit the text to the server 2000.

Also, the device 1000 may determine whether to transmit the text to the server 2000 based on at least one text with high reliability among a plurality of estimated texts estimated during the speech recognition process of the ASR model. For example, the plurality of estimated texts estimated in the speech recognition process of the ASR model include a first estimated text with high reliability and a second estimated text with high reliability, and the reliability of the first estimated text and the reliability of the second estimated text When the difference between is less than or equal to a predetermined threshold, the device 1000 may determine to transmit the text to the server 2000. In addition, for example, the plurality of estimated texts estimated in the speech recognition process of the ASR model include a first estimated text with high reliability and a second estimated text with high reliability, and the reliability of the first estimated text and the second estimated text When the difference in reliability of is greater than a predetermined threshold, the device 1000 may determine not to transmit the text to the server 2000.

When it is determined in operation S315 to transmit the text to the server 2000, the device 1000 may request the server 2000 to modify the text in operation S320.

The device 1000 may request the modified text from the server 2000 while transmitting the text to the server 2000. In this case, for example, the device 1000 may transmit the type of the ASR model in the device 1000 and the identification value of the ASR model to the server 2000 while requesting the text from the server 2000, but this is not limited thereto. Does not.

Also, for example, the device 1000 may provide domain information related to the text output from the ASR model of the device 1000 to the server 2000 while requesting the modified text from the server 2000. The domain information is information for identifying a domain, and may include, for example, a name of a domain and an identifier of a domain, but is not limited thereto. The device 1000 may identify a domain related to text by using the domain identification module 1440 in the device 1000. For example, the device 1000 may identify a domain related to the text based on the domain reliability of the text output from the ASR model of the device 1000. The domain reliability may be a numerical value indicating to what extent at least part of the text is related to a specific domain. For example, the device 1000 may calculate a confidence score indicating to what extent a text output from the ASR model is related to a domain registered in advance in the device 1000. Also, the device 1000 may identify a domain related to text based on the calculated domain reliability. The device 1000 may identify a domain related to the text based on a rule or obtain a domain reliability related to the text using an artificial intelligence model trained for domain identification. Also, for example, the artificial intelligence model for domain identification may be part of the NLU model or may be a model separate from the NLU model.

When it is determined not to transmit the text to the server 2000 in operation S315, the device 1000 may provide a voice assistant service by using the text output from the ASR model. For example, if the reliability of the text output from the ASR model is greater than or equal to a preset threshold, the device 1000 may perform operations for the voice assistant service by using the text output from the ASR model. In addition, for example, the plurality of estimated texts estimated in the speech recognition process of the ASR model include a first estimated text having the highest reliability and a second estimated text having the second highest reliability, and the reliability of the first estimated text and When the difference in reliability of the second estimated text is greater than a predetermined threshold, the device 1000 may provide the voice assistant service using the first estimated text having the highest reliability.

For example, the device 1000 may display text output from the ASR model on the screen. For example, the device 1000 may perform an operation for dialogue with the user based on the text output from the ASR model. Also, for example, the device 1000 may perform an operation based on the text output from the ASR model. Through dialogue with the user, broadcast service, content sharing service, content provision service, power management service, game provision service, chat service, document writing service, search service, call service, photo shooting service, transportation recommendation service, and Various services such as video playback service can be provided.

In operation S325, the server 2000 may identify a domain for text correction. When the server 2000 receives domain information from the device 1000, the server 2000 may identify a domain for text correction from the domain information. Alternatively, when the server 2000 does not receive domain information from the device 1000, the server 2000 uses the domain identification module 2312 in the server 2000 to write the text received from the device 1000. Related domains can be identified. For example, in this case, the server 2000 may identify a domain related to the text based on the domain reliability of the text received from the device 1000. For example, the server 2000 may calculate a confidence score indicating how relevant the text received from the device 1000 is to a domain registered in advance for text correction. Also, the server 2000 may identify a domain related to the text received from the device 1000 based on the domain reliability calculated for the domain registered in advance. The server 1000 may identify a domain related to the text based on a rule or obtain a domain reliability related to the text using an artificial intelligence model trained for domain identification. Also, for example, the artificial intelligence model for domain identification may be part of the NLU model or may be a model separate from the NLU model.

In operation S330, the server 2000 may modify the text using a text modification model corresponding to the determined domain. The server 2000 may include a plurality of text modification models corresponding to a plurality of domains, and among a plurality of text modification models, a text modification model corresponding to the domain identified in operation S325 may be selected.

The server 2000 may select a domain corresponding to the domain identified in operation S325 from among the domains registered in the server 2000, and select a text correction model of the selected domain. Among the domains registered in the server 2000, the domain corresponding to the domain identified in operation S325 may be the same domain as the identified domain or a similar domain. For example, when a plurality of domains registered in the server 2000 are “movie”, “place”, and “region name”, and the domain “movie” identified in operation S325, the server 2000 is “movie” You can choose For example, when a plurality of domains in the server 2000 are “video content”, “place”, and “region name”, and the domain “movie” identified in operation S325, the server 2000 is “video content” You can choose In this case, information about an identification value similar to the identification value of the domain of the server 2000 may be stored in the server 2000.

Also, the server 2000 may generate the modified text using the selected text modification model. The server 2000 may input text into the selected text correction model and obtain the corrected text output from the text correction model. In this case, the server 2000 may pre-process the format of the text received from the device 1000 to be suitable for the text correction model, and may input the preprocessed value into the text correction model.

If the text received from the device 1000 is related to a plurality of domains, the server 2000 may select a plurality of text correction models corresponding to the plurality of domains for text correction. In this case, the server 2000 may obtain a modified text to be provided to the device 1000 from modified texts output from a plurality of text modification models. For example, when the server 2000 generates a plurality of modified texts using a plurality of text modification models, the reliability of the plurality of modified texts is compared, and the modified text with high reliability is displayed on the device 1000. It can be decided by the modified text to be provided to it. The reliability of the corrected text may be a numerical value indicating the degree to which the corrected text matches the input voice, and may include, for example, a confidence score, but is not limited thereto.

In addition, for example, the server 2000 extracts some texts from modified texts output from a plurality of text modification models and combines the extracted texts to obtain modified texts to be provided to the device 1000 can do. For example, when the server 2000 generates a first modified text and a second modified text using a plurality of text modification models, and the reliability of a part of the first modified text and the reliability of a part of the second modified text are high In addition, the server 2000 may obtain a modified text to be provided to the device 1000 by combining a part of the first modified text and a part of the second modified text.

In operation S335, the server 2000 may provide the modified text to the device 1000.

Meanwhile, in FIG. 3, it has been described that the device 1000 requests the modified text from the server 2000 and the server 2000 provides the modified text to the device 1000, but is not limited thereto. The server 2000 may provide various types of voice assistant services to the device 1000 by using the modified text. The voice assistant service may be a service that provides a conversation with a user. In the voice assistant service, a response message can be provided to the user as if a person had a direct conversation with the user in consideration of the situation of the user and the device. In addition, in the voice assistant service, information required by the user, such as a personal assistant of the user, may be appropriately generated and provided to the user. The voice assistant service includes, for example, a broadcasting service, a content sharing service, a content providing service, a power management service, a game providing service, a chat service, a document writing service, a search service, a call service, a photo shooting service, a transportation recommendation service, and In connection with various services such as a video playback service, information or functions required by the user may be provided to the user.

In this case, the server 2000 uses a Natural Language Understanding (NLU) model, a Dialog Manager (DM) model, and a Natural Language Generating (NLG) model in the server 2000 in order to provide a voice assistant service based on text. Thus, information for performing a conversation with the user may be provided to the device 1000. Also, the server 2000 may directly control other devices (not shown) based on the result of analyzing the text. In addition, the server 2000 generates control information for allowing the device 1000 to control another device (not shown) based on the result of analyzing the modified text, and transmits the generated control information to the device 1000. You can also provide.

4 is a diagram illustrating an example in which a server according to an embodiment of the present disclosure identifies a domain related to text and selects a text correction model of a domain related to text.

Referring to FIG. 4, text output from the ASR model 40 of the device 1000 may be provided to the domain identification module 2312 in the server 2000. The server 2000 may identify a domain related to the text received from the device 1000 using the domain identification module 2312 in the server 2000. In this case, the server 2000 may identify a domain related to the text based on the domain reliability of the text received from the device 1000. For example, the server 2000 may calculate a confidence score indicating how relevant the text received from the device 1000 is to a domain registered in advance for text correction. The domain identification module 2312 is an artificial intelligence model trained for domain identification and may output domain reliability by using text as an input value. Further, for example, the domain identification module 2312 may be a part of the NLU model or may be a model separate from the NLU model. Alternatively, the domain identification module 2312 may identify a domain related to text based on a rule.

In FIG. 4, the domain identification module 2312 may obtain, for example, a domain reliability for a first domain, a domain reliability for a second domain, and a domain reliability for a third domain.

In addition, the model selection module 2313 of the server 2000 may select a text modification model to modify the text. For example, the model selection module 2313 compares the domain reliability for the first domain, the domain reliability for the second domain, and the domain reliability for the third domain, and determines a value with the highest domain reliability for the first domain. Have can be identified. Also, the model selection module 2313 may select the text correction model 41 of the first domain from among a plurality of text correction models 41, 42, and 33 in the server 2000.

The server 2000 may input the text received from the device 1000 into the text correction model 41 of the first domain, and obtain the corrected text output from the text correction model 41. Thereafter, the server 2000 may provide the modified text to the device 1000.

5 is a diagram illustrating an example in which a device identifies a domain related to text and a server selects a text modification model of a domain related to text according to an embodiment of the present disclosure.

Referring to FIG. 5, text output from the ASR model 50 of the device 1000 may be provided to the domain identification module 1440 in the device 1000. The device 1000 may use the domain identification module 1440 in the device 1000 to identify a domain related to the text output from the ASR model 50. In this case, the device 1000 may identify a domain related to the text based on the domain reliability of the text output from the ASR model 50. For example, the device 1000 may calculate a confidence score indicating how relevant the text output from the ASR model 50 is to a pre-registered domain. The domain identification module 1440 is an artificial intelligence model trained for domain identification and may output domain reliability by using text as an input value. Also, for example, the domain identification module 1440 may be a part of the NLU model or may be a model separate from the NLU model. Alternatively, the domain identification module 1440 may identify a domain related to text based on a rule.

In FIG. 5, the domain identification module 1440 may obtain, for example, a domain reliability for a first domain, a domain reliability for a second domain, and a domain reliability for a third domain.

Also, the device 1000 may provide the text output from the ASR model 50 to the server 2000. In addition, the device 1000 may provide the server 2000 with the domain reliability obtained by the domain identification module 1440. Alternatively, the device 1000 may identify a domain related to the text based on the domain reliability obtained by the domain identification module 1440 and provide identification information of the identified domain to the server 2000.

The model selection module 2313 of the server 2000 may select a text modification model to modify the text. When the device 1000 provides the server 2000 with the domain reliability, the model selection module 2313 may, for example, the domain reliability for the first domain, the domain reliability for the second domain, and the third domain. The domain reliability for the first domain may be compared, and it may be identified that the domain reliability for the first domain has the highest value. Also, the model selection module 2313 may select the text correction model 51 of the first domain from among a plurality of text correction models 51, 52, and 53 in the server 2000.

Alternatively, when the device 1000 provides the server 2000 with an identification value of a domain related to text, the model selection module 2313 may, for example, determine the domain identification value received from the device 1000. A text correction model 51 of one domain can be selected.

The server 2000 may input the text received from the device 1000 into the text correction model 51 of the first domain and obtain the corrected text output from the text correction model 51. Thereafter, the server 2000 may provide the modified text to the device 1000.

6 is a diagram illustrating an example in which a server and a device according to an embodiment of the present disclosure identify domains related to text, and the server selects a text modification model of a domain related to text.

Referring to FIG. 6, text output from the ASR model 60 of the device 1000 may be provided to the first domain identification module 61 in the device 1000. The first domain identification module 61 may be a domain identification module 1440. The device 1000 may obtain the first domain reliability of the text output from the ASR model 60 by using the first domain identification module 61 in the device 1000. Also, the device 1000 may provide the text output from the ASR model 60 and the first domain reliability obtained from the first domain identification module 61 to the server 2000.

The server 2000 may receive text from the device 1000 and provide the received text to the second domain identification module 62 in the server 2000. The second domain identification module 62 may be a domain identification module 2312. The server 2000 may acquire a second domain reliability of the text received from the device 1000 by using the second domain identification module 62 in the server 2000.

Thereafter, the model selection module 2313 of the server 2000 may select a text correction model to modify the text based on the reliability of the first domain and the reliability of the second domain. For example, the model selection module 2313 selects a first domain to which text is related from among domains registered in the server 2000 based on the sum of the weights of the reliability of the first domain and the reliability of the second domain, and the selected first domain The text editing module 63 of the domain can be selected. In addition, in this case, for example, since the reliability of the first domain and the reliability of the second domain are normalized, the weights may be respectively reflected in the reliability of the first domain and the reliability of the second domain, but are not limited thereto. A method for the server 2000 to select a domain related to the text based on the reliability of the first domain and the reliability of the second domain will be described in more detail with reference to FIG. 7 to be described later.

The server 2000 may input the text received from the device 1000 into the text correction model 63 of the first domain, and obtain the corrected text output from the text correction model 63. Thereafter, the server 2000 may provide the modified text to the device 1000.

7 is a flowchart of a method for selecting a domain related to text by using a domain reliability obtained from a device and a domain reliability obtained from a server according to an embodiment of the present disclosure.

In S700, the server 2000 may receive from the device 1000 the first domain reliability of the text calculated by the first domain identification module 61 of the device 1000. The first domain identification module 61 of the device 1000 may calculate a confidence score indicating to what extent the text output from the ASR model 60 is related to a pre-registered domain. In this case, for example, the domain identification module 61 may output the first domain reliability by using text as an input value as an artificial intelligence model trained for domain identification. When a plurality of domains are registered in the device 1000, the first domain identification module 61 may obtain a plurality of first domain reliability indicating how much text is related to each of the plurality of domains. have.

In S710, the server 2000 may calculate a second domain reliability of the text received from the device 1000 using the second domain identification module 62. The second domain identification module 62 of the server 2000 may calculate a confidence score indicating to what extent the text received from the device 1000 is related to a pre-registered domain. In this case, for example, the domain identification module 62 may output the second domain reliability by using text as an input value as an artificial intelligence model trained for domain identification. When a plurality of domains are registered in the server 2000, the second domain identification module 62 can obtain a plurality of second domain reliability indicating how much text is related to each of the plurality of domains. have.

In S720, the server 2000 may select a domain related to the text based on the reliability of the first domain and the reliability of the second domain. The server 2000 may select a domain related to text from among a plurality of pre-registered domains based on a weighted sum of the reliability of the first domain and the reliability of the second domain. For example, when a domain having a high first domain reliability and a domain having a high second domain reliability are different from each other, the server 2000 assigns a preset first weight to the first domain reliability and is based on the second domain reliability. The set second weight may be assigned. Also, the server 2000 may select a text-related domain from among a plurality of pre-registered domains based on the reliability of the first domain to which the first weight is assigned and the reliability of the second domain to which the second weight is assigned. In this case, for example, since the reliability of the first domain and the reliability of the second domain are normalized, weights may be respectively reflected in the reliability of the first domain and the reliability of the second domain, but are not limited thereto. For example, if the reliability of the first domain is the reliability of the domain of the upper layer and the reliability of the second domain is the reliability of the domain of the lower layer, a low weight is given to the first domain reliability and a high weight is applied to the second domain. Can be given.

According to an embodiment, the server 2000 may consider the reliability of text output from the ASR model of the device 1000 in order to select a domain. In this case, the reliability of the text output from the ASR model may be obtained from the device 1000 and provided to the server 2000, but is not limited thereto. Also, for example, the server 2000 assigns a predetermined third weight to the reliability of the text from the device 1000, the first domain reliability to which the first weight is assigned, and the second domain reliability to which the second weight is assigned. , And a text-related domain from among a plurality of previously registered domains may be selected based on the reliability of the text to which the third weight is assigned.

On the other hand, for example, when a domain having a high first domain reliability and a domain having a high second domain reliability are the same, the server 2000 does not consider the weight, and among a plurality of domains, a high first domain You can select a domain with reliability.

8 is a diagram illustrating an example in which a server according to an embodiment of the present disclosure selects a text correction model by using a domain identification module selected from among a plurality of domain identification modules in the server.

Referring to FIG. 8, text output from the ASR model 80 of the device 1000 may be provided to the first domain identification module 81 in the device 1000. The first domain identification module 81 may be a domain identification module 1440. The device 1000 may acquire the first domain reliability of the text output from the ASR model 80 by using the first domain identification module 81 in the device 1000. Further, the device 1000 may provide the text output from the ASR model 80 and the first domain reliability obtained from the first domain identification module 81 to the server 2000.

The server 2000 may select one of the plurality of second domain identification modules 82 in the server 2000 based on the reliability of the first domain received from the device 1000. Domains for speech recognition in a speech recognition system may be hierarchically set. Domains for speech recognition are, for example, a first layer domain, a second layer domain that is a subdomain of a first layer domain, a third layer domain that is a lower domain of a second layer domain, and It may include domains of the fourth layer, which are subdomains of the domain of the third layer. Further, for example, the second domain identification modules 82 may include, for example, at least one second layer domain identification module 82-1, at least one third layer domain identification module 82- 2) and at least one fourth layer domain identification module 82-3. In addition, for example, the domain of the first layer corresponds to the first domain identification module 81, the domain of the second layer corresponds to the domain identification module 82-1 of the second layer, and The domain corresponds to the domain identification module 82-2 of the third layer, the domain of the fourth layer corresponds to the domain identification module 82-3 of the fourth layer, and the domain of the fifth layer corresponds to the text correction model. Can correspond. In this case, the server 2000 uses the domain identification module selection module 2311 to have a high reliability among a plurality of second layer domains according to the first domain reliability calculated from the first domain identification module 81. The domain of the second layer can be identified. Also, the domain identification module selection module 2311 of the server 2000 may select the second layer domain identification module 82-1 corresponding to the identified second layer domain.

Also, for example, the server 2000 may acquire the second domain reliability of the text by using the domain identification module 82-1 of the selected second layer. For example, the domain identification module selection module 2311 of the server 2000 identifies a third layer domain having a high reliability among a plurality of third layer domains based on the second domain reliability, and The third layer domain identification module 82-2 corresponding to the third layer domain may be selected.

Also, for example, the server 2000 may acquire the third domain reliability of the text using the selected third layer domain identification module 82-2. For example, the domain identification module selection module 2311 of the server 2000 identifies a fourth layer domain having a high reliability among a plurality of fourth layer domains based on the third domain reliability, and The fourth layer domain identification module 82-3 corresponding to the 4th layer domain may be selected.

In addition, for example, the server 2000 may acquire the reliability of the fourth domain of the text using the domain identification module 82-3 of the selected fourth layer. Also, the model selection module 2313 of the server 2000 may select the text correction model 85 corresponding to the third domain from among a plurality of text correction models based on the fourth domain reliability.

Thereafter, the server 2000 may input the text received from the device 1000 into the text correction model 85 of the third domain, and obtain the corrected text output from the text correction model 65. The server 2000 may provide the modified text to the device 1000.

In FIG. 8, the layer of the second domain identification module 82 in the server 2000 includes a second layer, a third layer, and a fourth layer, and the server 2000 is a second layer domain identification module 82- 1), the domain identification module 82-2 of the third layer and the domain identification module 82-3 of the fourth layer are sequentially selected, but are not limited thereto.

The server 2000 includes the second domain reliability calculated from the domain identification module 82-1 of the second layer, the third domain reliability calculated from the domain identification module 82-2 of the third layer, and the domain of the fourth layer. In consideration of the reliability of the fourth domain calculated from the identification module 82-3 together, a domain for correcting the text may be selected. In this case, the server 2000 uses the second domain reliability calculated from the second layer domain identification module 82-1, the third domain reliability calculated from the third layer domain identification module 82-2, and the fourth The fourth domain reliability calculated from the hierarchical domain identification module 82-3 is normalized, and the normalized values are compared with each other to select a domain for correcting the text.

For example, the first domain identification module 81 calculates domain reliability for the domains of the first layer, and the domain identification module 82-1 of the second layer is the domain reliability for the domains of the second layer. The domain identification module 82-2 of the third layer calculates domain reliability for the domains of the third layer, and the domain identification module 82-3 of the fourth layer calculates the domains of the fourth layer. Domain reliability levels for can be calculated.

Meanwhile, the layer of the second domain identification module 82 in the server 2000 may include only the second layer. Alternatively, the layer of the second domain identification module 82 in the server 2000 may include more layers in addition to the second to fourth layers. In this case, according to the layers of the second domain identification module 82 in the server 2000, the server 2000 may include a domain identification module corresponding to each layer.

For example, the first domain identification module 81 may calculate the reliability of the domain “location” of the first layer as 60% and the reliability of the domain “weather” of the first layer as 30%. have. The domain identification module 82-1 of the second layer calculates the reliability of the domain “Canada” of the second layer as 40%, the reliability of the domain “USA” of the second layer is calculated as 20%, and The reliability of the layer 2 domain “Rain” can be calculated as 25%. The domain identification module 82-2 of the third layer calculates the reliability of the domain “Brish Columbia” of the third layer as 20%, the reliability of the domain “Ontario” of the third layer is calculated as 30%, and the third layer The reliability of the domain “New York” of'New York' can be calculated as 10%, and the reliability of the domain'precipitation' of the third layer can be calculated as 5%.

In addition, for example, the domain selection module 2313 may assign a first weight to the reliability of the domain “location” of the first layer and the reliability of the domain “weather” of the first layer, respectively. The domain selection module 2313 assigns a second weight to the reliability of the domain “Canada” of the second layer, the reliability of the domain “USA” of the second layer, and the reliability of the domain “Rain” of the second layer, respectively. I can. The domain selection module 2313 has the reliability of the domain “Brish Columbia” of the third layer, the reliability of the domain “Ontario” of the third layer, the reliability of the domain “New York” of the third layer, and the domain “precipitation” of the third layer. Each of the third weights can be assigned to the reliability. In this case, the second weight may be larger than the first weight and smaller than the third weight. In addition, the domain selection module 2313 may select a domain for text correction by considering the reliability levels to which the first weight is added, the reliability levels to which the second weight is assigned, and the reliability levels to which the third weight is assigned together.

Also, for example, the domain selection module 2313 may calculate a first weight sum of the reliability for “location”, the reliability for “Canada”, and the reliability for “Brish Columbia”. The domain selection module 2313 may calculate a second weight sum of the reliability for “location”, the reliability for “Canada”, and the reliability for “Ontario”. The domain selection module 2313 may calculate a third weight sum of the reliability for “location”, the reliability for “USA”, and the reliability for “New York”. In addition, for example, the domain selection module 2313 may calculate a fourth weight sum of the reliability for “Weather”, the reliability for “Rain”, and the reliability for “Precipitation”.

For example, by comparing the calculated weight sums, the domain selection module 2313 may identify that the first weight sum is as high as possible and determine “Brish Columbia” as a domain for text correction.

Alternatively, for example, the domain selection module 2313 may select the domain of the second layer based on the reliability of the domains of the first layer and the domains of the second layer. The domain selection module 2313 may select subdomains related to the selected second layer domain, and text may be modified using text modification models corresponding to the selected subdomains.

In addition, in FIG. 8, the first domain identification module 81 in the device 1000 corresponds to the first layer, and the second domain identification module 82 in the server 2000 corresponds to the second to fourth layers. It has been described as being, but is not limited thereto. For example, the first domain identification module 81 in the device 1000 may correspond to the first layer, and the second domain identification module 82 in the server 2000 may correspond to the first layer to the third layer. have.

Meanwhile, weights for domain reliability levels may be assigned based on context information related to the device 1000. The context information may include at least one of surrounding environment information of the device 1000, state information of the device 1000, user state information, user's device 1000 usage history information, and user schedule information. It is not limited. The surrounding environment information of the device 1000 means environment information within a predetermined radius from the device 1000, and may include, for example, weather information, temperature information, humidity information, illumination information, noise information, sound information, etc. However, it is not limited thereto. The state information of the device 1000 includes mode information of the device 1000 (eg, sound mode, vibration mode, silent mode, power saving mode, blocking mode, multi-window mode, automatic rotation mode, etc.), and the location of the device 1000. Information, time information, activation information of the communication module (e.g., Wi-Fi ON / Bluetooth OFF / GPS ON / NFC ON, etc.), network connection status information of the device 1000, application information running on the device 1000 (e.g. , Application identification information, application type, application use time, application use period), etc., but are not limited thereto. The user's state information is information about the user's movement and life pattern, and may include information about the user's walking state, exercising state, driving state, sleeping state, user's mood state, etc., but is not limited thereto. . The user's device 1000 usage history information is information on the history of the user's use of the device 1000, and includes the execution history of the application, the history of functions executed in the application, the user's call history, and the user's text history. It may include, but is not limited thereto.

For example, weights for domain reliability levels may be determined based on context information about an application running in the device 1000. For example, when a user's voice input is an input for an application running in the device 1000, a high weight may be given to the domain reliability of a domain related to the application. Alternatively, the domain related to the application may be directly determined as a domain for text correction. For example, when a voice input “Acrovista” is input while a map application is being executed in the device 1000, a high weight is assigned to the map domain, or a domain for text correction may be directly determined as the map domain.

For example, based on the conversation history of the user through the voice assistant service provided by the device 1000, weights for domain reliability levels may be determined. For example, through the voice assistant service during user is having a conversation related to the device (1000) with music, when "IU Search me," called a voice input is entered in the device 1000, or given a higher weight in the music domain, The domain for text correction can be directly determined as the music domain.

For example, weights for domain reliability levels may be determined based on sensing information collected by the device 1000. A weight may be assigned to a domain based on location information (eg, GPS information) obtained from the device 1000. For example, when the location of the device 1000 is near a movie theater, a high weight may be given to the movie domain. For example, when a user's voice input is input to the device 1000 while the device 1000 is searching for a restaurant, a high weight may be given to a domain related to a location where the device 1000 is located.

Meanwhile, weights for domain reliability levels may be assigned based on trend information. For example, a high weight may be given to a domain for major news or a domain for a real-time search word through a portal site.

9 is a flowchart of a method for selecting a domain for text correction by using a domain identification module selected from among a plurality of domain identification modules by a server according to an embodiment of the present disclosure.

10 is a diagram illustrating an example of a first domain identification module, a second domain identification module, and a text correction model related to hierarchically classified domains according to an embodiment of the present disclosure.

In FIGS. 9 and 10, domains for speech recognition may be classified into a first layer, a second layer, and a third layer, for example.

In S900, the server 2000 may receive from the device 1000 the first domain reliability of the text calculated by the first domain identification module 81 of the device 1000. The server 2000 may receive from the device 1000 the first domain reliability of the text output from the ASR model of the device 1000. Referring to FIG. 10, for example, the first domain identification module 100 of the device 1000 may correspond to “all”, which is a domain of the first layer related to “location”, and the first domain identification module ( The first domain reliability calculated from 100) may be the domain reliability for the domain of the second layer related to “country”. For example, the first domain reliability may include domain reliability for “Canada” and domain reliability for domain “USA”. Also, for example, the second domain identification module 101 may correspond to the domain “Canada”, and the second domain identification module 102 may correspond to the domain “USA”.

In S910, the server 2000 may select at least one of the plurality of second domain identification modules 82 based on the first domain reliability. The domain identification module selection module 2311 of the server 2000 may select the domain identification module 82-1 of the second layer from among the plurality of second domain identification modules 82 based on the first domain reliability. Referring to FIG. 10, for example, the server 2000 compares the domain reliability for “Canada” and the domain reliability for the domain “USA”, and identifies that the domain reliability for “Canada” is higher than a predetermined threshold. I can. In addition, the server 2000 is, among the second domain identification module 101 corresponding to the domain “Canada” and the second domain identification module 102 corresponding to the domain “USA”, the second domain corresponding to “Canada” The identification module 101 can be selected. In this case, the second domain identification module 101 and the second domain identification module 102 may be a second domain identification module corresponding to the second layer.

In S920, the server 2000 may calculate the reliability of the second domain of the text using the domain identification module 82-1 of the selected second layer. The domain identification module 82-1 of the second layer may calculate the reliability of the second domain by inputting text. Referring to FIG. 10, the second domain reliability calculated by the second domain identification module 101 may be a domain reliability for a domain of a third layer related to “province or state”. For example, the second domain reliability may include domain reliability for “British Columbia”, domain reliability for domain “Ontario”, domain reliability for domain “New York”, and domain reliability for domain “Illinois”. have. In addition, for example, the text correction model 103 corresponds to the domain “British Columbia”, the text correction model 104 corresponds to the domain “Ontario”, and the text correction model 105 corresponds to the domain “New York”. Correspondingly, the text correction model 106 may correspond to the domain "Illinois".

In S930, the server 2000 may select a domain related to text correction based on the second domain reliability. The model selection module 2313 of the server 2000 may select one of the plurality of text correction models 83, 84, and 85 based on the second domain reliability. Referring to FIG. 10, for example, the server 2000 has domain reliability for “British Columbia”, domain reliability for domain “Ontario”, domain reliability for domain “New York”, and domain “Illinois”. The domain reliability is compared, and it can be identified that the domain reliability for “British Columbia” is higher than a predetermined threshold. Further, the server 2000 can select a domain "British Columbia" as a domain related to text correction. Accordingly, the text may be modified by the text modification model 103 corresponding to the domain “British Columbia”.

9 and 10, the first domain identification module 81 corresponds to the first layer of the domain, the second domain identification module 82 corresponds to the second layer of the domain, and text correction models 83 and 84 , 85) has been described as corresponding to the third layer, but is not limited thereto. For example, the second domain identification module 82 may correspond to more layers. For example, the first domain identification module 81 corresponds to the first layer of the domain, the second domain identification module 82 corresponds to the second layer, the third layer and the fourth layer of the domain, and text correction The models 83, 84, and 85 may correspond to the fifth layer, but are not limited thereto.

In FIG. 10, the server 2000 calculates the domain reliability calculated from the first domain identification module 100, the domain reliability calculated from the second domain identification module 101, and the domain reliability calculated from the second domain identification module 102. It is also possible to select a text correction model for text correction by normalizing each and comparing the normalized values.

11 is a diagram illustrating an example in which a server corrects text using a plurality of text correction models according to an embodiment of the present disclosure.

Referring to FIG. 11, text output from the ASR model 110 of the device 1000 may be provided to the domain identification module 2312 in the server 2000. The server 2000 may identify a domain related to the text received from the device 1000 using the domain identification module 2312 in the server 2000. In this case, the server 2000 may identify a domain related to the text based on the domain reliability of the text received from the device 1000.

The domain identification module 2312 may obtain, for example, a domain reliability for a first domain, a domain reliability for a second domain, and a domain reliability for a third domain. For example, the domain identification module 2312 may divide the text into a plurality of sections, and obtain a domain reliability for a first domain, a domain reliability for a second domain, and a domain reliability for a third domain for each section. . For example, the domain identification module 2312 divides the text into a first section, a second section, and a third section, and for each of the first section, the second section, and the third section, the domain reliability of the first domain , Domain reliability for the second domain and domain reliability for the third domain may be obtained. For example, when the server 2000 receives a text stream, domain reliability for the text of the first section is obtained, domain reliability for the text of the second section is obtained, and the text of the third section is Domain credibility can be obtained. In this case, the server 2000 can modify the text already received in real time without waiting for the text to be received later while receiving the text stream, and can identify the domain related to the text at a faster rate.

Alternatively, for example, when the server 2000 receives a text stream, domain reliability for a plurality of sections of text may be accumulated and calculated. For example, the server 2000 may divide the text of one sentence into a plurality of sections while receiving the text stream, obtain domain reliability for the text of the first section, and obtain the text of the first section and the second section. Domain reliability levels for the text may be obtained, and domain reliability levels for the text of the first to third period may be obtained. In this case, the server 2000 calculates the domain reliability in sentences, but by accumulating a plurality of sections to calculate the domain reliability, it is possible to more efficiently identify the domain related to the text.

Alternatively, for example, the domain identification module 2312 may obtain a domain reliability for a first domain, a domain reliability for a second domain, and a domain reliability for a third domain for text in a sentence unit.

According to an embodiment, the model selection module 2313 of the server 2000 may select a text correction model to modify text. For example, when the text is divided into a plurality of sections, the server 2000 may select a different text correction model for each section of the text. For example, when the text is divided into a first section, a second section, and a third section, selecting the text correction model 111 to correct the first section of the text, and correcting the second section of the text The text correction model 112 may be selected to correct the text, and the text correction model 113 may be selected to correct the third section of the text.

Alternatively, for example, the server 2000 may select a plurality of text correction models to correct text in units of sentences. For example, the server 2000 may select a text correction model 111, a text correction model 112, and a text correction model 113 to correct text in a sentence unit.

Thereafter, the server 2000 includes a first modified text output from the text modification model 111, a second modified text output from the text modification model 112, and a third modified text output from the text modification model 113. Using the text, a modified text for the text received from the device 1000 may be obtained.

For example, the server 2000 selects one or more of a first modified text, a second modified text, and a third modified text, and uses at least a portion of the selected text to receive the text received from the device 1000. You can get the modified text for. Or, for example, the server 2000 may obtain the modified text for the text received from the device 1000 by selecting one of the first modified text, the second modified text, and the third modified text. I can. Alternatively, for example, the server 2000 may combine at least a part of the first modified text, at least a part of the second modified text, and at least a part of the third modified text, thereby adding the text received from the device 1000. You can get the modified text for

Thereafter, the server 2000 may provide the modified text to the device 1000.

12 is a flowchart of a method of accumulating and calculating domain reliability for texts in a plurality of sections by a server according to an embodiment of the present disclosure.

In S1200, the server 2000 may acquire the first section of the text. The text may be divided into a plurality of sections, and the sections of the text may be divided into, for example, words, words, or phrases. The server 2000 may receive text from the device 1000 in the form of a text stream. In this case, the server 2000 may acquire the first section of the text while receiving the text stream in real time. Alternatively, the server 2000 may receive text in a sentence form from the device 1000 and extract the text of the first section from the received text.

In S1210, the server 2000 may calculate a domain reliability for the text of the first section. The server 2000 may calculate the domain reliability for the text of the first section for domains registered in the server 2000.

In S1220, the server 2000 may acquire a second section of the text. When the server 2000 receives text from the device 1000 in the form of a text stream, the server 2000 may acquire the second section of the text while receiving the text stream in real time. Alternatively, the server 2000 may receive text in the form of a sentence from the device 1000 and extract the text of the second section from the received text.

In S1230, the server 2000 may calculate the domain reliability for the text of the first section and the second section. The server 2000 may accumulate the text of the first section and the text of the second section, and calculate a domain reliability for the accumulated text including the first section and the second section.

In S1240, the server 2000 may acquire the nth section of the text. When the server 2000 receives text from the device 1000 in the form of a text stream, the server 2000 may acquire the n-th section of the text while receiving the text stream in real time. Alternatively, the server 2000 may receive text in the form of a sentence from the device 1000 and extract the text of the nth section from the received text.

In S1250, the server 2000 may calculate a domain reliability for the text of the first to nth interval. The server 2000 may accumulate the text of the first section through the text of the nth section and calculate a domain reliability for the accumulated text including the first section through the nth section.

In S1260, the server 2000 may determine a domain to modify the text received from the device 1000 based on the domain reliability of the text in the first to nth period.

13 is a diagram illustrating an example in which a server obtains domain reliability for a text stream accumulated in word units according to an embodiment of the present disclosure.

Referring to FIG. 13, the domain identification module 2312 of the server 2000 may classify text by word unit, accumulate the text, and obtain domain reliability with respect to the accumulated text.

For example, when “new”, which is the text of the first section, is input to the domain identification module 2312, the domain identification module 2312 lowers the domain reliability related to “new” to “0.1” and identifies the domain. Can be output as “Rejected”.

Thereafter, when "Twice", which is the text of the second section, is input to the domain identification module 2312, the domain identification module 2312 is used as the text of the first section, "New" and the text of "Twice", which is the second section. ”Is accumulated, and the domain identification value “Music” related to the accumulated text “New Twice” and the domain reliability level “0.7” can be output.

Thereafter, when the domain identification module 2312 inputs the text of the third section, “Yes Oh No”, the domain identification module 2312 uses the text of the first section, “New”, and the text of the second section. The domain identification value “Music” related to the accumulated text “New Twice Yes Ono” and the domain reliability level “0.9” can be output. have.

Thereafter, when the domain identification module 2312 inputs the text of the fourth section, "Play it," the domain identification module 2312 uses the text of the first section, "New", and the text of the second section, "Twice. ”, the text of the third section, “Yes Oh No”, and the text of the fourth section, “Play me” are accumulated, and the domain identification value related to the accumulated text “New Twice Yes Oh No Play” is “Music "" and "1.0" with the domain reliability can be output.

Accordingly, the server 2000 may select a domain modification model of the domain “Music” as a domain modification model for correcting “New Twice Yes Oh Play”.

In FIG. 13, it has been described that the domain identification module 2312 outputs the domain having the highest value and the domain reliability, but is not limited thereto. The domain identification module 2312 may output domain reliability for each domain with respect to a plurality of domains registered in the server 2000.

In FIG. 13, it has been described that the domain reliability for the text of a plurality of sections is calculated by accumulating, but is not limited thereto. For example, the server 2000 may sequentially select a text correction model to modify the text of each section with respect to the text of a plurality of sections. For example, when the server 2000 receives the text stream, the server 2000 calculates the domain reliability of the text in the first section while receiving the text stream, and modifies the text to modify the text in the first section. Select the model, calculate the domain reliability for the text of the second section, and modify the text of the n-th section by selecting a text correction model to modify the text of the second section, and calculate the domain reliability for the text of the n-th section. You can also choose a text correction model.

14 is a flowchart illustrating a method of dividing text into a plurality of sections by a server and selecting a domain for each text of each section according to an embodiment of the present disclosure.

In S1400, the server 2000 may calculate the domain reliability of the text for each of the plurality of domains. The server 2000 may calculate a domain reliability for each of a plurality of domains registered in the server 2000 with respect to the text received from the device 1000.

In S1410, the server 2000 may divide the text into a plurality of sections by comparing the calculated domain reliability. The server 2000 may divide the text into a plurality of sections by identifying a text section with high domain reliability for each domain.

In operation S1420, the server 2000 may select a domain for text correction for each text of each segment. The server 2000 may select, for each text of each section, a domain having the highest domain reliability for the text of each section, as a domain corresponding to the text of each section.

15 is a diagram illustrating an example in which a server compares the domain reliability of texts for each of a plurality of domains, and selects and corrects a text correction model for each text in each section.

Referring to FIG. 15, the server 2000 may classify sections of text while receiving a text stream, and calculate domain reliability in real time with respect to the text of each section. For example, the server 2000 may receive a text stream stating "I will meet Gil Hong-i near Yeomtong Station today and see A Betters". While receiving the text stream, the server 2000 identifies “near Yeomtong Station today” as the first section, identifies “Meet Gilhongi” as the second section, and suggests “I'll see a betters”. It can be identified by 3 sections. In addition, the server 2000 receives the text stream, the domain reliability related to “near Yeomtong Station today”, the domain reliability related to “Meet Gilhongi” and the domain reliability related to “I'm going to see a betters” Can be calculated sequentially.

For example, the domain identification module 2312 of the server 2000, for each section of the text “I will meet Gil Hong-i near Yeomtong Station today and see A Betters”, the domain reliability of the “movie” domain, “ You can calculate the domain reliability of the “place” domain and the domain reliability of the “contact” domain.

For example, the domain selection module 2313 of the server 2000 may compare the domain reliability of the “movie” domain, the domain reliability of the “place” domain, and the domain reliability of the “contact” domain. The domain selection module 2313 may identify that the domain reliability of the “place” domain is high with respect to “near the Yeomtong Station today”. The domain selection module 2313 may identify that the domain reliability of the "contact" domain is high with respect to "Meeting Gil Hong". The domain selection module 2313 may identify that the domain reliability of the “movie” domain is high with respect to “I will watch A Betters”. Accordingly, the domain selection module 2313 identifies “I will meet Gil Hong-i near Yeomtong Station today and see A Betters” as the first section from the text stream “I will meet Gilhongi near Yeomtong Station today” as the first section, and "Meet me" can be identified as the second section, and "I'll see a betters" can be sequentially identified as the third section.

In addition, for example, the domain selection module 2313 selects the “place” domain as the domain related to “Today near Yeomtong Station”, and selects the “contact” domain as the domain related to “Meet Gil Hong-i”, You can select the “Movies” domain as the domain related to “Ah We're Watching Veggies”.

The text correction model in the “Place” domain is “Today near Yeomtong Station” to “Today near Yeongtong Station”, and the text correction model in the “Contact” domain is “Meet Gil Hong” to “Meet Gildong”. The text correction model in the "Movie" domain can be modified from "I'm going to see the Avengers" to "I'm going to see the Avengers". At least one of the text correction operation of the text correction model of the “place” domain, the text correction operation of the text correction model of the “contact” domain, and the text correction operation of the text correction model of the movie domain are sequential while the text stream is being received. Can be done with

16 is a diagram illustrating an example in which a server according to an embodiment of the present disclosure modifies text received from a device by using text modified from a plurality of text correction models.

Referring to FIG. 16, for example, the server 2000 writes a text 160 stating "I will meet Gilhongi near Yeomtong Station today and see A Betters", a text correction model in the "place" domain, "contact information" It can be provided to the text modification model of the ”domain and the text modification model of the “movie” domain respectively.

Accordingly, the text correction model in the “Place” domain outputs a revised text 161 stating “I will meet Gil Hong-i near Yeongtong Station today and see A Betters”, and the text correction model in the “Movie” domain is “Today. The corrected text (162) is printed, saying, "I will meet Gil Hong-i near Yeomtong Station and see the Avengers." The corrected text 163 may be output.

Thereafter, the server 2000 is the corrected word “Youngtong” in the corrected text 161, “Avengers” as the corrected word in the corrected text 162, and “Gildongi” that is the corrected word in the corrected text 163. "I will meet Gildong near Yeongtong Station today and see the Avengers", which is a modified text 164 to be provided to the device 1000 may be generated.

17 is a block diagram of a server according to an embodiment of the present disclosure.

Referring to FIG. 17, a server 2000 according to an embodiment of the present disclosure includes a communication interface 2100, a processor 2200, and a storage unit 2300, and the storage unit 2300 is a domain management module 2310. ), a text modification module 2320, an NLU module 2330, and a speech analysis management module 2340.

The communication interface 2100 may include one or more components for communication with the device 1000 and another server (not shown). The communication interface 2100 may transmit/receive information for voice recognition and voice assistant service with the device 1000 and other servers (not shown). The communication interface 2100 may include, for example, a local area network (LAN), a wide area network (WAN), a value added network (VAN), a mobile radio communication network, and Communication may be performed through a satellite communication network and a combination thereof, but is not limited thereto.

The processor 2200 controls the overall operation of the server 2000. The processor 2200 may generally control operations of the server 2000 in the present specification by executing programs stored in the storage unit 2300.

The storage unit 2300 may store a program for processing and controlling the processor 2200, and may store data input to the server 2000 or output from the server 2000. The storage unit 2300 includes, for example, a flash memory type, a hard disk type, a multimedia card micro type, and a card type memory (eg, SD or XD memory, etc.), RAM (RAM, Random Access Memory) SRAM (Static Random Access Memory), ROM (ROM, Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory) , Magnetic memory, magnetic disk, optical disk may include at least one type of storage medium, but is not limited thereto.

Programs stored in the storage unit 2300 can be classified into a plurality of modules according to their functions. For example, a domain management module 2310, text modification modules 2321 and 2320, NLU module 2330, and voice The analysis management module 2340 may be classified.

The domain management module 2310 provides the text received from the device 1000 to the text correction module 2320. The domain management module 2310 may include a domain identification module selection module 2311, at least one domain identification module 2312, and a domain selection module 2313.

The domain identification module selection module 2311 may select the domain identification module 2312. When there are a plurality of domain identification modules 2312, the domain identification module selection module 2311 may select at least some of the plurality of domain identification modules 2312.

The domain identification module selection module 2311 may select one of the plurality of domain identification modules 2312 in the server 2000 based on the first domain reliability received from the device 1000. Domains for speech recognition in a speech recognition system may be hierarchically set. Domains for speech recognition include, for example, a domain of a first layer, domains of a second layer that is a subdomain of a domain of a first layer, and domains of a third layer that are a subdomain of a domain of a second layer. can do. Also, for example, the domain of the first layer corresponds to the domain identification module 1440 of the device 1000, the domain of the second layer corresponds to the domain identification module 2312 of the server 2000, and the third The hierarchical domain may correspond to the text modification module 2320. In this case, the domain identification module selection module 2311 is a second layer domain having a high reliability among a plurality of second layer domains according to the first domain reliability calculated from the domain identification module 1440 of the device 1000. Can be identified. Also, the domain identification module selection module 2311 may select the domain identification module 2312 corresponding to the identified second layer domain.

The domain identification module 2312 may identify a domain for text correction. When the server 2000 receives domain information from the device 1000, the domain identification module 2312 may identify a domain for text correction from the domain information. Alternatively, when the server 2000 does not receive domain information from the device 1000, the domain identification module 2312 may identify a domain related to the text based on the domain reliability of the text received from the device 1000. I can. For example, the domain identification module 2312 may calculate a confidence score indicating how relevant a text received from the device 1000 is to a domain registered in advance for text correction. Also, the domain identification module 2312 may identify a domain related to the text received from the device 1000 based on the domain reliability calculated for the pre-registered domain. The domain identification module 2312 may identify a domain related to the text based on a rule or obtain a domain reliability related to the text by using an artificial intelligence model trained for domain identification. Also, for example, the artificial intelligence model for domain identification may be part of the NLU model or may be a model separate from the NLU model.

The domain identification module 2312 may identify a domain related to the text by accumulating a plurality of sections of text and calculating a domain reliability for the accumulated text. Alternatively, the domain identification module 2312 may divide the text into a plurality of sections and identify a related domain for each text of each section.

The domain selection module 2313 may select a text modification model corresponding to the domain identified by the domain identification module 2312 from among a plurality of text modification models 2321, 2322, 2323.

The domain selection module 2313 may select a domain corresponding to the domain identified by the domain identification module 2312 from among domains registered in the server 2000 and select a text correction model of the selected domain.

When the domain identification module 2312 divides the text into a plurality of sections and identifies a domain related to each text of each section, the domain selection module 2313 may select a domain for each text of each section.

The text modification module 2320 modifies the text received from the device 1000. The text modification module 2320 may modify text using a text modification model corresponding to the determined domain. The text correction module 2320 may include a text correction model 2321 of a first domain, a text correction model 2322 of a second domain, and a text correction model 2323 of a third domain.

The text correction module 2320 may generate the corrected text using the selected text correction model. The text correction module 2320 may input text into the selected text correction model, and obtain the corrected text output from the text correction model. In this case, the text modification module 2320 may pre-process the format of the text received from the device 1000 to fit the text modification model, and may input the preprocessed value into the text modification model.

If the text received from the device 1000 is related to a plurality of domains, the text modification module 2320 may select a plurality of text modification models corresponding to the plurality of domains to modify the text. In this case, the text modification module 2320 may obtain a modified text to be provided to the device 1000 from modified texts output from a plurality of text modification models. For example, when the text modification module 2320 generates a plurality of modified texts using a plurality of text modification models, the text modification module 2320 compares the reliability of the plurality of modified texts, and the reliability is The highly modified text may be determined as the modified text to be provided to the device 1000. The reliability of the corrected text may be a numerical value indicating the degree to which the corrected text matches the input voice, and may include, for example, a confidence score, but is not limited thereto.

Also, for example, when the text modification module 2320 generates a plurality of modified texts using a plurality of text modification models, the text modification module 2320 extracts modified portions from the plurality of modified texts. Then, the modified text to be provided to the device 1000 may be obtained using the extracted modified portions.

In addition, for example, the text modification module 2320 extracts some texts from modified texts output from a plurality of text modification models, and combines the extracted texts, thereby providing the modified text to be provided to the device 1000. Can be obtained. For example, the text modification module 2320 generates a first modified text and a second modified text using a plurality of text modification models, and the reliability of a part of the first modified text and the reliability of a part of the second modified text In a high case, the text correction module 2320 may obtain the corrected text to be provided to the device 1000 by combining a part of the first corrected text and a part of the second corrected text.

In addition, for example, when a domain related to each text of each section is selected, the text modification module 2320 may provide the text of each section to each corresponding domain modification model. In this case, the text correction module 2320 may obtain the corrected text to be provided to the device 1000 by combining the corrected texts of each section output from each domain correction model.

The corrected text output from the NLU module 2330 text correction module 2320 may be analyzed. The NLU module 2330 may include a plurality of NLU models of a plurality of domains, such as a first NLU model 2331 and a second NLU model 2332. The result value generated by the NLU module 233 interpreting the text may include, for example, an intent and a parameter. The intent is information determined by interpreting the text using the NLU model, and may indicate, for example, the user's speech intention. The intent may include not only information indicating the user's speech intention (hereinafter, intention information), but also a numerical value corresponding to information indicating the user's intention. Numerical values may indicate the probability that the text will be associated with information indicating a specific intent. As a result of analyzing the text using the NLU model, when a plurality of pieces of information indicating the intention of the user are obtained, intention information having a maximum numerical value corresponding to each intention information may be determined as the intent. Also, the parameter may indicate detailed information related to the intent. The parameter is information related to the intent, and a plurality of types of parameters may correspond to one intent.

In addition, the result value generated by the NLU module 233 interpreting the text may be used to provide a predetermined voice assistant service to the device 1000.

The speech analysis management module 2340 may evaluate the corrected text by the text correction module 2320 and determine whether to perform NLU processing on the corrected text. The speech analysis management module 2340 may include a speech recognition evaluation module 2341 and an NLU determination module 2342.

The speech recognition evaluation module 2341 may calculate the reliability of the text modified by the text correction module 2320. Reliability of the revised text may be a numerical value representing the probability that the revised text matches the input voice, and may include, for example, a confidence score, but is not limited thereto. In addition, the speech recognition evaluation module 2341 may calculate the domain reliability of the modified text. The speech recognition evaluation module 2341 may calculate a domain reliability indicating how relevant the modified text is to a domain previously registered in the server 2000 for NLU processing.

The NLU determination module 2342 may determine whether to perform NLU processing in the server 2000 for the modified text. Based on the reliability of the modified text and the domain reliability of the modified text, the server 2000 may determine whether to perform NLU processing. The NLU determination module 2342 may determine whether the domain related to the modified text is a domain capable of NLU processing by the device 1000 or a domain capable of NLU processing by the server 2000.

18 is a block diagram of a device according to an embodiment of the present disclosure.

Referring to FIG. 18, a device 1000 according to an embodiment of the present disclosure includes a communication interface 1100, an input/output unit 1200, a processor 1300, and a memory 1400, and the memory 1400 is at least One ASR model 1410, at least one NLU model 1420, a speech recognition evaluation module 1430, a domain identification module 1440, and an NLU determination module 1450 may be included.

The communication interface 1100 may include one or more components for communication with the server 2000 and an external device (not shown). The communication interface 1100 may transmit/receive information for voice recognition and voice assistant service with the server 2000 and an external device (not shown). The communication interface 1100 may include, for example, a local area network (LAN), a wide area network (WAN), a value added network (VAN), a mobile radio communication network, and Communication may be performed through a satellite communication network and a combination thereof, but is not limited thereto.

The input/output unit 1200 may receive data input to the device 1000 and may output data from the device 1000. The input/output unit 1200 may include a user input unit, a camera, a microphone, a display unit, and an audio output unit. The user input unit is, for example, a key pad, a dome switch, a touch pad (contact type capacitance method, pressure type resistive film method, infrared detection method, surface ultrasonic conduction method, integral tension measurement) Method, piezo effect method, etc.), a jog wheel, a jog switch, and the like, but are not limited thereto.

The display unit may display and output information processed by the device 1000. For example, the display unit may display a GUI for voice assistant service. When the display unit and the touch pad are layered to form a touch screen, the display unit may be used as an input device in addition to an output device. The display unit is a liquid crystal display, a thin film transistor-liquid crystal display, an organic light-emitting diode, a flexible display, a 3D display, It may include at least one of an electrophoretic display.

The sound output unit may output audio data, and may include, for example, a speaker, a buzzer, and the like.

The camera may obtain an image frame such as a still image or a video through an image sensor in a video call mode or a photographing mode. The image captured through the image sensor may be processed through the processor 1300 or a separate image processing unit (not shown).

The microphone can receive the user's speech and process it as electrical voice data.

The processor 1300 controls the overall operation of the device 1000. The processor 1300 may generally control operations of the device 1000 in the present specification by executing programs stored in the memory 1400.

The memory 1400 may store a program for processing and control of the processor 1300 and may store data input to the device 1000 or output from the device 1000. The memory 1400 is, for example, a flash memory type, a hard disk type, a multimedia card micro type, and a card type memory (eg, SD or XD Memory, etc.), RAM (RAM, Random Access Memory) SRAM (Static Random Access Memory), ROM (ROM, Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), The storage medium may include at least one type of magnetic memory, magnetic disk, and optical disk, but is not limited thereto.

Programs stored in the memory 1400 can be classified into a plurality of modules according to their functions. For example, an ASR model 1410, an NLU model 1420, a speech recognition evaluation module 1430, and a domain identification module ( 1440) and the NLU determination module 1450.

The ASR model 1410 may obtain text from a feature vector generated from a user's voice input. The processor 1300 of the device 1000 may input a feature vector into the ASR model 1410 to recognize the user's voice. When a plurality of ASR models 1410 are included in the device 1000, the processor 1300 of the device 1000 selects one of the plurality of ASR models 1410, and selects a feature vector to the selected ASR model 1410. It can be converted to a suitable format. The ASR model 1410 may be, for example, an artificial intelligence model including an acoustic model, a pronunciation dictionary, and a language model. Alternatively, the ASR model 1410 may be, for example, an end-to-end speech recognition model having a structure including an integrated neural network without separately including an acoustic model, a pronunciation dictionary, and a language model.

The NLU model 1420 may interpret text output from the ASR model 1410. Alternatively, the modified text provided from the server 2000 may be analyzed. A result value generated by the NLU model 1420 interpreting text or modified text may be used to provide a predetermined voice assistant service to a user.

The speech recognition evaluation module 1430 may acquire the reliability of the text output from the ASR model 1410. The reliability of the text may be a numerical value indicating the degree to which the text output from the ASR model 1410 matches the input voice, and may include, for example, a confidence score, but is not limited thereto. Also, the reliability of the text may be related to the probability that the text will match the input speech. For example, the reliability of the text is determined by at least one of the likelihood of a plurality of estimated texts output from the ASR model 1410 of the device 1000 and the posterior probabilities that at least one character in the text is replaced with another character. It can be calculated on the basis of. For example, the speech recognition evaluation module 1430 may calculate a reliability based on a likelihood output as a result of Viterbi decoding. Alternatively, for example, the speech recognition evaluation module 1430 may calculate the reliability based on posterior probabilities output from the softmax layer in the end-to-end ASR model. Alternatively, for example, the speech recognition evaluation module 1430 determines a plurality of estimated texts estimated in the speech recognition process of the ASR model 1410 of the device 1000, and determines a correlation of characters in the plurality of estimated texts. Based on this, the reliability of the text can be calculated.

Also, the speech recognition evaluation module 1430 may determine whether to transmit the text output from the ASR model 1430 to the server 2000. The speech recognition evaluation module 1430 may determine whether to transmit the text to the server 2000 by comparing the reliability of the text with a preset threshold. The speech recognition evaluation module 1430 may determine not to transmit the text to the server 2000 when the reliability of the text is greater than or equal to a preset threshold. Also, the speech recognition evaluation module 1430 may determine to transmit the text to the server 2000 when the reliability of the text is less than a preset threshold.

In addition, the speech recognition evaluation module 1430 may determine whether to transmit the text to the server 2000 based on at least one text with high reliability among a plurality of estimated texts estimated in the speech recognition process of the ASR model 1410. have. For example, a plurality of estimated texts estimated in the speech recognition process of the ASR model 1410 include a first estimated text with high reliability and a second estimated text with high reliability, and the reliability and second estimation of the first estimated text When the difference in the reliability of the text is less than or equal to a predetermined threshold, the speech recognition evaluation module 1430 may determine to transmit the text to the server 2000.

The domain identification module 1440 may identify a domain related to the text output from the ASR model 1410. The domain identification module 1440 may identify a domain related to the text based on the domain reliability of the text output from the ASR model 1410. For example, the domain identification module 1440 may calculate a confidence score indicating how relevant the text output from the ASR model 1410 is to a pre-registered domain. The domain identification module 1440 is an artificial intelligence model trained for domain identification and may output domain reliability by using text as an input value. Also, for example, the domain identification module 1440 may be a part of the NLU model or may be a model separate from the NLU model. Alternatively, the domain identification module 1440 may identify a domain related to text based on a rule.

Domains for speech recognition in the speech recognition system may be hierarchically set, and the domains identified by the domain identification module 1440 of the device 1000 are identified by the domain identification module 2312 of the server 2000. It may be a higher layer domain than the identified domain.

The NLU determination module 1450 may determine whether to perform NLU processing in the device 1000 or NLU processing in the server 2000 for text output from the ASR model 1410. The NLU determination module 1450 may determine whether the domain related to the text output from the ASR model 1410 is a domain in which the device 1000 can perform NLU processing. When the domain related to the text output from the ASR model 1410 is a domain previously registered in the device 1000, the NLU determination module 1450 may determine that the device 1000 performs NLU processing. In addition, when the domain related to the text output from the ASR model 1410 is not a domain previously registered in the device 1000, the NLU determination module 1450 may determine that the device 1000 does not perform NLU processing. have.

Functions related to artificial intelligence according to the present disclosure are operated through a processor and a memory. The processor may be composed of one or a plurality of processors. In this case, one or more processors may be a general-purpose processor such as a CPU, AP, or Digital Signal Processor (DSP), a graphics-only processor such as a GPU, a Vision Processing Unit (VPU), or an artificial intelligence-only processor such as an NPU. One or more processors control to process input data according to a predefined operation rule or an artificial intelligence model stored in the memory. Alternatively, when one or more processors are dedicated artificial intelligence processors, the dedicated artificial intelligence processor may be designed with a hardware structure specialized for processing a specific artificial intelligence model.

A predefined motion rule or an artificial intelligence model is characterized by being created through learning. Here, to be made through learning means that a basic artificial intelligence model is learned using a plurality of learning data by a learning algorithm, so that a predefined motion rule or an artificial intelligence model set to perform a desired characteristic (or purpose) is created. Means Jim. Such learning may be performed in the device itself on which the artificial intelligence according to the present disclosure is performed, or may be performed through a separate server and/or system. Examples of the learning algorithm include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but are not limited to the above-described examples.

The artificial intelligence model may be composed of a plurality of neural network layers. Each of the plurality of neural network layers has a plurality of weight values, and a neural network operation is performed through an operation between the operation result of a previous layer and a plurality of weights. The plurality of weights of the plurality of neural network layers can be optimized by the learning result of the artificial intelligence model. For example, a plurality of weights may be updated to reduce or minimize a loss value or a cost value acquired from an artificial intelligence model during a learning process. The artificial neural network may include a deep neural network (DNN), for example, CNN (Convolutional Neural Network), DNN (Deep Neural Network), RNN (Recurrent Neural Network), RBM (Restricted Boltzmann Machine), DBN (Deep Belief Network), BRDNN (Bidirectional Recurrent Deep Neural Network), or deep Q-Networks (Deep Q-Networks), and the like, but is not limited to the above-described example.

An embodiment of the present disclosure may also be implemented in the form of a recording medium including instructions executable by a computer, such as a program module executed by a computer. Computer-readable media can be any available media that can be accessed by a computer, and includes both volatile and nonvolatile media, removable and non-removable media. Further, the computer-readable media may include computer storage media and communication media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Communication media may typically contain computer readable instructions, data structures, or other data in a modulated data signal such as a program module.

In addition, in the present specification, the “unit” may be a hardware component such as a processor or a circuit, and/or a software component executed by a hardware configuration such as a processor.

In addition, in the present specification, “including at least one of a, b, or c” means “including only a, only b, only c, including a and b, or including b and c, It may mean including a and c, or including all of a, b and c.

The above description of the present disclosure is for illustrative purposes only, and those of ordinary skill in the art to which the present disclosure pertains will be able to understand that it is possible to easily transform it into other specific forms without changing the technical spirit or essential features of the present disclosure. will be. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not limiting. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as being distributed may also be implemented in a combined form.

The scope of the present disclosure is indicated by the claims to be described later rather than the detailed description, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present disclosure. do.

Claims (20)

In a method for a server to modify a speech recognition result provided from a device,
Receiving a text output from the device's Automatic Speech Recognition (ASR) model from the device;
Identifying at least one domain related to the received text;
Selecting at least one text correction model corresponding to the identified at least one domain from among a plurality of text correction models included in the server;
Modifying the received text using the selected at least one text modification model; And
Providing the modified text to the device;
Including,
The text correction model is an artificial intelligence model that receives the received text and outputs the corrected text for speech recognition.
The method of claim 1,
The text correction model is an artificial intelligence model trained using a text output from an ASR model and a ground truth text of a preset domain.
The method of claim 2,
The text correction model is an artificial intelligence model trained using texts output from a plurality of types of ASR models.
The method of claim 1,
The plurality of text correction models respectively correspond to a plurality of domains,
The operation of selecting the at least one text modification model comprises selecting the text modification model corresponding to the identified domain from among the plurality of text modification models corresponding to the plurality of domains.
The method of claim 1,
The operation of receiving the text includes receiving a text stream output from the ASR model,
The operation of identifying the domain,
By accumulating and using the text stream in units of a predetermined section, a domain related to the received text stream is identified.
The method of claim 5,
The operation of identifying the domain,
And identifying a domain related to the received text stream by calculating the domain reliability of the text stream accumulated in units of the predetermined interval.
The method of claim 1,
In the operation of identifying the domain, classifying the text into a plurality of sections, identifying a plurality of domains related to each of the plurality of sections,
The operation of selecting the text correction model includes selecting a plurality of text correction models corresponding to the identified plurality of domains,
The operation of modifying the text includes modifying the text of the plurality of sections, respectively, using the selected plurality of text modification models.
The method of claim 1,
Receiving from the device a first domain reliability for the text calculated by the device; And
Calculating a second domain reliability for the received text;
It further includes,
The operation of identifying the at least one domain,
And identifying a domain related to the received text based on the received first domain reliability and the calculated second domain reliability.
The method of claim 8,
Selecting one of a plurality of domain identification modules in the server based on the first domain reliability;
It further includes,
The operation of calculating the second domain reliability,
And calculating the second domain reliability for the text by using the selected domain identification module.
The method of claim 1,
The ASR model of the device is an end-to-end ASR model,
The text modification model of the server is a sequence-to-sequence model for speech recognition.
In the server for modifying the speech recognition result provided from the device,
Communication interface;
A storage unit for storing a program including one or more instructions; And
A processor that executes one or more instructions of a program stored in the storage unit; Including,
The processor,
Receive text output from the device's Automatic Speech Recognition (ASR) model from the device, identify at least one domain related to the received text, and among a plurality of text correction models included in the server, the identification Selecting at least one text correction model corresponding to the at least one selected domain, correcting the received text using the selected at least one text correction model, and providing the modified text to the device,
The text modification model is an artificial intelligence model that receives the received text and outputs the modified text for speech recognition.
The method of claim 11,
The text modification model is an artificial intelligence model trained using text output from the ASR model and ground truth text of a preset domain.
The method of claim 12,
The text modification model is an artificial intelligence model trained using texts output from a plurality of types of ASR models.
The method of claim 11,
The plurality of text correction models respectively correspond to a plurality of domains,
Wherein the processor executes the one or more instructions and selects the text correction model corresponding to the identified domain from among the plurality of text correction models corresponding to the plurality of domains.
The method of claim 11,
The processor executes the one or more instructions, receives the text stream output from the ASR model, and identifies a domain related to the received text stream by accumulating and using the text stream in units of a predetermined section, server.
The method of claim 15,
The processor is to identify a domain related to the received text stream by executing the one or more instructions and calculating a domain reliability of the text stream accumulated in units of the predetermined interval.
The method of claim 11,
The processor executes the one or more instructions to classify the text into a plurality of sections, identify a plurality of domains related to each of the plurality of sections, and modify a plurality of texts corresponding to the identified plurality of domains Selecting a model, and using the selected plurality of text correction models, to modify the text of the plurality of sections, respectively.
The method of claim 11,
The processor executes the one or more instructions to receive a first domain reliability for the text calculated by the device from the device, calculate a second domain reliability for the received text, and calculate the received text. To identify a domain related to the received text based on the first domain reliability and the calculated second domain reliability.
The method of claim 18,
The processor executes the one or more instructions to select one of a plurality of domain identification modules in the server based on the first domain reliability, and the second domain for the text using the selected domain identification module. Server, which is to calculate the reliability.
The method of claim 11,
The ASR model of the device is an end-to-end ASR model,
The server's text correction model is a sequence-to-sequence model for speech recognition.

Images