KR20210001937A - The device for recognizing the user's speech input and the method for operating the same - Google Patents

Abstract

Disclosed are a device recognizing voice input of a user including a named entity, and an operating method thereof. According to one embodiment of the present invention, the device creates a weighted finite state transducer model by using a vocabulary list including a plurality of named entities; obtains a first character string from voice input received from a user by using a first decoding model; obtains a second character string including a word sequence corresponding to at least one named entity and an unrecognized word sequence which is not identified as a named entity by using a second decoding model using the weighted finite state transducer model; and replaces the unrecognized word sequence of the second character string with a word sequence included in the first character string, thereby outputting a text corresponding to the voice input.

Description

A device that recognizes the user's voice input and its operation method {THE DEVICE FOR RECOGNIZING THE USER'S SPEECH INPUT AND THE METHOD FOR OPERATING THE SAME}

The present disclosure relates to a device for recognizing a voice input received from a user using an artificial intelligence model and a method of operating the same.

The voice recognition function is a function of easily controlling a device by recognizing a user's voice input without operating a separate button or touching a touch module. In recent years, as artificial intelligence (AI) technology develops, artificial intelligence technology is also applied to the speech recognition function, enabling fast and accurate speech recognition for various utterances.

As a method of recognizing a user's voice input using artificial intelligence technology, a voice signal, which is an analog signal, is received through a microphone, and the voice part is converted into a computer-readable text using an ASR (Automatic Speech Recognition) model. can do. The ASR model may be an artificial intelligence model. The artificial intelligence model can be processed by an artificial intelligence dedicated processor designed with a hardware structure specialized for processing the artificial intelligence model. Artificial intelligence models can be created through learning. Here, to be made through learning means that a basic artificial intelligence model is learned using a plurality of learning data by a learning algorithm, so that a predefined motion rule or an artificial intelligence model set to perform a desired characteristic (or purpose) is created. Means Jim. The artificial intelligence model may be composed of a plurality of neural network layers. Each of the plurality of neural network layers has a plurality of weight values, and a neural network operation is performed through an operation between the operation result of a previous layer and a plurality of weights.

The artificial intelligence model for using the speech recognition function is created through learning, and in particular, in the case of a named entity, it is necessary to learn using a number of pattern sentences such as sentences combined with the entity name, so a large amount of data computation is required. , It takes a lot of time to learn. In particular, in recent years, the device side uses an on-device voice recognition function that performs voice recognition such as ASR, but the amount of data computation is too large to learn pattern sentences including entity names in an on-device environment. This takes a long time. In addition, when learning pattern sentences about entity names in an on-device environment, all pattern sentences that combine the number of entity names and related statements, etc., must be learned. Speech recognition through artificial intelligence models generated on-device is less accurate. There is a problem.

An object of the present disclosure is to provide a speech recognition method and device capable of improving the recognition accuracy of a speech input including an entity name without learning a pattern sentence.

In order to solve the above-described technical problem, an embodiment of the present disclosure uses a vocabulary list including a plurality of named entities, and subwords extracted from each of the plurality of entity names ( generating a weighted finite state transducer model by training the probability that subword) can be predicted as a word or word column representing an entity name, receiving a user's voice input, the first Using a decoding model, a feature vector including a probability value that the received speech input can be predicted as a specific subword is obtained, and a first including a plurality of prediction strings using the probability value of the feature vector 1 obtaining a character string, inputting the obtained feature vector to a second decoding model using the weighted finite state converter model, and at least one of the plurality of entity names from the feature vector using the second decoding model Obtaining a second character string including a word sequence corresponding to an entity name and an unrecognized word sequence not identified by the at least one entity name, and the unrecognized word sequence among the second character strings. It provides a method for a device to recognize a voice input, including the step of outputting a text corresponding to the voice input by replacing it with a word string included in the first character string.

For example, the step of generating the weighted finite state converter model may include obtaining a vocabulary list including a plurality of entity names, and subwording words or strings constituting the plurality of entity names into phoneme or syllable units. By learning the weight through the step of segmentation and state transition using the frequency of the subword and the order of the arrangement of the subwords, the subword is specified among the plurality of entity names. It may include the step of obtaining a confidence score including a posterior probability that can be predicted by the name of the entity.

For example, the step of generating the weighted finite state converter model may include performing filtering to remove entity names that overlap with words previously stored in the memory of the device among a plurality of entity names included in the obtained vocabulary list. It may include.

For example, the weighted finite state transformer model includes a Lexicon Finite State Transducer (L FST) including mapping information that is a probability value that the subword can be predicted as a specific word, and the specific word or word sequence. When input, includes a Grammar Finite State Transducer (G FST) including weight information for predicting a word sequence that can be arranged after the input word or word sequence, and the weighted finite state The transducer model may be generated through the synthesis of the Lexicon finite state transducer and the Grammer finite state transducer.

For example, the first decoding model may be an end-to-end automatic speech recognition (ASR) model.

For example, generating the weighted finite state converter model includes classifying a plurality of entity names according to a plurality of different domains, and using the classified entity names, the plurality of domains It may include generating a plurality of weighted finite state converter models for each.

For example, the method includes the steps of identifying words corresponding to an entity name from an application executed through the device or a web page provided through the device, and generating the identified words for each of the plurality of domains. The method may further include determining a domain in which the currently running application or the web page can be classified by comparing it with a plurality of entity names included in the vocabulary list of each of the weighted finite state converter models.

For example, the method comprises the steps of receiving update information of a vocabulary list including at least one of addition of a new entity name, deletion of entity name, and change of entity name from a server, and using the update information, the vocabulary list is The step of updating may further include generating the weighted finite state transformer model, wherein the weighted finite state transformer model may be generated through learning using the updated vocabulary list.

For example, the method includes recognizing that the device enters a new area by acquiring location information of the device, transmitting information about entering the new area of the device to a server of an application service provider, and the Receiving a point of interest (POI) vocabulary list including an entity name of at least one of the new area name, attraction, tourist attraction, and famous restaurant from the server of the application service provider, In the step of generating the weighted finite state transformer model, the weighted finite state transformer model may be generated through learning using an entity name included in the received place of interest vocabulary list.

For example, the step of generating the weighted finite state converter model reflects the user's characteristics from at least one of an application frequently executed through the device, log data of a messenger application, and a search term record in a content streaming application. The weighted finite state transformer model may be generated through learning using a personalized vocabulary list including a plurality of entity names.

In order to solve the above technical problem, an embodiment of the present disclosure includes a voice input unit for receiving a voice input from a user, a memory storing a program including one or more instructions, and a program stored in the memory. And a processor that executes one or more instructions, wherein the processor includes a subword extracted from each of the plurality of entity names using a vocabulary list including a plurality of named entities A weighted finite state transducer model is generated by training a probability that can be predicted as a word or word string representing an entity name, receiving the speech input from the speech input unit, and first decoding A first character string including a plurality of predicted character strings using a model to obtain a feature vector including a probability value that the received speech input can be predicted as a specific subword, and using the probability value of the feature vector And inputting the obtained feature vector to a second decoding model using the weighted finite state converter, and corresponding to at least one entity name among the plurality of entity names from the feature vector using the second decoding model. A second character string including an entity name word sequence and an unrecognized word sequence not identified by the at least one entity name is obtained, and the unrecognized word sequence among the second character strings is transferred to the first character string. A device is provided that obtains text corresponding to the voice input by replacing it with an included word string.

For example, the processor obtains a vocabulary list including the plurality of entity names, and divides a word or string constituting the plurality of entity names into subwords that are phoneme or syllable units, and A posterior probability value at which the subword can be predicted as a specific entity name among the plurality of entity names by learning a weight through a state transition using the frequency of the subwords and the arrangement order of the subwords The weighted finite state converter model may be generated by obtaining a confidence score including a posterior probability.

For example, the processor may perform filtering to remove an entity name overlapping with a word previously stored in a memory of the device among a plurality of entity names included in the acquired vocabulary list.

For example, the weighted finite state transformer model includes a Lexicon Finite State Transducer (L FST) including mapping information that is a probability value that the subword can be predicted as a specific word, and the specific word or word sequence. When input, includes a Grammar Finite State Transducer (G FST) including weight information for predicting a word sequence that can be arranged after the input word or word sequence, and the weighted finite state The transducer model may be generated through the synthesis of the Lexicon finite state transducer and the Grammer finite state transducer.

For example, the first decoding model may be an end-to-end automatic speech recognition (ASR) model.

For example, the processor classifies the plurality of entity names according to a plurality of different domains, and uses the classified entity names to generate a plurality of weighted finite state converter models for each of the plurality of domains. Can be generated.

For example, the processor identifies words corresponding to an entity name from an application being executed or a web page being accessed, and uses the identified words for each of the plurality of weighted finite state converter models generated for each of the plurality of domains. By comparing the names of a plurality of entities included in the vocabulary list, it is possible to determine a domain in which the currently running application or the web page can be classified.

For example, the device further includes a communication interface for transmitting and receiving data with the server, and the processor further includes at least one of adding a new entity name, deleting an entity name, and changing the entity name from the server using the communication interface. The weighted finite state converter model may be generated by receiving update information of a vocabulary list including, updating the vocabulary list using the update information, and learning using the updated vocabulary list.

For example, the device further includes a location sensor for acquiring location information and a communication interface for transmitting and receiving data with a voice assistant server or an external server, and the processor uses the location sensor to allow the device to enter a new area. And in response to recognizing the entry into the new area, the entity name for at least one of the place name, attraction, tourist destination, and famous restaurant of the new area from the server of the application service provider through the communication interface The weighted finite state converter model may be generated through learning using an entity name included in the received point of interest vocabulary list and receiving a POI vocabulary list.

For example, the processor includes a plurality of entity names reflecting the characteristics of a user from at least one of an application frequently executed through the device, log data of a messenger application, and a search term record in a content streaming application. The weighted finite state transformer model may be generated through learning using a personalized vocabulary list.

In order to solve the above technical problem, another embodiment of the present disclosure provides a computer-readable recording medium in which a program for execution on a computer is recorded.

1 is a diagram illustrating an operation in which a device recognizes a user's voice input according to an embodiment of the present disclosure.
2 is a diagram illustrating an operation of recognizing a user's voice input by a device and a server according to an embodiment of the present disclosure.
3 is a block diagram showing components of a device according to an embodiment of the present disclosure.
4 is a block diagram illustrating components of a server according to an embodiment of the present disclosure.
5 is a flowchart illustrating a method for a device to recognize a voice input according to an embodiment of the present disclosure.
6 is a flowchart illustrating an embodiment in which the device of the present disclosure generates a weighted finite state converter model for an entity name.
7 is a diagram illustrating an embodiment in which a device of the present disclosure automatically selects a domain using a weighted finite state converter model.
8 is a flowchart illustrating an embodiment in which a device of the present disclosure automatically selects a domain using a weighted finite state converter model.
9 is a flowchart illustrating an embodiment in which the device of the present disclosure generates a weighted finite state converter model for an entity name using information received from a server.
FIG. 10 is a conceptual diagram illustrating an embodiment of generating a weighted finite state converter model using a place of interest vocabulary list related to a new area when a device of the present disclosure enters a new area.
11 is a conceptual diagram illustrating an embodiment of generating a weighted finite state converter model by using a place of interest vocabulary list for a new area when a device of the present disclosure enters a new area.
12 is a diagram illustrating an embodiment in which the device of the present disclosure generates a personalized weighted finite state converter model using an entity name reflecting a user's personal characteristics.

The terms used in the embodiments of the present specification have selected general terms that are currently widely used as possible while taking the functions of the present disclosure into consideration, but this may vary depending on the intention or precedent of a technician working in the field, the emergence of new technologies, etc. . In addition, in certain cases, there are terms arbitrarily selected by the applicant, and in this case, the meaning of the terms will be described in detail in the description of the corresponding embodiment. Therefore, the terms used in the present specification should be defined based on the meaning of the term and the contents of the present disclosure, not the name of a simple term.

Singular expressions may include plural expressions unless the context clearly indicates otherwise. Terms used herein, including technical or scientific terms, may have the same meaning as commonly understood by a person of ordinary skill in the technical field described herein.

When a certain part "includes" a certain component throughout the present disclosure, it means that other components may be further included rather than excluding other components unless specifically stated to the contrary. In addition, terms such as "... unit" and "... module" described in this specification mean a unit that processes at least one function or operation, which is implemented as hardware or software, or is a combination of hardware and software. Can be implemented.

The expression "configured to (configured to)" used in this specification is, for example, "suitable for", "having the capacity to" depending on the situation. It can be used interchangeably with ", "designed to", "adapted to", "made to", or "capable of". The term "configured to (or set)" may not necessarily mean only "specifically designed to" in hardware. Instead, in some situations, the expression "a system configured to" may mean that the system "can" along with other devices or parts. For example, the phrase “a processor configured (or configured) to perform A, B, and C” means a dedicated processor (eg, an embedded processor) for performing the operation, or by executing one or more software programs stored in memory, It may mean a generic-purpose processor (eg, a CPU or an application processor) capable of performing corresponding operations.

In the present disclosure,'letter' means a symbol used to express and write human language in a visible form. For example, characters may include Korean, alphabet, Chinese characters, numbers, pronunciation marks, punctuation marks, and other symbols.

In the present disclosure, the term'character string' means a sequence of characters.

In the present disclosure, the'grapheme' is the smallest unit representing sound, which is composed of at least one character. For example, in the case of the alphabet notation system, one character may be a character element, and a character string may mean an arrangement of character elements.

In the present disclosure,'text' may include at least one letter element. For example, the text may include morphemes or words.

In the present disclosure, a'word' is a basic unit of a language composed of at least one character string, used independently, or representing a grammatical function.

In the present disclosure, a'word sequence' refers to an arrangement of one or more words.

In the present disclosure, a'sub word' is a basic unit constituting a word, and means, for example, a phoneme or a syllable. As a technique used to model the subword, the Hidden Marcov model (HMM) method is mainly used, which is to calculate the probability distribution after extracting feature vectors by collecting speech signals corresponding to each subword unit. to be.

In the present disclosure, a'label' is an arbitrary subword representing a phoneme or syllable. The label can be output by an end-to-end ASR model (End-to-End Automatic Speech Recognition).

1 is a diagram illustrating an operation of recognizing a user's voice input by a device 1000a according to an embodiment of the present disclosure.

Referring to FIG. 1, a device 1000a includes a WFST model generation module 1330, a WFST model 1340, a speech recognition module 1350, a deep neural network 1360, a communication interface 1400, and an output unit 1500. It may include. In FIG. 1, only essential components for describing the operation of the device 1000a are illustrated. The configuration included in the device 1000a is not limited as illustrated in FIG. 1.

The device 1000a may generate a WFST model (Weighted Finite State Transducer) 1340 for a Named Entity by using instructions or program codes related to the WFST model generation module 1330. 'Object name' means a word or word string having a unique meaning, such as a person's name, company name, place name, area name, time, or date. The WFST model generation module 1330 may obtain a vocabulary list including a plurality of entity names. In one embodiment, the WFST model generation module 1330 receives a user input, receives from an external server, or crawls an application, web page, etc. that are executed by the device 1000a to name a plurality of entities. It is possible to obtain a vocabulary list including. In the embodiment shown in FIG. 1, the entity name vocabulary list may include a plurality of entity names related to singer names such as Chris Brown, August Burns Red, Cardi-B, Cody Jinks, and Road Trip.

The WFST model generation module 1330 may include a text preprocessing module 1332, a filtering module 1334, and a probability model generation module 1336.

The text preprocessing module 1332 is configured to receive a plurality of entity names included in a vocabulary list, and to output a sub word by performing preprocessing on the inputted plurality of entity names. In an embodiment, the text preprocessing module 1332 may segment a plurality of entity names into sub-word units. The text preprocessing module 1332 may tokenize a word or a word string included in a plurality of entity names in units of subwords using, for example, a byte pair encoding (BPE) algorithm. In an embodiment, the text preprocessing module 1332 may perform preprocessing of removing punctuation marks, special characters, and special symbols, and removing stop words.

The filtering module 1334 removes the vocabulary previously stored in the dictionary DB 1320 (refer to FIG. 3) of the device 1000a among the subwords output from the text preprocessing module 1332 to perform filtering. Is composed. In an embodiment, the filtering module 1334 may remove a word or word string identical to a vocabulary previously stored in the dictionary DB 1320 from among a plurality of entity names included in the entity name vocabulary list. The filtering module 1334 may output only sub-words extracted from out of vocabulary (OOV) words corresponding to entity names not included in the dictionary DB 1320.

Probability model generation module 1336 is a word or word string representing at least one entity name among a plurality of entity names included in the entity name vocabulary list by subwords output through the text preprocessing module 1332 and the filtering module 1334. It is configured to train a predictable probability and generate a WFST model 1340 as a result of the training. In one embodiment, the probability model generation module 1336 includes a finite state transducer that encodes a mapping using each of the subwords extracted from each of the plurality of entity names as inputs and the entity names as outputs. A WFST model 1340 including it may be generated.

The WFST model 1340 is a language model that outputs a word or word sequence when a sub word is input based on a probability that the input sub word can be predicted as a word or word sequence representing an entity name. The WFST model 1340 may include a finite state transducer. The finite state transducer performs a state transition based on a preset rule with respect to the input subword, and may determine a subword that may be listed after the input subword according to the state conversion result. A finite state transducer is a finite automaton in which state transitions are labeled by input and output subwords. The finite state transducer can be represented as a graph composed of a node and an arc connecting the nodes. Nodes represent state and arcs represent state transitions. Each arc is given an input subword and an output subword. The WFST model 1340 adds weight to each arc. The concept of probability can be expressed by this weight. A hypothesis is generated by following each arc at the root node, and the probability of occurrence of a hypothesis can be calculated by multiplying the weight assigned to the arc. The WFST model 1340 may output a word or word string from a specific subword based on the probability of occurrence.

The device 1000a may acquire text corresponding to a voice input received from a user and output the text by using a command or program code related to the voice recognition module 1350. The speech recognition module 1350 may include a first decoding model 1352, a second decoding model 1354, and a recognition result integration module 1356.

The first decoding model 1352 receives a voice signal received from a user, obtains a feature vector, which is a vector of probability values that can predict a phoneme as a specific label according to the length of each phoneme of the voice signal, and Is configured to output a first character string based on the probability value of. Here,'label' is a subword representing a phoneme or syllable, and is a token defined by the pre-trained deep neural network 1360. In an embodiment, the first decoding model 1352 may perform speech recognition on a speech signal in an end-to-end automatic speech recognition (ASR) method. The end-to-end ASR scheme is a speech recognition scheme using a deep neural network 1360 trained to directly map a speech signal to a string or word string. Unlike other speech recognition methods that use multiple models such as acoustic models and language models, the end-to-end ASR method can simplify the speech recognition process by using one trained deep neural network 1360. As a lower embodiment of the end-to-end ASR model, there are, for example, a Recurrent Neural Network Transducer (RNN-T) model and an attention-based model.

In one embodiment, the first decoding model 1352 may use an end-to-end ASR model based on an attention-based model. The attention-based model may include, for example, Transformer or Listen-Attend-Spell (LAS).

The first decoding model 1352 may select a corresponding label from phonemes for each frame from feature vectors. In one embodiment, the first decoding model 1352 uses an end-to-end ASR model (softmax) that includes a probability value that a phoneme for each frame can match a specific label. Can be obtained. The first decoding model 1352 selects a label from phonemes for each frame using a probability value of Softmax, concatenates the selected labels, and expresses each phoneme by a corresponding label. candidates) can be obtained. The first decoding model 1352 may obtain a first character string using a posterior probability of a label candidate. In the embodiment shown in FIG. 1, the user uttered'Please search Cardi-B', and the first decoding model 1352 acquires a voice signal from the user's utterance, and a'Please search card' from the voice signal. The same first character string can be obtained. The first decoding model 1352 cannot accurately predict a character string corresponding to'Cardi-B' which is an Out of Vocabulary that is not stored in the dictionary DB 1320 (see FIG. 3).

In an embodiment, the first decoding model 1352 may convert a label-unit feature vector including a probability value that a phoneme for each frame may match a specific label into a sub-word feature vector. The first decoding model 1352 may output a feature vector in units of subwords obtained as a result of the conversion as the second decoding model 1354.

The second decoding model 1354 receives a subword-unit feature vector from the first decoding model 1352, and corresponds to at least one entity name among a plurality of entity names included in the entity name vocabulary list from the subword-unit feature vector. And a second character string including an unrecognized word string that is not identified by a word to be used and an entity name. The second decoding model 1354 may output a second character string predicted from a sub-word unit feature vector by using the WFST model 1340. The second decoding model 1354 calculates a confidence score based on the likelihood of a word for each subword, dictionary information, and a language model, selects a character string with a high confidence score, and outputs I can.

In one embodiment, the second decoding model 1354 may include a Lexicon FST 1354L and a Grammer FST 1354G. The second decoding model 1354 may output a word or word sequence from a feature vector for each sub-word by composing the Lexicon FST 1354L and the grammar FST 1354G. However, the present invention is not limited thereto, and the second decoding model 1354 may be configured as one integrated module that outputs a word corresponding to an entity name and an unrecognized word sequence from a feature vector for each subword.

The Lexicon FST 1354L is configured to receive a feature vector in units of subwords and output a word or word sequence predicted based on the feature vector in units of subwords. The Lexicon FST 1354L may include mapping information, which is a probability value at which a subword can be predicted as a specific word. In one embodiment, the Lexicon FST 1354L may convert the sub-word sequence s into P(s|W), which is a probability related to a word sequence W.

The grammar FST (1354G) is configured to receive a word or word string of a sub-word sequence from the Lexicon FST (1354L) and output a second string including a word string corresponding to an entity name and an unrecognized word string. . The grammar FST 1354G may be a model in which weights for predicting a word sequence that may be arranged after the input word or word sequence when a specific word or word sequence is input. The grammar FST 1354G may predict a word or word sequence that may appear after a specific word or word sequence using, for example, a recurrent neural network (RNN) or a statistical n-Gram model. In an embodiment, the grammar FST 1354G includes information on the language model probability P(W) as a weight for the word sequence W, and may output a word or word sequence to which the probability P(W) is added.

In the embodiment shown in FIG. 1, the second decoding model 1354 includes Cardi-B corresponding to the entity name from the sub-word unit feature vector output from the first decoding model 1352 using the WFST model 1340 and A second string including the unrecognized word may be obtained.

The recognition result integration module 1356 is configured to integrate the first character string obtained from the first decoding model 1352 and the second character string obtained from the second decoding model 1354. The recognition result integration module 1356 may output text corresponding to the voice input by replacing the unrecognized word string among the second strings with the word string included in the first string. In an embodiment, the recognition result integration module 1356 may replace the unrecognized word string included in the second string with a word string corresponding to a position of the unrecognized word string among the first strings. In the embodiment shown in Figure 1, the recognition result integration module (1356) is by replacing the second string unrecognized word column of each of a 'Please', 'search' contained a <unk>, <unk> to the first string, , "Please search Cardi-B", which is a text corresponding to the voice input, may be output.

The text output from the recognition result integration module 1356 may be provided to the communication interface 1400 or the output unit 1500.

In the conventional speech recognition technology, when creating a language model, for example, "<object name> + show me" or "Let's meet at <object name> + today" has text as a completed sentence. Learning had to be performed in consideration of the pattern sentence according to. In the case of conventional speech recognition, since the amount of data used for learning is large and the amount of calculation is large, processing speed is very slow, such as several hours required even in a server environment, and it is difficult to create a language model in an on-device environment. There was a problem. In addition, since it is necessary to learn all pattern sentences according to each entity name, there is a problem in that the recognition rate using the language model is lowered when the pattern does not exist.

The device 1000a according to an embodiment of the present disclosure generates a WFST model 1340 through learning using a vocabulary list of entity names that are words that are not included in the dictionary DB 1320 (see FIG. 3). And, among the user's voice input, a first decoding model 1352 using an end-to-end ASR method is used for general utterances other than an entity name, and a word corresponding to the entity name is decoded using a second decoding model 1354, respectively. 2 An unrecognized word sequence that cannot be converted through the decoding model 1354 may be replaced with a first character string obtained through the first decoding model 1352 to output text corresponding to the voice input. The device 1000a of the present disclosure generates a WFST model 1340 including only a plurality of entity names, and outputs a word or word string corresponding to the entity name using the WFST model 1340, thereby The processing time used for decoding the input to text can be significantly reduced. In the device 1000a according to the embodiment of the present disclosure, since both the outputs from the first decoding model 1352 and the second decoding model 1354 are words, the operation of separately aligning the position of the pattern sentence with respect to the entity name ( There is also an advantage of being able to omit the alignment).

In addition, since the device 1000a according to an embodiment of the present disclosure can recognize speech without pattern sentences for each of a plurality of entity names, the amount of data calculation is reduced, and therefore, a device having a relatively low computing power compared to the server ( 1000a) can also perform speech recognition. The device 1000a according to an embodiment of the present disclosure recognizes a voice input regarding an entity name using the WFST model 1340 and converts it into words, thereby improving accuracy of voice recognition.

FIG. 2 is a diagram illustrating an operation in which the device 1000b and the server 2000 recognize a user's voice input according to an embodiment of the present disclosure.

Referring to FIG. 2, the device 1000b may include a speech recognition module 1350, a deep neural network 1360, a communication interface 1400, and an output unit 1500. The server 2000 may include a communication interface 2100 and a WFST decoder 2310. In FIG. 2, only essential components for describing the operation of the device 1000b and the server 2000 are illustrated. The configurations included in the device 1000b and the server 2000 are not limited as illustrated in FIG. 2.

The device 1000b may acquire text corresponding to the voice input received from the user and output the text by using a command or program code related to the voice recognition module 1350. The speech recognition module 1350 may include a decoding model 1352 and a recognition result integration module 1356.

The decoding model 1352 receives a voice signal received from a user, obtains a feature vector, which is a vector of probability values that a phoneme can predict as a specific subword according to the length of each phoneme of the voice signal, and calculates the probability value of the feature vector. It is configured to output the first character string on the basis of. The decoding model 1352 shown in FIG. 2 is shown in FIG. 1 except that a feature vector in units of subwords is output to the communication interface 1400, and the first character string is output to the recognition result integration module 1356. It is the same as that of the first decoding model 1352, so redundant descriptions are omitted.

In one embodiment, the decoding model 1352 is a label having the highest probability value among the probability values of the feature vectors, that is, the likelihood that a phoneme in an arbitrary time domain among feature vectors in sub-word units can be predicted with a specific label. A lattice may be generated by connecting them to each other, and the generated lattice may be provided to the communication interface 1400.

The communication interface 1400 transmits the feature vector in units of subwords received from the decoding model 1352 to the server 2000. In an embodiment, the communication interface 1400 may transmit a feature vector in units of subwords to the server 2000 through a wired or wireless network.

In an embodiment, the communication interface 1400 may transmit the lattice received from the decoding model 1352 to the server 2000. Since the data capacity of the lattice is smaller than the data capacity of the feature vector of each subword, when the communication interface 1400 transmits the lattice to the server 2000, the transmission time is reduced compared to the case of transmitting the feature vector in units of subwords. And, it can prevent unexpected data loss.

The server 2000 may receive a feature vector in units of subwords from the device 1000b through the communication interface 2100. The server 2000 may obtain a second character string including an entity name and an unrecognized word from a subword unit feature vector using a command or program code related to the WFST decoder 2310.

The WFST decoder 2310 includes a word corresponding to at least one entity name among a plurality of entity names included in the entity name vocabulary list using the WFST model 2320 and an unrecognized word string not identified as entity name. It is configured to obtain a second character string. The WFST model 2320 is a model that outputs a word or word sequence when a sub word is input based on a probability that the sub word can be predicted as a word or word sequence representing a specific entity name. The WFST model 2320 is the same as the WFST model 1340 shown in FIG. 1, except that the WFST model 2320 is generated by the server 2000 and stored in the server 2000, and a duplicate description will be omitted. The WFST decoder 2310 may include a Lexicon FST 1354L and a grammar FST 1354G, similar to the second decoding model 1354 illustrated in FIG. 1. However, the present invention is not limited thereto, and the WFST decoder 2310 is a single integrated module that outputs a word or word sequence having a high likelihood based on a probability value of the feature vector when a feature vector in units of subwords is input. It can also be configured.

The WFST decoder 2310 performs the same operation or function as the second decoding model 1354 illustrated in FIG. 1, and a duplicate description will be omitted.

The WFST decoder 2310 may output a second character string including an entity name and an unrecognized word to the communication interface 2100.

The server 2000 may transmit a second character string including an entity name and an unrecognized word to the device 1000 using the communication interface 2100.

The device 1000b may receive the second character string from the server 2000 by using the communication interface 1400. The communication interface 1400 may provide the second character string received from the server 2000 to the recognition result integration module 1356.

The recognition result integration module 1356 is configured to integrate the first character string obtained from the decoding model 1352 and the second character string obtained from the WFST decoder 2310 of the server 2000. The recognition result integration module 1356 may output text corresponding to the voice input by replacing the unrecognized word string among the second strings with the word string included in the first string. In the embodiment shown in Figure 2, the recognition result integration module (1356) is by replacing the second string unrecognized word column of each of a 'Please', 'search' contained a <unk>, <unk> to the first string, , "Please search Cardi-B", which is a text corresponding to the voice input, may be output.

The text output from the recognition result integration module 1356 may be provided to the communication interface 1400 or the output unit 1500.

Unlike the device 1000a illustrated in FIG. 1, in the embodiment illustrated in FIG. 2, the device 1000b uses a decoding model 1352 using an end-to-end ASR model, and a general character string, for example, “Please search Decodes only a first character string such as "card", and the server 2000 generates a second character string including an entity name and an unrecognized word through the WFST decoder 2310 using the WFST model 2320 learned through a plurality of entity names. Can be decoded. In the embodiment shown in FIG. 2, the server 2000 has relatively higher data calculation or processing capability for training than the device 1000b, so that the WFST model 2320 through a plurality of entity names is It can be created, and the WFST model 2320 can be updated by learning a new entity name in real time. Accordingly, in the embodiment shown in FIG. 2, the device 1000b can accurately convert the voice input for the latest entity name into text through the server 2000, and use a vocabulary list including a plurality of entity names. Since there is no need to generate a WFST model through training, processing speed can also be improved. In addition, in an embodiment, the device 1000b transmits a lattice to the server 2000, rather than transmitting the feature vector in units of subwords from the decoding model 1352 to the server 2000, thereby reducing the amount of data transmission. It can be reduced, and the data communication speed with the server 2000 can be improved.

3 is a block diagram illustrating components of a device 1000 according to an embodiment of the present disclosure.

The device 1000 may be an electronic device that receives a user's voice input and processes the voice input to convert the voice input into text. The device 1000 is, for example, a smart phone, a tablet personal computer (PC), a mobile phone, a video phone, an e-book reader, and a desktop personal computer (desktop personal computer). , Laptop personal computer, netbook computer, workstation, server, personal digital assistant (PDA), portable multimedia player (PMP), MP3 player, mobile medical device, camera, Alternatively, it may be configured with at least one of a wearable device. However, the device 1000 is not limited to the above-described example.

The device 1000 may include a voice input unit 1100, a processor 1200, a memory 1300, a communication interface 1400, and an output unit 1500.

The voice input unit 1100 may receive a voice input from a user. In one embodiment, the voice input unit 1100 may include a microphone. The voice input unit 1100 may receive a voice input (eg, a user's utterance) from a user through a microphone and obtain a voice signal from the received voice input. In one embodiment, the processor 1200 of the device 1000 converts the sound received through the microphone into an acoustic signal, and removes noise (eg, non-speech component) from the acoustic signal to obtain a speech signal. I can.

Although not shown in the drawing, the device 1000 has a function of detecting a designated voice input (for example, a wake-up input such as'Hi Bixby','Okay Google', etc.) or preprocessing the voice signal obtained from some voice inputs. It may include a voice pre-processing module having a function of.

The processor 1200 may execute one or more instructions of a program stored in the memory 1300. The processor 1200 may be configured with hardware components that perform arithmetic, logic, input/output operations, and signal processing. The processor 1200 is, for example, a central processing unit (Central Processing Unit), a microprocessor (microprocessor), a graphics processor (Graphic Processing Unit), ASICs (Application Specific Integrated Circuits), DSPs (Digital Signal Processors), DSPDs (Digital Signal Processing Devices), Programmable Logic Devices (PLDs), and Field Programmable Gate Arrays (FPGAs), but are not limited thereto.

The memory 1300 may store a program including instructions for processing a user's voice input received through the voice input unit 1100 and converting it into text. The memory 1300 may store instructions and program codes that the processor 1200 can read. In the following embodiments, the processor 1200 may be implemented by executing instructions or codes of a program stored in a memory.

The memory 1300 includes an entity name vocabulary list DB 1310, a dictionary DB (dictionary database) 1320, a WFST model generation module 1330, a WFST model 1340, a speech recognition module 1350, and a deep neural network 1360. ) Data corresponding to each may be stored.

The memory 1300 is, for example, a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (eg, SD or XD memory). Etc.), RAM (RAM, Random Access Memory) SRAM (Static Random Access Memory), ROM (ROM, Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), Magnetic It may include at least one type of storage medium among memory, magnetic disk, and optical disk.

The processor 1200 may generate a WFST model 1340 for a Named Entity by using instructions or program code related to the WFST model generation module 1330 stored in the memory 1300. 'Object name' means a word or word string having a unique meaning, such as a person's name, company name, place name, area name, time, or date.

The processor 1200 may acquire a vocabulary list including a plurality of entity names, and store a vocabulary list related to the acquired entity names in the entity name vocabulary list DB 1310. In one embodiment, the processor 1200 receives a user input, receives a user input from an external server, or an application executed by the device 1000, using a command or program code of the WFST model generation module 1330, A vocabulary list including a plurality of entity names may be obtained by crawling a web page or the like. For example, the processor 1200 may obtain a word or word string corresponding to a plurality of entity names included in an application from an application programming interface (API) of an application being executed. For example, in the case of a music streaming application, the processor 1200 acquires an entity name, such as a song name, artist name, composer name, etc., which is included in the user's playlist or is currently playing, and stores the acquired entity name as an entity. It can be stored in the name vocabulary list DB 1310.

The WFST model generation module 1330 may include a text preprocessing module 1332, a filtering module 1334, and a probability model generation module 1336.

The text preprocessing module 1332 is configured to receive a plurality of entity names included in a vocabulary list, and to output a sub word by performing preprocessing on the inputted plurality of entity names. In one embodiment, the processor 1200 divides a plurality of entity names stored in the entity name vocabulary list DB 1310 into sub-word units by using a command or program code related to the text preprocessing module 1332 ( segmentation). 'Subword' is a basic unit constituting a word, and means, for example, a phoneme or a syllable. The processor 1200 may divide a plurality of entity names into subword units using, for example, a Hidden Marcov model (HMM). However, it is not limited thereto. For another example, the processor 1200 may tokenize a word or word string included in a plurality of entity names in units of subwords using a byte pair encoding (BPE) algorithm.

In an embodiment, the processor 1200 may use the text preprocessing module 1332 to remove punctuation marks, special characters, and special symbols included in a plurality of entity names, and perform preprocessing of removing stop words.

The filtering module 1334 is configured to perform filtering to remove an in vocabulary previously stored in the dictionary DB 1320 among subwords output from the text preprocessing module 1332. In one embodiment, the processor 1200 uses the filtering module 1334 to provide a word or word string identical to a vocabulary previously stored in the dictionary DB 1320 among a plurality of entity names included in the entity name vocabulary list DB 1310. You can perform filtering to remove. The processor 1200 may acquire only sub-words extracted from words (Out of Vocabulary) corresponding to entity names not included in the dictionary DB 1320 through filtering.

Probability model generation module 1336 is a word or word string representing at least one entity name among a plurality of entity names included in the entity name vocabulary list by subwords output through the text preprocessing module 1332 and the filtering module 1334. It is configured to train a predictable probability and generate a WFST model 1340 as a result of the training. In one embodiment, the processor 1200 inputs each of the subwords extracted from each of the plurality of entity names stored in the entity name vocabulary list DB 1310 using an instruction or program code of the probability model generation module 1336 In addition, a WFST model 1340 including a finite state transducer encoding a mapping using an entity name as an output may be generated.

In one embodiment, the processor 1200 classifies a plurality of entity names stored in the entity name vocabulary list DB 1310 according to a plurality of different domains, and uses the classified plurality of entity names for each domain. A plurality of WFST models 1340 may be generated. Here, the'domain' means a field or category related to a voice input received from a user, and may be set in advance according to, for example, a meaning of a voice input or a property of a voice input. Domains may be classified according to services related to voice input. The domain may include, for example, one or more of a movie domain, a music domain, a book domain, a game domain, an aviation domain, and a food domain. An embodiment in which the processor 1200 generates a plurality of WFST models 1340 for each domain and automatically determines a domain using the generated plurality of WFST models 1340 will be described in detail with reference to FIGS. 7 and 8. To

In one embodiment, the processor 1200 uses the communication interface 1400, from a server of an application service provider, to a place of interest including an entity name of at least one of a place name, a tourist attraction, a tourist destination, and a famous restaurant. A WFST model 1340 may be generated by receiving a (Point of Interest; POI) vocabulary list and learning using an entity name included in the received point of interest vocabulary list. For example, the application may be a navigation application. A specific embodiment in which the processor 1200 receives a place of interest vocabulary list using an entity name for a new area and generates a WFST model 1340 using the place of interest vocabulary list is described in detail in FIGS. 10 and 11. I will explain.

In one embodiment, the processor 1200 stores a plurality of entity names reflecting the user's characteristics from at least one of a frequently executed application, log data of a messenger application, and a search term record in a content streaming application. The WFST model 1340 may be generated through learning that is stored in the list DB 1310 and uses the stored personalized vocabulary list. A specific embodiment in which the processor 1200 generates the WFST model 1340 using a personalized vocabulary list will be described in detail with reference to FIG. 12.

The WFST model 1340 is a language model that outputs a word or word sequence when a sub word is input based on a probability that the input sub word can be predicted as a word or word sequence representing an entity name. The WFST model 1340 may output a word or word string from a specific subword based on the probability of occurrence.

The voice recognition module 1350 is a module configured to output a text corresponding to the voice input by processing the user's voice input acquired through the voice input unit 1100. The speech recognition module 1350 may include a first decoding model 1352, a second decoding model 1354, and a recognition result integration module 1356.

The first decoding model 1352 receives a speech input, obtains a speech signal from the received speech input, and is a vector of probability values at which a phoneme can be predicted with a specific label according to the length of each phoneme of the speech signal. It is configured to obtain a feature vector and output a first character string based on a probability value of the feature vector. Here, the'label' is a token defined by the deep neural network 1360 that has already been learned. In one embodiment, the'label' may be any subword representing a phoneme or syllable. The label may have the same concept as the sub word, but is not limited thereto.

In an embodiment, the first decoding model 1352 may perform speech recognition on a speech signal in an end-to-end automatic speech recognition (ASR) method. The end-to-end ASR scheme is a speech recognition scheme using a deep neural network 1360 trained to directly map a speech signal to a string or word string. Unlike other speech recognition methods that use multiple models such as acoustic models and language models, the end-to-end ASR method can simplify the speech recognition process by using one trained deep neural network 1360. As a lower embodiment of the end-to-end ASR model, there are, for example, a Recurrent Neural Network Transducer (RNN-T) model and an attention-based model.

In one embodiment, the first decoding model 1352 may use an end-to-end ASR model based on an attention-based model. The attention-based model may include, for example, Transformer or Listen-Attend-Spell (LAS).

In an embodiment, the voice recognition module 1350 may further include a voice input preprocessing module. The processor 1200 performs analog/digital conversion (A/D) on the voice input using a command or program code of the voice input preprocessing module, and converts the voice signal output as a digital signal to a predetermined length and a predetermined shift amount. It can be framed using some overlapping windows. The processor 1200 may use the speech preprocessing module to perform predetermined signal processing on each of the frames acquired from the speech preprocessing module, and extract a feature vector by extracting a speech feature amount of the frame. As the speech feature quantity, a Mel-Frequency Cepstrum Coefficient (MFCC), a first derivative, a second derivative, and the like may be used.

The processor 1200 extracts a feature vector representing a posterior probability that a frame at each time corresponds to a specific label from the signal-processed speech signal using a command or program code related to the first decoding model 1352 can do. In an embodiment, the feature vector may be a softmax column including posterior probability values from which a speech signal having a predetermined length can be predicted with a specific label. When the probability values included in each softmax column are summed, it becomes 1.

The processor 1200 uses the first decoding model 1352 to select one label from each frame from feature vectors obtained in chronological order, concatenating the selected labels, and use each of the corresponding labels. Label candidates can be obtained by expressing the phoneme. The processor 1200 may obtain the first character string using the posterior probability of the label candidate. For example, if the voice input received from the user is an utterance of'Please search Cardi-B', the processor 1200 uses the first decoding model 1352 to select the'Please search card' from the voice signal extracted from the user's utterance. A first character string such as' may be obtained. In this case, the processor 1200 cannot accurately predict the character string corresponding to the object name'Cardi-B' that is not stored in the dictionary DB 1320.

In one embodiment, the first decoding model 1352 converts a feature vector in units of labels into a feature vector in units of sub words, and outputs the feature vector in units of sub words obtained as a result of the conversion as the second decoding model 1354. I can.

The second decoding model 1354 includes a word corresponding to at least one entity name among a plurality of entity names included in the entity name vocabulary list from the subword unit feature vector, and an unrecognized word sequence that is not identified as entity name. It is configured to obtain 2 strings. The second decoding model 1354 may output a second character string predicted from a sub-word unit feature vector by using the WFST model 1340. The processor 1200 uses a command or program code related to the second decoding model 1354 to calculate a confidence score based on the likelihood of a word for each subword, dictionary information, and a language model. Calculate, select a character string having a high reliability score, and output a second character string including the selected character string.

The second decoding model 1354 may include a Lexicon FST 1354L and a grammar FST 1354G. The second decoding model 1354 may output a word or a word sequence from a sub word sequence (label) by composing a token FST 1354T, a Lexicon FST 1354L, and a grammar FST 1354G.

The Lexicon FST 1354L is configured to receive a feature vector in units of subwords from the first decoding model 1352 and output a word or word sequence predicted based on the feature vectors in units of subwords. The processor 1200 may include mapping information, which is a probability value by which a subword can be predicted as a specific word, using a command or program code related to the Lexicon FST 1354L. In one embodiment, the Lexicon FST 1354L may convert the sub-word sequence s into P(s|W), which is a probability related to a word sequence W.

The grammar FST (1354G) is configured to receive a word or word string of a sub-word sequence from the Lexicon FST (1354L) and output a second string including a word string corresponding to an entity name and an unrecognized word string. . The grammar FST 1354G may be a model in which weights for predicting a word sequence that may be arranged after the input word or word sequence when a specific word or word sequence is input. The grammar FST 1354G may predict a word or word sequence that may appear after a specific word or word sequence using, for example, a recurrent neural network (RNN) or a statistical n-Gram model. In one embodiment, the processor 1200 includes information on the language model probability P(W) as a weight for the word string W, using an instruction or program code related to the grammar FST 1354G, and the probability P(W A word or word string with) can be displayed.

The processor 1200 uses the instruction or program code of the second decoding model 1354 to determine the probability that the input X of the label-unit feature vector is predicted as the subword s, P(s|X) and the subword s, are words or Calculate the word posterior probability P(W|X) by combining P(W|s), which is the probability predicted by the word string W, and search for a hypothesis where the word posterior probability (W|X) is maximum. A word or word string corresponding to a name can be displayed. The processor 1200 may acquire a word corresponding to an entity name learned through the WFST model 1340, for example,'Cardi-B'. However, the processor 1200 is a second case where the decoding model (1354), the word or word sequence, except for the named entity learned by WFST model 1340 is unable to identify, unrecognized word sequence (<unk> <unk >). The processor 1200 may obtain a second character string including an entity name and an unrecognized word sequence.

The recognition result integration module 1356 is configured to integrate the first character string obtained from the first decoding model 1352 and the second character string obtained from the second decoding model 1354. The processor 1200 outputs text corresponding to the speech input by replacing the unrecognized word string among the second strings with the word string included in the first string using a command or program code related to the recognition result integration module 1356 can do. In an embodiment, the processor 1200 may replace the unrecognized word string included in the second string with a word string corresponding to a position of the unrecognized word string among the first strings. A processor, for example, the second string unrecognized word column of <unk>, the text that is by replacing each of the <unk> a 'Please', 'search' contained in the first character string, corresponding to the voice input "Please search Cardi-B" can be obtained.

The processor 1200 may provide the text acquired through the speech recognition module 1350 to the communication interface 1400 or the output unit 1500.

The communication interface 1400 may perform data communication with a server 2000 (refer to FIG. 2 ), a server of an application service provider, or another device. The communication interface 1400 is, for example, wired LAN, wireless LAN, Wi-Fi, Bluetooth, zigbee, Wi-Fi Direct (WFD), infrared communication (IrDA, infrared Data Association), BLE (Bluetooth Low Energy), NFC (Near Field Communication), Wibro (Wireless Broadband Internet, Wibro), WiMAX (World Interoperability for Microwave Access, WiMAX), SWAP (Shared Wireless Access Protocol), Wiigg Data can be transmitted and received with the server 2000, a server of an application service provider, or other devices using at least one of a data communication method including (Wireless Gigabit Allicance, WiGig) and RF communication.

The output unit 1500 may output text corresponding to a voice input. The output unit 1500 may notify a user of a result of speech recognition, that is, a text, or may transmit it to an external device (eg, a smart phone, a home appliance, a wearable device, a server, etc.). The output unit 1500 may include a display unit 1510 and a speaker 1520.

The display 1510 may display text converted from a voice input. The display unit 1510 is, for example, a liquid crystal display, a thin film transistor-liquid crystal display, an organic light-emitting diode, a flexible display, It may be composed of at least one of a 3D display and an electrophoretic display.

The speaker 1520 may output an audio signal corresponding to text.

4 is a block diagram showing the components of the server 2000 according to an embodiment of the present disclosure.

The server 2000 may receive a feature vector in units of subwords from the device 1000, convert the received feature vector into a character string, and transmit the converted character string to the device 1000.

Referring to FIG. 4, the server 2000 may include a communication interface 2100, a processor 2200, and a memory 2300.

The communication interface 2100 may transmit and receive data between the server 2000 and the device 1000. The communication interface 2100 is, for example, wired LAN, wireless LAN, Wi-Fi, WFD (Wi-Fi Direct), Wibro (Wireless Broadband Internet, Wibro), WiMAX (World Interoperability for Microwave Access, WiMAX), SWAP (Shared Wireless). Access Protocol), Wireless Gigabit Allicance (WiGig), and data communication with the device 1000 may be transmitted and received using at least one of a data communication method including RF communication.

The communication interface 2100 may receive a feature vector in units of subwords from the device 1000. In an embodiment, the communication interface 2100 may receive a lattice from the device 1000. The lattice is generated by the device 1000 by connecting labels having the highest probability value among the probability values of the feature vector, and may be transmitted from the device 1000 to the server 2000. In an embodiment, the communication interface 2100 may transmit the second character string acquired by the WFST decoder 2310 to the device 1000 under the control of the processor 2200. The second character string may include a word related to an entity name and an unrecognized word sequence that is not identified as an entity name.

The processor 2200 may execute one or more instructions of a program stored in the memory 2300. The processor 2200 may be composed of hardware components that perform arithmetic, logic, input/output operations, and signal processing. The processor 2200 includes, for example, a central processing unit, a microprocessor, a graphic processing unit, application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processors (DSPs). Signal Processing Devices), Programmable Logic Devices (PLDs), and Field Programmable Gate Arrays (FPGAs), but are not limited thereto.

A program including instructions for each of the WFST decoder 2310, the WFST model 2320, and the entity name vocabulary list DB 2330 may be stored in the memory 2300. The memory 2300 may store instructions and program codes that the processor 2200 can read. In the following embodiments, the processor 2200 may be implemented by executing instructions or codes of a program stored in a memory.

The memory 2300 is, for example, a flash memory type, a hard disk type, a multimedia card micro type, and a card type memory (eg, SD or XD memory). Etc.), RAM (RAM, Random Access Memory) SRAM (Static Random Access Memory), ROM (ROM, Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), Magnetic It may include at least one type of storage medium among memory, magnetic disk, and optical disk. However, it is not limited thereto.

The WFST decoder 2310 includes a word corresponding to at least one entity name among a plurality of entity names included in the entity name vocabulary list using the WFST model 2320 and an unrecognized word string not identified as entity name. It is configured to obtain a second character string. In an embodiment, the WFST decoder 2310 may be configured as a module that outputs a word or word sequence having a high likelihood based on a probability value of the feature vector when a feature vector in units of subwords is input.

The processor 2200 includes a word corresponding to an entity name and an unrecognized word by decoding a feature vector in units of sub-words received from the device 1000 using instructions or program codes for the WFST decoder 2310. You can get the character string. The processor 2200 may control the communication interface 2100 to transmit the acquired character string to the device 1000.

The WFST model 2320 is a model that outputs a word or word sequence when a sub word is input based on a probability that the sub word can be predicted as a word or word sequence representing a specific entity name. The WFST model 2320 is a model that is created or updated by the processor 2200, and a specific subword is a word or word string representing at least one entity name among a plurality of entity names included in the entity name vocabulary list DB 2330. It can be created by training the probabilities that can be predicted. In one embodiment, the processor 2200 receives each of the subwords extracted from each of the plurality of entity names stored in the entity name vocabulary list DB 2330 as inputs, and encodes a mapping using the entity name as an output. A WFST model 2320 including a Finite State Transducer may be created.

The entity name vocabulary list DB 2330 is a database that stores vocabularies corresponding to a plurality of entity names. In one embodiment, the processor 2200 acquires vocabularies corresponding to a new entity name from a new application, a web page, a video streaming service, or a game, and uses the acquired vocabulary to obtain the entity name vocabulary list DB 2330. Can be updated.

The processor 2200 may transmit update information of the entity name vocabulary list DB 2330 to the device 1000 through the communication interface 2100. Here, the'update information of the object name vocabulary list DB 2330' is, for example, at least one new object name newly added to the object name vocabulary list DB 2330, and each of at least one new object name can be classified. It may include at least one of existing domain information, application information related to a new entity name, deletion information of a previously stored entity name, and change information of a previously stored entity name. In an embodiment, the processor 2200 may transmit update information of the entity name vocabulary list DB 2330 to the device 1000 at a preset period, for example, every 6 hours, 1 day, 1 week, or month. However, it is not limited thereto. In another embodiment, the processor 2200 stores the update information of the entity name vocabulary list DB 2330 every time a new entity name vocabulary is added, deleted, or changed to the entity name vocabulary list DB 2330. ).

The device 1000 receives the update information of the entity name vocabulary list DB 2330 from the server 2000, and uses the updated information of the received entity name vocabulary list DB 2330 to store the entity name in the device 1000. The vocabulary list DB 1310 can be updated. A specific embodiment in which the device 1000 updates the entity name vocabulary list DB 1310 using the update information received from the server 2000 will be described in detail with reference to FIG. 9.

5 is a flowchart illustrating an embodiment in which the device 1000 of the present disclosure recognizes a voice input.

In step S510, the device 1000 generates a weighted finite state transducer model by using a vocabulary list including a plurality of named entities. The device 1000 is a weighted finite state converter model using words or word strings corresponding to a plurality of entity names stored in the entity name vocabulary list DB 1310 (refer to FIG. 3) in the background while the device is in standby mode or while an application is running. Can be created. However, it is not limited thereto. In one embodiment, the device 1000 detects a preset voice input, for example, a wake-up input such as'Hi Bixby','Okay Google', etc., and a weight finite state regarding the entity name in response to the wake-up input. You can create a transducer model. In another embodiment, the device 1000 presses a button for executing a voice assistant service (for example, Bixby or Google Assistant), or a graphic user interface displayed on the display unit 1510 (refer to FIG. 3). When a user input touching (GUI) is received, a weighted finite state converter model for an entity name may be generated.

In one embodiment, the device 1000 receives a user input for entering an entity name, receives an entity name from an external server, or crawls applications, web pages, etc. executed by the device 1000 By doing so, you can get the object name. The device 1000 may store the acquired entity name in the entity name vocabulary list DB 1310.

The device 1000 may segment words or strings constituting a plurality of entity names included in the entity name vocabulary list DB 1310 into subwords that are phoneme or syllable units. In an embodiment, the device 1000 may tokenize a word or a word string included in a plurality of entity names in units of subwords using a byte pair encoding (BPE) algorithm.

In an embodiment, the device 1000 may generate a statistical model comprising a probability graph of a posterior probability value in which a subword can be predicted as at least one specific entity name among a plurality of entity names. The device 1000 may generate a weighted finite state converter model (WFST model) for an entity name by learning a weight through a state transition using the frequency of the subwords and the arrangement order of the subwords. have.

The WFST model includes a Lexicon Finite State Transducer (FST) containing mapping information, which is a probability value that a subword can be predicted into a specific word, and a weight for predicting the next word column when a specific word or word column is input. weight) information may include a Grammar Finite State Transducer (FST). The WFST model can be composed of a composition of Lexicon FST and Gramer FST.

In step S520, the device 1000 receives a user's voice input. In an embodiment, the device 1000 may receive a voice input (eg, a user's utterance) from a user through a microphone and obtain a voice signal from the received voice input. In one embodiment, the processor 1200 of the device 1000 (refer to FIG. 3) converts the sound received through the microphone into an acoustic signal, and removes noise (eg, non-speech component) from the sound signal to The signal can be acquired.

The user's voice input received in operation S520 may be a voice command related to an operation or function that the user intends to execute through the device 1000, but is not limited thereto. In one embodiment, the user's voice input may include a wake-up input and a voice command related to an operation or function.

When the user's voice input includes a wake-up input, step S520 may be performed before step S510. For example, when the user utters'Hi Bixby, tell me today's weather', the device 1000 may simultaneously receive a wake-up input'Hi Bixby' and a voice command'Tell me today's weather'. . In this case, the device 1000 may generate a weighted finite state converter model for the entity name in response to the received voice input.

In step S530, the device 1000 obtains a feature vector for speech input by using the first decoding model, and obtains a first character string by using a probability value of the feature vector. In an embodiment, the device 1000 may extract a feature vector, which is a vector of probability values for which pronunciation can be predicted with a specific label according to the length of each phoneme of the speech signal using the first decoding model. The feature vector may be a softmax column including posterior probability values from which a speech signal having a predetermined length can be predicted with a specific label.

In one embodiment, the device 1000 selects one label in each frame from feature vectors obtained in chronological order using the first decoding model, concatenating the selected labels, and uses the corresponding label. Label candidates can be obtained by expressing each phoneme. The device 1000 may obtain the first character string using the posterior probability of the label candidate.

In one embodiment, the first decoding model may be a model configured to perform speech recognition on a speech signal in an end-to-end automatic speech recognition (ASR) scheme.

In step S540, the device 1000 inputs a feature vector to the second decoding model using the WFST model generated in step S510, and is not identified as a word sequence and entity name corresponding to a specific entity name from the feature vector. A second character string including an unrecognized word sequence is acquired. In one embodiment, the device 1000 calculates a confidence score based on a likelihood for a word for each subword, dictionary information, and a language model using a second decoding model, and a confidence score A character string having a high value may be selected, and a second character string including the selected character string may be output. The device 1000 uses the second decoding model to predict P(s|X), which is a probability that the input X of the feature vector of the label unit is predicted as the subword s, and the probability that the subword s is predicted as a word or word string W. The word or word corresponding to the entity name is calculated by calculating the word posterior probability P(W|X) by combining P(W|s), and searching for a hypothesis that maximizes the word posterior probability (W|X). Heat can be obtained. The device 1000 may not identify a word or word sequence other than the entity name learned by the WFST model, and may output it in the form of an unrecognized word sequence.

In step S550, the device 1000 outputs text corresponding to the voice input by replacing the unrecognized word string among the second strings with the word string included in the first string. In an embodiment, the device 1000 may replace the unrecognized word string included in the second string with a word string corresponding to a position of the unrecognized word string among the first strings.

6 is a flowchart illustrating an embodiment in which the device 1000 of the present disclosure generates a weighted finite state converter model for an entity name. Steps S610 to S640 shown in FIG. 6 are steps embodied in step S510 of FIG. 5. Step S520 of FIG. 5 may be performed after step S640 shown in FIG. 6.

In step S610, the device 1000 acquires a vocabulary list including a plurality of entity names. In one embodiment, the device 1000 receives a user input for entering an entity name, receives from an external server, or crawls an application, web page, etc., executed by the device 1000 A vocabulary list including entity names can be obtained. The device 1000 may obtain, for example, a word or word string corresponding to a plurality of entity names included in the application from an application programming interface (API) of an application being executed. For example, in the case of a music streaming application, the device 1000 may acquire a word or word string corresponding to an entity name, such as a song name, artist name, composer name, etc. that is included in the user's playlist or is currently playing. I can. In an embodiment, the device 1000 may store a word or word string related to the acquired entity name in a vocabulary list DB 1310 (see FIG. 3) of the memory 1300 (see FIG. 3 ).

After operation S610 is performed, the device 1000 may perform preprocessing of removing punctuation marks, special characters, and special symbols included in the plurality of entity names, and removing stop words.

In step S620, the device 1000 performs filtering to remove an entity name overlapping with a word previously stored in the device 1000 among a plurality of entity names included in the vocabulary list. In one embodiment, the device 1000 is the same entity name as a word or word string previously stored in the dictionary DB 1320 (refer to FIG. 3) among a plurality of entity names included in the entity name vocabulary list DB 1310 of the memory 1300 Can be removed. By performing filtering, only words (Out of Vocabulary; OOV) corresponding to entity names not included in the dictionary DB 1320 may be stored in the entity name vocabulary list DB 1310.

Step S620 (performing filtering) is not an essential step. In an embodiment of the present disclosure, step S620 may be omitted. In this case, after step S610 is performed, step S630 may be performed.

In step S630, the device 1000 divides a plurality of entity names into sub-word units (segmentation). 'Subword' is a basic unit constituting a word, and means, for example, a phoneme or a syllable. The device 1000 may divide a plurality of entity names into subword units using, for example, a Hidden Marcov model (HMM). However, it is not limited thereto. For another example, the device 1000 may tokenize a word or a word sequence corresponding to a plurality of entity names in units of sub-words using a byte pair encoding (BPE) algorithm.

In step S640, the device 1000 acquires a confidence score for predicting a sub-word as a specific entity name through training using the frequency and arrangement order of the sub-words. In an embodiment, the device 1000 uses a prior probability obtained based on the frequency of the subwords output in step S630 and the arrangement order of the subwords, and each subword is at least one specific entity among a plurality of entity names. It is possible to create a statistical model consisting of a probability graph that can be predicted by a person. Here, the'priority probability' means a statistically pre-calculated probability value based on the frequency and arrangement order in which a predetermined subword is used as a specific word or word string.

The device 1000 may generate a weighted finite state converter model (WFST model) for an entity name by learning a weight through state transition using a prior probability of a subword.

The weighted finite state converter model generated in step S640 may be a language model in which a sub word is input and a word representing an entity name or a word string is output. The weighted finite state converter model may output a word or word sequence corresponding to an entity name when a sub word is input based on a probability that an input sub word can be predicted as a word or word sequence.

FIG. 7 is a diagram illustrating an embodiment in which the device 1000 of the present disclosure automatically selects a domain using weighted finite state converter models 1341-1, 1340-2, and 1340-3.

Referring to FIG. 7, the device 1000 may identify a word or word sequence corresponding to a plurality of entity names from an application being executed. In the embodiment illustrated in FIG. 7, the device 1000 is executing a music streaming application, and album art of currently played music and a user's playlist may be displayed on the display unit 1510. The playlist may include a song name, artist name, composer name, album art, or the like related to music currently played or to be played. The device 1000 may obtain a word or word string corresponding to a plurality of entity names included in the application from an application programming interface (API) of an application being executed. In an embodiment, the device 1000 may identify a word or a word string related to an entity name including at least one of a song name, an artist name, and a composer name from the music streaming application.

The device 1000 may store the identified at least one word or word string in the entity name vocabulary list NE. In the embodiment shown in FIG. 7, the entity name vocabulary list NE may store entity names related to artist names including ZICO, Heize, Mark Ronson, Bruno Mars, and Michael Jackson.

In an embodiment, the device 1000 may include a domain selection module 1370. The domain selection module 1370 compares a plurality of entity names identified by the device 1000 with an entity name vocabulary list included in each of a plurality of pre-generated weighted finite state converter models, thereby selecting domains associated with the identified plurality of entity names. It is a module that is configured to be automatically selected. In an embodiment, the domain selection module 1370 may be included in the memory 1300 (refer to FIG. 3) of the device 1000.

The device 1000 uses a command or program code related to the domain selection module 1370 to obtain an entity name included in each of the previously generated weighted finite state converter models 1341-1, 1340-2, and 1340-3. The entity names included in the entity name vocabulary list NE may be compared, and a domain related to an application currently being executed may be automatically selected from among a plurality of domains based on the comparison result. Here, the'domain' means a field or category related to voice input. For example, domains may be classified according to the meaning of voice input, attributes of voice input, and services related to voice input. The domain may include, for example, one or more of a movie domain, a music domain, a book domain, a game domain, an aviation domain, and a food domain.

The plurality of weighted finite state converter models 1341-1, 1340-2, and 1340-3 are models trained using a plurality of entity names classified according to a plurality of different domains. For example, the first weighted finite state converter model 1341-1 is a model trained using a plurality of entity names related to music, and the second weighted finite state converter model 1340-2 is It is a model trained using entity names, and the third weighted finite state converter model 1340-3 is a model trained using a plurality of entity names related to a game. In FIG. 7, a total of three weighted finite state converter models 1341-1, 1340-2, and 1340-3 are illustrated, but are not limited thereto.

The device 1000 uses the domain selection module 1370 to compare the entity name included in the entity name vocabulary list NE with the entity name included in the first weighted finite state converter model 1341-1, and You can count the number of entity names. Similarly, the device 1000 uses the domain selection module 1370 to configure the entity name included in the entity name vocabulary list NE, the second weighted finite state converter model 1340-2, and the third weighted finite state converter model ( 1340-3) The entity names included in each may be compared, and the number of duplicate entity names may be counted according to the weighted finite state converter model.

The device 1000 may select a domain based on a weighted finite state converter model in which the number of counted entity names is the maximum. In the embodiment shown in FIG. 7, the entity name vocabulary list NE includes a plurality of entity names related to the music domain (eg, artist names in the playlist of a music streaming application), and the number of duplicate entity names The weighted finite state converter model having a maximum value of may be a first weighted finite state converter model 1341-1 learned through a plurality of entity names corresponding to the music domain. The device 1000 may determine'music', which is a domain in which the first weighted finite state converter model 1341-1 is learned, as a domain related to an application currently being executed.

When a user's voice input related to a running application is received, the device 1000 may analyze the voice input based on the determined domain and perform an operation or function according to the analysis result. For example, when a voice input such as "Play a Bruno Mars song~" is received, the device 1000 interprets the user's voice input based on the determined domain'music', and is included in the playlist according to the analysis result. You can play Bruno Mars' song among the multiple songs that have been made.

8 is a flowchart illustrating an embodiment in which the device 1000 of the present disclosure automatically selects a domain using a weighted finite state converter model.

In step S810, the device 1000 identifies words corresponding to the entity name from the executed application or web page. In an embodiment, the device 1000 may obtain a word or word string corresponding to a plurality of entity names included in the application from an API of an application executed by the device 1000. In another embodiment, the device 1000 may acquire words or word strings for a plurality of entity names included in the web page by crawling the currently accessed web page.

The device 1000 may store words or word strings corresponding to the acquired plurality of entity names in the entity name vocabulary list DB 1310.

In step S820, the device 1000 compares the identified words with the names of entities included in the vocabulary list of each of the plurality of weighted finite state converter models previously generated for each domain. The plurality of weighted finite state converter models are models trained using a plurality of entity names classified according to a plurality of different domains. Each of the plurality of weighted finite state transformer models may include a vocabulary list including a plurality of entity names classified into a plurality of different domains. In one embodiment, the device 1000 compares the words corresponding to the entity name identified in step S810 with a plurality of entity names in a vocabulary list included in each of the plurality of weighted finite state converter models, and as a result of the comparison, duplicate entity names Can be identified, and the number of identified entity names can be counted.

In step S830, the device 1000 determines a domain in which the running application or web page can be classified based on the comparison result. In an embodiment, the device 1000 may determine a domain based on a weighted finite state converter model in which the number of duplicate entity names counted in step S820 is the largest among a plurality of weighted finite state converter models. For example, if the weighted finite state converter model in which the number of counted entity names is the largest is a model including a vocabulary list of entity names related to the'music' domain, the device 1000 determines the domain related to the running application or web page. It can be decided by'music'.

In the conventional speech recognition technology, when a user's voice input is received, in order to automatically select a domain related to the received voice input, a pattern sentence according to an entity name is learned, and a domain based on the learned pattern sentence Should be selected. Pattern sentences according to the entity name have a high probability of combining specific voice commands according to the specific entity name, such as "<object name> + show" or "<object name> + hear". There is a problem in that the amount of data computation is large and the learning speed takes a long time to perform the learning about the problem. In addition, when there is no specialized pattern for the domain, there is a problem in that the domain related to the voice input cannot be accurately selected.

In the embodiments shown in FIGS. 7 and 8, a plurality of entity names are identified from an application running through the device 1000 or a web page accessed through the device 1000, and a plurality of previously generated weights The domain is selected by comparing the entity names included in the state converter models 1341-1, 1340-2, and 1340-3 with the identified plurality of entity names, which has an advantage in that the processing speed is high compared to the prior art. In addition, in the embodiment of the present disclosure, the identified entity name and the entity name included in the plurality of weighted finite state converter models (1340-1, 1340-2, 1340-3) are compared, so that consideration of the pattern sentence is unnecessary. The accuracy of domain selection can be improved.

9 is a flowchart illustrating an embodiment in which the device 1000 according to the present disclosure generates a weighted finite state converter model for an entity name using information received from the server 2000.

In step S910, the device 1000 receives update information of the entity name vocabulary list from the server 2000. The'object name vocabulary list update information' is, for example, at least one new entity name newly added to the entity name vocabulary list DB (2330, see Fig. 4) stored in the server 2000, and at least one new entity name, respectively. It may include at least one of domain information that can be classified, application information related to a new entity name, deletion information of a previously stored entity name, and change information of a previously stored entity name. In an embodiment, the device 1000 may receive update information of the entity name vocabulary list DB 2330 from the server 2000 according to a preset period. The preset period may be, for example, 6 hours, 1 day, 1 week, or 1 month, but is not limited thereto.

In another embodiment, the device 1000 is the entity name vocabulary list DB (the entity name vocabulary list DB 2330) of the server 2000 at each time when a new entity name vocabulary is added, deleted, or updated. Update information of 2330 may be received from the server 2000.

In step S920, the entity name vocabulary list of the device 1000 is updated using the received update information. The device 1000 uses the update information received from the server 2000 to add a new entity name to the entity name vocabulary list DB 1310 previously stored in the memory 1300 (refer to FIG. 3), or save the entity name. You can delete or change the previously saved object name.

In step S930, the device 1000 generates a weighted finite state converter model through learning using the updated vocabulary list. In one embodiment, the device 1000 divides at least one word or word column included in the updated entity name vocabulary list DB 1310 into sub words, and prior probability information of the divided sub words (for example, A weighted finite state transformer model may be generated by learning a weight for a posterior probability that each subword can be predicted as a specific word or word sequence using subword frequencies and arrangement order). A detailed method of generating a weighted finite state converter model by using a word or word string for a plurality of entity names by the device 1000 is the same as the method described with reference to FIGS. 1 and 3, and a duplicate description will be omitted.

In an embodiment, the device 1000 may update the previously generated weighted finite state converter model by using the update information of the entity name vocabulary list DB 1310.

In the embodiment shown in FIG. 9, the device 1000 receives update information of the entity name from the server 2000, and generates a weighted finite state converter model for the entity name using the received entity name update information, or By updating, the object name can be kept up to date. Accordingly, the device 1000 according to an exemplary embodiment of the present disclosure may improve recognition accuracy regarding a user's voice input including speech regarding the latest entity name.

10 is a conceptual diagram illustrating an embodiment of generating a weighted finite state converter model by using a place of interest vocabulary list related to the new area when the device 1000 of the present disclosure enters a new area.

Referring to FIG. 10, a user may execute a navigation application through the device 1000 while using the vehicle 100. In one embodiment, the device 1000 may include a location sensor, for example a GPS sensor. The device 1000 may acquire location information of the current device 1000 moving through the vehicle 100 using a GPS sensor.

When the vehicle 100 enters a new area, the device 1000 transmits entry information indicating that it has entered the new area using a GPS sensor to the server 3000 of an application service provider (step S1010). The'new area entry information' may include, for example, at least one of location information of a new area, an entry time point, and location information about an entry point during a preset time period. In the embodiment illustrated in FIG. 10, the'application service provider' may be a navigation application service provider.

The'new area' may refer to an area installed in the device 1000 and not stored in a location-based application executed by the device 1000, but stored in the server 3000 of an application service provider. Here, the'location-based application' refers to an application that provides information based on location information of the device 1000, such as a navigation application or a map application, or executes a specific operation or function. In the case of a navigation application executed by the device 1000, due to the limitation of the installed capacity in the device 1000, it is not possible to include all information about places such as place names, attractions, or tourist attractions. In the case of a navigation application installed in the device 1000, only place information regarding a minimum area may be stored, and place information regarding a new area may be stored only in the server 3000.

When receiving information about entering a new area from the device 1000, the server 3000 of the application service provider provides the device 1000 with a point of interest vocabulary list POI related to the new area (step S1020 ). The point of interest vocabulary list (POI) may include, for example, an entity name of at least one of a place name for a new area, a tourist attraction, a tourist destination, and a famous restaurant. In the embodiment shown in FIG. 10, the point of interest vocabulary list (POI) is an example of an entity name related to a place name, attraction, or tourist destination of a new area, and includes Times Square, Gershwin Theater, The Town Hall, Madame Tussauds, and Bryant Park. Can include.

The device 1000 may generate a weighted finite state converter model through learning using an entity name included in a POI received from the server 3000 of an application service provider. In one embodiment, the device 1000 divides a word or word column corresponding to an entity name included in the POI vocabulary list in units of sub-words, and prior probability information of the divided sub-words (for example, A weighted finite state transformer model may be generated by learning a weight for a posterior probability that each subword can be predicted as a specific word or word sequence using subword frequencies and arrangement order). A detailed method of generating a weighted finite state converter model by using a plurality of entity names by the device 1000 is the same as the method described in FIGS. 1 and 3, and a duplicate description will be omitted.

11 is a conceptual diagram illustrating an embodiment of generating a weighted finite state converter model by using a place of interest vocabulary list related to the new area when the device 1000 of the present disclosure enters a new area.

In step S1110, the device 1000 recognizes that it enters a new area. In one embodiment, the device 1000 may include a location sensor such as a GPS sensor. The device 1000 may recognize that the device 1000 has entered a new area by obtaining location information of the device 1000 using a location sensor. The'new area' may refer to an area installed in the device 1000 and not stored in a location-based application executed by the device 1000, but stored in a server of an application service provider.

In step S1120, the device 1000 transmits entry information of a new area to the server of the application service provider. In an embodiment, the device 1000 may transmit information about entering a new area to a server of an application service provider using the communication interface 1400 (see FIG. 3 ). The'new area entry information' may include, for example, at least one of location information of a new area, an entry time point, and location information about an entry point during a preset time period.

In step S1130, the device 1000 receives a point of interest vocabulary list including an entity name for a new area from a server of an application service provider. The point of interest vocabulary list (POI) may include, for example, an entity name of at least one of a place name for a new area, a tourist attraction, a tourist destination, and a famous restaurant.

In step S1140, the device 1000 generates a weighted finite state converter model through learning using the entity name included in the received place of interest vocabulary list. In one embodiment, the device 1000 divides a word or a word column corresponding to an entity name included in a place of interest vocabulary list into sub-word units, and prior probability information of the divided sub-words (eg, A weighted finite state transformer model may be generated by learning a weight for a posterior probability that each subword can be predicted as a specific word or word sequence using a frequency number and an order of arrangement).

12 is a diagram illustrating an embodiment in which the device 1000 of the present disclosure generates a personalized weighted finite state converter model using an entity name reflecting a user's personal characteristics.

Referring to FIG. 12, the device 1000 may execute a messenger application, and a chat window of the messenger application may be displayed on the display unit 1510. In one embodiment, the device 1000 may obtain a personalized entity name vocabulary list NE including a plurality of entity names reflecting the personal characteristics of the user by analyzing log data of the messenger application. have. 'Personal characteristics' may include personal information such as age, gender, school, and work, as well as personal relationships such as a user's friend or company employee, and fields of interest such as games, sports, music, and movies that the user is interested in. . In the embodiment shown in FIG. 12, the device 1000 identifies soccer-related entity names such as Manchester United, Tottenham, Harry Kane, Pogba, Mourinho, and White Hart Lane from log data of a messenger application, and identified soccer-related entities A personalized entity name vocabulary list (NE) including names may be obtained.

The device 1000 may generate the personalized weight finite state converter model 1340p by using instructions or program codes related to the WFST model generation module 1330. In one embodiment, the device 1000 divides a plurality of entity names included in the personalized entity name vocabulary list NE into subword units, and prior probability information of the divided subwords (e.g., A personalized weighted finite state transformer model 1340p may be generated by learning a weight for a posterior probability that each subword can be predicted as a specific word or word sequence using a frequency number and an arrangement order).

12 illustrates that the device 1000 acquires a personalized entity name vocabulary list NE using log data of a messenger application, but is not limited thereto. In one embodiment, the device 1000 is a personalized entity name vocabulary list including a plurality of entity names reflecting personal characteristics of a user from at least one of a search term record in a frequently executed application, a messenger application, and a content streaming application. (NE) can be obtained.

A program executed by the device 1000 described through the present disclosure may be implemented as a hardware component, a software component, and/or a combination of a hardware component and a software component. A program can be executed by any system capable of executing computer-readable instructions.

Software may include a computer program, code, instructions, or a combination of one or more of them, and configure the processing unit to operate as desired or process it independently or collectively. You can command the device.

The software may be implemented as a computer program including instructions stored in a computer-readable storage media. The computer-readable recording medium includes, for example, a magnetic storage medium (e.g., read-only memory (ROM), random-access memory (RAM), floppy disk, hard disk, etc.) and optical reading medium (e.g., CD-ROM (CD-ROM) and DVD (Digital Versatile Disc)). The computer-readable recording medium is distributed over networked computer systems, so that computer-readable codes can be stored and executed in a distributed manner. The medium is readable by a computer, stored in memory, and executed on a processor.

The computer-readable storage medium may be provided in the form of a non-transitory storage medium. Here,'non-transient' means that the storage medium does not contain a signal and is tangible, but does not distinguish between semi-permanent or temporary storage of data in the storage medium.

In addition, a program according to the embodiments disclosed in the present specification may be included in a computer program product and provided. Computer program products can be traded between sellers and buyers as commodities.

The computer program product may include a software program and a computer-readable storage medium in which the software program is stored. For example, a computer program product is a product (for example, a downloadable application) in the form of a software program that is electronically distributed through a device manufacturer or an electronic market (eg, Google Play Store, App Store). It may include. For electronic distribution, at least a part of the software program may be stored in a storage medium or may be temporarily generated. In this case, the storage medium may be a server of a manufacturer, a server of an electronic market, or a storage medium of a relay server temporarily storing a software program.

The computer program product may include a storage medium of a server or a storage medium of a device in a system composed of a server and a device. Alternatively, when there is a server or a third device (eg, a smartphone) that is communicatively connected to the device, the computer program product may include a storage medium of the third device. Alternatively, the computer program product may include a software program itself transmitted from a server to a device or a third device, or transmitted from a third device to the device.

In this case, one of the server, the device and the third device may execute the computer program product to perform the method according to the disclosed embodiments. Alternatively, two or more of a server, a device, and a third device may execute a computer program product to distribute and implement the method according to the disclosed embodiments.

For example, the server may execute a computer program product stored in the server, and control a device communicating with the server to perform the method according to the disclosed embodiments.

As another example, a third device may execute a computer program product, and a device connected in communication with the third device may be controlled to perform a method according to the disclosed embodiment.

When the third device executes the computer program product, the third device may download the computer program product from the server and execute the downloaded computer program product. Alternatively, the third device may perform a method according to the disclosed embodiments by executing a computer program product provided in a pre-loaded state.

As described above, although the embodiments have been described by the limited embodiments and drawings, various modifications and variations are possible from the above description by those of ordinary skill in the art. For example, the described techniques are performed in a different order from the described method, and/or components such as a computer system or module described are combined or combined in a form different from the described method, or other components or equivalents Even if substituted or substituted by, appropriate results can be achieved.

Claims (21)

In the method for the device to recognize voice input,
Using a vocabulary list including a plurality of named entities, the probability that a subword extracted from each of the plurality of entity names can be predicted as a word or word string representing the entity name is determined. Generating a Weighted Finite State Transducer model by training;
Receiving a user's voice input;
Using a first decoding model, a feature vector including a probability value that the received speech input can be predicted as a specific subword is obtained, and a plurality of prediction strings are included using the probability value of the feature vector Obtaining a first character string to perform;
A word sequence corresponding to at least one entity name among the plurality of entity names from the feature vector by inputting the obtained feature vector into a second decoding model using the weighted finite state converter model obtaining a second character string including a (word sequence) and an unrecognized word sequence that is not identified by the at least one entity name; And
Outputting text corresponding to the voice input by replacing the unrecognized word string among the second strings with a word string included in the first string;
Containing, method. The method of claim 1,
Generating the weighted finite state transformer model,
Obtaining a vocabulary list including the plurality of entity names;
Segmenting the words or strings constituting the plurality of entity names into subwords, which are phoneme or syllable units; And
A posterior probability value at which the subword can be predicted as a specific entity name among the plurality of entity names by learning a weight through a state transition using the frequency of the subwords and the arrangement order of the subwords obtaining a confidence score including (posterior probability);
Containing, method. The method of claim 2,
Generating the weighted finite state transformer model,
Performing filtering to remove an entity name overlapping with a word previously stored in a memory of the device among a plurality of entity names included in the obtained vocabulary list;
Containing, method. The method of claim 2,
The weighted finite state converter model is input when a Lexicon Finite State Transducer (L FST) including mapping information that is a probability value that the sub word can be predicted as a specific word and the specific word or word string are input A Grammar Finite State Transducer (G FST) including weight information for predicting a word sequence that may be arranged after the word or word sequence,
Wherein the weighted finite state transformer model is generated through synthesis of the Lexicon finite state transformer and the grammar finite state transformer. The method of claim 1,
The first decoding model is an end-to-end ASR model (End-to-End Automatic Speech Recognition). The method of claim 1,
Generating the weighted finite state transformer model,
Classifying the plurality of entity names according to a plurality of different domains; And
Generating a plurality of weighted finite state transformer models for each of the plurality of domains using the classified plurality of entity names;
Containing, method. The method of claim 6,
Identifying words corresponding to an entity name from an application executed through the device or a web page provided through the device; And
By comparing the identified words with a plurality of entity names included in the vocabulary list of each of the plurality of weighted finite state converter models generated for each of the plurality of domains, the currently executing application or the domain in which the web page can be classified is determined. Determining;
The method further comprising. The method of claim 1,
Receiving update information of a vocabulary list including at least one of addition of a new entity name, deletion of entity name, and change of entity name from the server; And
Updating the vocabulary list using the update information;
Including more,
The method of generating the weighted finite state transformer model comprises generating the weighted finite state transformer model through learning using the updated vocabulary list. The method of claim 1,
Recognizing that the device enters a new area by acquiring location information of the device;
Transmitting information on entering a new region of the device to a server of an application service provider; And
Receiving a Point of Interest (POI) vocabulary list including an entity name of at least one of a place name, a tourist attraction, a tourist destination, and a famous restaurant of the new area from the server of the application service provider;
Including more,
The generating of the weighted finite state transformer model comprises generating the weighted finite state transformer model through learning using an entity name included in the received place of interest vocabulary list. The method of claim 1,
Generating the weighted finite state transformer model,
Learning using a personalized vocabulary list including a plurality of entity names reflecting user characteristics from at least one of frequently executed applications through the device, log data of messenger applications, and search term records in content streaming applications Generating the weighted finite state transformer model through. In the device for recognizing voice input,
A voice input unit for receiving a voice input from a user;
A memory storing a program including one or more instructions; And
A processor that executes one or more instructions of a program stored in the memory;
Including,
The processor,
Using a vocabulary list including a plurality of named entities, the probability that a subword extracted from each of the plurality of entity names can be predicted as a word or word string representing the entity name is determined. By training, a Weighted Finite State Transducer model is created,
Receiving the voice input from the voice input unit,
Obtaining a feature vector including a probability value that the received speech input can be predicted as a specific subword using a first decoding model, and including a plurality of prediction strings using the probability value of the feature vector Get the first string,
An entity name word corresponding to at least one entity name among the plurality of entity names from the feature vector by inputting the obtained feature vector into a second decoding model using the weighted finite state converter Acquiring a second character string including a word sequence and an unrecognized word sequence not identified by the at least one entity name,
A device for obtaining a text corresponding to the voice input by replacing the unrecognized word string among the second strings with a word string included in the first string. The method of claim 11,
The processor obtains a vocabulary list including the plurality of entity names, divides a word or string constituting the plurality of entity names into subwords in phoneme or syllable units, and performs the subword By learning the weight through a state transition using the frequency of and the order of subword arrangement, the posterior probability value that the subword can be predicted as a specific entity name among the plurality of entity names. A device for generating the weighted finite state transformer model by obtaining a confidence score including ). The method of claim 12,
The processor, of the plurality of entity names included in the obtained vocabulary list, performs filtering to remove entity names that overlap with words previously stored in the memory of the device. The method of claim 12,
The weighted finite state converter model is input when a Lexicon Finite State Transducer (L FST) including mapping information that is a probability value that the sub word can be predicted as a specific word and the specific word or word string are input A Grammar Finite State Transducer (G FST) including weight information for predicting a word sequence that may be arranged after the word or word sequence,
The device, wherein the weighted finite state transformer model is generated through synthesis of the Lexicon finite state transformer and the grammar finite state transformer. The method of claim 11,
The first decoding model is an end-to-end ASR model (End-to-End Automatic Speech Recognition). The method of claim 11,
The processor classifies the plurality of entity names according to a plurality of different domains, and generates a plurality of weighted finite state converter models for each of the plurality of domains using the classified plurality of entity names, device. The method of claim 16,
The processor identifies words corresponding to an entity name from an application being executed or a web page being accessed, and converts the identified words into a vocabulary list of each of the plurality of weighted finite state converter models generated for each of the plurality of domains. A device for determining a domain in which the currently running application or the web page can be classified by comparing it with a plurality of included entity names. The method of claim 11,
A communication interface for transmitting and receiving data with a server;
Including more,
The processor,
Receive update information of a vocabulary list including at least one of addition of a new entity name, deletion of entity name, and change of entity name from the server using the communication interface,
The device for generating the weighted finite state converter model through learning using the updated vocabulary list and updating the vocabulary list using the update information. The method of claim 11,
A location sensor that obtains location information of the device; And
A communication interface for transmitting and receiving data with a voice assistant server or an external server;
Including more,
The processor recognizes that the device enters a new area using the location sensor,
In response to recognizing the entry into the new area, a place of interest including an entity name of at least one of a place name, a tourist attraction, a tourist destination, and a famous restaurant of the new area from the server of the application service provider through the communication interface ( A device for receiving a Point of Interest (POI) vocabulary list and generating the weighted finite state transformer model through learning using an entity name included in the received place of interest vocabulary list. The method of claim 11,
The processor,
Learning using a personalized vocabulary list including a plurality of entity names reflecting user characteristics from at least one of frequently executed applications through the device, log data of messenger applications, and search term records in content streaming applications And generating the weighted finite state transformer model through. A computer-readable recording medium storing a program for executing the method of claim 1 on a computer.

Images