KR20210019930A - Server supporting speech recognition of device and operating method thereof - Google Patents

Abstract

A server according to one embodiment of the present disclosure comprises: a memory storing one or more instructions; a processor executing the one or more instructions stored in the memory; and a communication unit receiving a first character string from a device. The processor may identify a plurality of estimated character strings from the first character string, obtain a second character string based on the plurality of estimated character strings, and control the communication unit so as to transmit the second string to the device. The first character string is outputted through voice recognition processing from a voice signal inputted to the device.

Description

Server supporting voice recognition of device and its operation method {SERVER SUPPORTING SPEECH RECOGNITION OF DEVICE AND OPERATING METHOD THEREOF}

The present disclosure relates to a server supporting voice recognition of a device and a method of operating the same. Specifically, it relates to a method of enhancing a speech recognition result by using post-processing at the server side.

As electronic devices that perform various functions in combination are developed, electronic devices equipped with a voice recognition function are being released to improve operability. The voice recognition function has the advantage of being able to easily control a device by recognizing a user's voice without using a separate button operation or touching a touch module.

According to such a voice recognition function, for example, it is possible to perform a call function or write a text message without pressing a separate button on a portable terminal such as a smart phone and a home appliance such as a TV and a refrigerator. Various functions such as alarm setting can be easily set.

In recent years, as artificial intelligence (AI) technology develops, artificial intelligence technology is also applied to the speech recognition function, enabling fast and accurate speech recognition for various utterances. The artificial intelligence system is a computer system that implements human-level intelligence, and unlike the existing rule-based smart system, the machine learns, judges, and becomes smarter. As artificial intelligence systems are used, the recognition rate improves and users' tastes can be more accurately understood, and the existing rule-based smart systems are gradually being replaced by deep learning-based artificial intelligence systems.

On-device speech recognition, in which automatic speech recognition (ASR) is performed on the device side, has an advantage that latency is short and can be used even when the network is not connected. On the other hand, server-based speech recognition has an advantage that speech recognition is performed based on a large number of named entities stored in the server.

In order to take advantage of both on-device voice recognition and server-based voice recognition, there is a need for a device to selectively use both methods.

According to an embodiment of the present disclosure, a server includes: a memory for storing one or more instructions; A processor that executes the one or more instructions stored in the memory; And a communication unit receiving a first character string from a device, wherein the processor identifies a plurality of estimated character strings from the first character string, obtains a second character string based on the plurality of estimated character strings, and the second character string The communication unit may be controlled to transmit a signal to the device, and the first character string may be output through a speech recognition process from a voice signal input to the device.

According to an embodiment of the present disclosure, a method of operating a server may include: receiving a first character string from a device; Identifying a plurality of estimated character strings from the first character string; Obtaining a second character string based on the plurality of estimated character strings; And transmitting the second character string to the device, wherein the first character string is output through a speech recognition process from a voice signal input to the device.

According to an embodiment of the present disclosure, a device includes: a memory that stores one or more instructions; A processor that executes the one or more instructions stored in the memory; And a communication unit communicating with a server, wherein the processor performs speech recognition on a voice signal to obtain a first character string, determines whether to replace the first character string with another character string, and based on the determination The communication unit to control the communication unit to transmit the first character string to the server, and to receive a second character string obtained by replacing at least one character in the first character string with another character by the server from the server. It can be characterized by controlling.

According to an embodiment of the present disclosure, a method of operating a device may include: acquiring a first character string by performing voice recognition on a voice signal; Determining whether to replace the first character string with another character string; Transmitting the first character string to a server based on the determination; And receiving a second character string from the server, wherein the second character string is obtained by replacing at least one character in the first character string with another character by the server.

1 is a diagram for describing a comparison between on-device voice recognition and server-based voice recognition.
2A is a diagram of a speech recognition system according to an embodiment.
2B is a diagram illustrating a speech recognition system according to an embodiment.
2C is a diagram illustrating a speech recognition system according to an embodiment.
3 is a block diagram of a device according to an embodiment.
4A is a detailed block diagram of a device according to an embodiment.
4B is a detailed block diagram of a device according to an embodiment.
5A is a diagram for describing a method of determining that a device performs on-device voice recognition, according to an exemplary embodiment.
5B is a diagram illustrating a method of determining that a device performs server-based voice recognition, according to an exemplary embodiment.
6 is a diagram for describing a frame-synchronized character string according to an exemplary embodiment.
7 is a block diagram of a server according to an embodiment.
8A is a diagram illustrating a method for a server to support voice recognition of a device according to an exemplary embodiment.
8B is a diagram illustrating a method of determining a replacement character string by obtaining a likelihood for each character corresponding to each voice signal frame by a server according to an exemplary embodiment.
9 is a detailed block diagram of a server according to an embodiment.
10A is a diagram illustrating a structure of an artificial intelligence recurrent neural network (RNN) used to calculate a posterior probability according to an embodiment.
10B illustrates an example of a confusion matrix used to calculate a likelihood according to an embodiment.
FIG. 11A is a diagram illustrating a process of calculating a likelihood matrix for replacement characters to be replaced by each character for each character in a first character string received from a device, according to an exemplary embodiment.
11B is a diagram illustrating a process of calculating a likelihood matrix for replacement characters to be replaced by each character for each character in a first character string received from a device according to an exemplary embodiment.
12 is a block diagram of a device that selectively uses two voice recognition modules according to an embodiment.
13 is a flowchart illustrating a method of performing voice recognition by a device according to an exemplary embodiment.
14 is a flowchart of a specific method of performing voice recognition by a device according to an embodiment.
15 is a flowchart illustrating a method of operating a server according to an exemplary embodiment.
16 is a flowchart of a detailed operation method of a server according to an embodiment.
FIG. 17 is a diagram for explaining weighted finite state transducer (WFST) decoding performed in a server according to an embodiment.
18 illustrates an example of a screen on which a voice recognition result is displayed on a device according to an embodiment.
19 is a detailed block diagram of a device according to an embodiment.

Terms used in the embodiments of the present disclosure have selected general terms that are currently widely used as possible while taking the functions of the present disclosure into consideration, but this may vary according to the intention or precedent of a technician working in the field, the emergence of new technologies, and the like. In addition, in certain cases, there are terms arbitrarily selected by the applicant, and in this case, the meaning of the terms will be described in detail in the description of the corresponding embodiment. Therefore, the terms used in the present disclosure should be defined based on the meaning of the term and the contents of the present disclosure, not the name of a simple term.

When a part of the specification is said to "include" a certain component, it means that other components may be further included rather than excluding other components unless otherwise stated. In addition, terms such as "... unit" and "... module" described in the specification mean a unit that processes at least one function or operation, which is implemented as hardware or software, or as a combination of hardware and software. Can be.

In the present disclosure, "text" refers to a symbol used to express and write human language in a visible form. For example, characters may include Korean characters, alphabets, Chinese characters, numbers, pronunciation marks, punctuation marks, and other symbols.

In the present disclosure, the term "string" refers to a sequence of characters.

In the present disclosure, “grapheme” is the smallest unit representing a sound, composed of at least one character. For example, in the case of an alphabetic notation system, one character may be a character element. Accordingly, in the present disclosure, a character may refer to a character place, and in the present disclosure, a character string may refer to an arrangement of character elements. Also, in the present disclosure, a character string may be referred to as text.

A "morpheme" is the smallest unit that has a meaning, consisting of at least one letter. A "word" is the smallest basic unit of a language that can be used independently or represents a grammatical function, composed of at least one morpheme. A "phoneme" is a unit of sound that distinguishes one word from another in human language.

The speech recognition model according to an embodiment of the present disclosure may convert a speech signal into a character string and output it. A character string output by the speech recognition model according to an embodiment of the present disclosure may be a “frame synchronized character string”. The term "frame" may mean a unit in which an audio signal is divided at regular time intervals for processing an audio signal, or a divided audio signal itself. In the present disclosure, the term "frame-synchronized character string" refers to a character string including characters individually corresponding to each of the frames of the speech signal when the speech signal is converted into a character string by the speech recognition model and output.

For example, the speech recognition model receives a speech signal that the user pronounces "baseball", and is frame-synchronized string [b, b, a,, a, a, s, s, e,, b, b , a, a, l] can be printed.

In the present disclosure, when the speech recognition model outputs a predetermined character string from a speech signal, "the reliability of the predetermined character string" indicates how accurately the speech recognition model outputting the predetermined character string performs speech recognition. For example, the reliability of a predetermined character string may be calculated according to a predetermined equation based on a likelihood obtained from a predetermined character string, a partial likelihood output in the process of estimating the predetermined character string, or a posterior probability value. As the reliability of the predetermined character string is higher, it may be determined that the predetermined character string is accurately estimated from the speech recognition model.

In the present disclosure, "evaluation information for a predetermined character string" may mean information on a predetermined character string used by the server to recommend and output another character string having a higher reliability than the predetermined character string according to an embodiment. For example, the evaluation information for a predetermined character string may include the likelihood of a plurality of estimated character strings obtained from the predetermined character string. For example, the server according to an embodiment may select and output another character string that maximizes likelihood from among a plurality of estimated character strings.

On the other hand, in the present disclosure, "likelihood" means a possible degree, and "the likelihood of event B with respect to event A" is a conditional probability P indicating the likelihood that event B will occur when event A occurs. It can mean (B|A).

In the present disclosure, when the speech recognition model outputs a predetermined character string from a speech signal, "likelihood obtained from the predetermined character string" means the likelihood of a plurality of estimated character strings estimated from the predetermined character string. The plurality of estimated character strings estimated from the predetermined character string may mean a plurality of character strings obtained by replacing at least one character in the predetermined character string with another character.

More specifically, if the character string that is output when speech recognition is correctly performed is referred to as a ground truth character string, the "likelihood obtained from a predetermined character string" is a speech when it is assumed that each of the plurality of estimated character strings is a true value character string. As a result of recognition, it may mean a possibility that a predetermined character string is estimated. According to an embodiment of the present disclosure, the "likelihood obtained from a predetermined character string" may include likelihood matrices regarding replacement characters to which each character is replaced with respect to each character in a predetermined character string.

According to an embodiment of the present disclosure, "the likelihood obtained from a predetermined character string" is to identify replacement characters having a similar pronunciation to each character in a predetermined character string, and at least one character in a predetermined character string based on the identified replacement characters. Can be used to determine estimated strings corrected to other characters. Furthermore, among the determined estimated character strings, the most suitable estimated character string may be selected based on pre-stored information such as language model and dictionary information and recommended instead of a predetermined character string.

When the speech recognition model performs speech recognition, the result of speech recognition performed previously may affect the result of speech recognition performed later. When a predetermined character is misrecognized as another character having a similar pronunciation, as the linguistic information is misrecognized due to the misrecognition, the probability that the characters following the predetermined character are also misrecognized may increase. That is, when a predetermined character is misrecognized as another character, words determined by combining the predetermined character and subsequent characters, and words determined by combining other misrecognized characters and subsequent characters may be different.

Accordingly, the device or server according to an embodiment of the present disclosure may use the likelihood obtained from the predetermined character string in order to obtain a replacement character string by considering and decoding pronunciation information and language information for a predetermined character string.

In the present disclosure, "a likelihood matrix obtained for a predetermined character" may mean a matrix including likelihood values for replacement characters to which a predetermined character is to be replaced. "The likelihood value for a replacement character to be replaced with a predetermined character" may mean a probability that a predetermined character is estimated as a result of speech recognition when the replacement character is assumed to be a ground truth character. For example, for a character "a" included in a character string obtained as a result of speech recognition, a probability value of "a" as a ground truth character, a probability value of "b", a probability value of "c", .. ., and a likelihood matrix [0.4 0.01 0.01 0.01 0.2 ... 0.01] including a probability value of "z" may be obtained. In obtaining a likelihood matrix including likelihood values for replacement characters corresponding to each character included in a character string, a high likelihood value may be assigned to replacement characters having a similar pronunciation to each character.

In the present disclosure, "the likelihood obtained from a predetermined character string" may be obtained from likelihood values for replacement characters corresponding to each character in the predetermined character string. At this time, likelihood values for replacement characters corresponding to each character in a predetermined character string may be calculated in consideration of previously accumulated characters of each character. However, the present disclosure is not limited thereto, and the likelihood values for replacement characters corresponding to each character in a predetermined character string may be calculated by considering only each character itself, without considering the previously accumulated characters of each character. .

According to an embodiment of the present disclosure, in consideration of the previously accumulated characters of each character in a predetermined character string, "the likelihood obtained from the predetermined character string" is "posterior probabilities of each character included in the predetermined character string" And a "sequence probability" of a predetermined character string.

The "post-probability" of event A means the conditional probability that event A is expected given the event, observational facts, or background knowledge associated with event A.

In the present disclosure, when the speech recognition model outputs a character string from a speech signal, "the posterior probabilities of a predetermined character in the character string" are, when considering the previous characters of the predetermined character in the character string, the speech recognition model should accurately predict the predetermined character Probability and probability of incorrectly predicting another character as a given character may be included.

In the present disclosure, when the speech recognition model outputs a character string from a speech signal, the "character arrangement probability of a character string" may mean a probability that characters are arranged according to the corresponding character string.

According to an embodiment of the present disclosure, considering only each character in a predetermined character string, "the likelihood obtained from a predetermined character string" is calculated from a "confusion matrix" including probability values that each character is incorrectly predicted. Can be. In the present disclosure, the term "error matrix" is also referred to as an error matrix, and when the speech recognition model converts and outputs a speech signal into a predetermined character string, the speech recognition model accurately predicts a predetermined character included in the predetermined character string. It includes the probability and the probability of incorrectly predicting another character as a given character. For example, for characters having similar pronunciation to a predetermined character, a high probability that characters having similar pronunciation are erroneously predicted as a predetermined character by a speech recognition model may be given.

In the present disclosure, the term “sound model” may mean a model including information capable of determining which character or pronunciation symbol is matched with a speech signal in units of letters. For example, the device according to an embodiment of the present disclosure may calculate a probability that each of a voice signal and characters are matched based on an acoustic model.

In the present disclosure, "dictionary information" may include mapping information of a plurality of words and characters included in each of the plurality of words. The "language model" may be an artificial intelligence model in which a relationship between words is learned so that a probability of the next word appearing can be estimated when a specific word sequence is given.

In the present disclosure, the term "artificial neural network" is a generic term for a computing system implemented with the focus on a neural network of a human or animal brain. Artificial neural networks are one of the detailed methodologies of machine learning, and are in the form of a network in which multiple neurons, which are neurons, are connected. Artificial neural networks may be implemented in hardware, but are mainly implemented in computer software. An artificial neural network is a form in which several neurons, which are basic computing units, are connected by weighted links. A weighted link can adjust its weight to adapt to a given environment.

Artificial neural network is a generic term for various models such as Self-Organizing Map (SOM), Recurrent Neural Network (RNN), and Convolutional Neural Network (CNN). To come.

In the present disclosure, a "domain" refers to a set of words related to a certain attribute as a domain of the attribute.

In the present disclosure, "an operation of correcting a first character string" means an operation of recommending and outputting a second character string having a higher reliability than the first character string by replacing at least one character included in the first character string with another character. can do. Accordingly, in the present disclosure,'correction of a character string','correction of a character','replace a predetermined character with another character','recommend another character instead of a predetermined character','replace a predetermined character string with another character string', and ' The expression'recommending another character string instead of a predetermined character string' may be substituted and used.

A device or server included in the voice recognition system according to an embodiment of the present disclosure may provide a "voice assistant service" to a user. The voice assistant service may be a service that provides a conversation with a user. The voice assistant service may provide a response message to a user as if a person had a direct conversation with the user in consideration of the user's situation and the device's situation. In addition, the voice assistant service can appropriately generate and provide information required by the user, like the user's personal assistant. The voice assistant service includes, for example, a broadcasting service, a content sharing service, a content providing service, a power management service, a game providing service, a chat service, a document writing service, a search service, a call service, a photo shooting service, a transportation method recommendation service, and In connection with various services such as a video playback service, information or functions required by the user can be provided to the user.

Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art may easily implement the exemplary embodiments. However, the present disclosure may be implemented in various different forms and is not limited to the exemplary embodiments described herein.

Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the drawings.

1 is a diagram for describing a comparison between on-device voice recognition and server-based voice recognition.

On-device voice recognition means that voice recognition for the user's 10 utterance is performed on the device 100 side, and server-based voice recognition is the utterance of the user 10 received from the device 100 It means that voice recognition for is performed on the server 200 side.

With the development of end-to-end speech recognition and compression technology, on-device speech recognition technology is gradually developing, and the performance difference from server-based speech recognition is gradually decreasing. In particular, there is almost no difference in performance between the device and the server in speech recognition or general dictation for open domain utterances that are not limited to a specific field. General dictation refers to dictation for utterances that do not correspond to a domain based on a named entity. The entity name may include a specific place name, a person name, a device name, a brand name, and the like. With a domain, a set of words related to a certain attribute is called a domain of the attribute.

On-device speech recognition has a latency of less than about 50ms, less than that of server-based speech recognition, which is hundreds of ms, and can be used in suburbs, inside airplanes, or in areas where there is no network connection. There is an advantage. In addition, on-device voice recognition is more advantageous in security and privacy invasion issues, and has the advantage of reducing the cost of managing the server.

On the other hand, server-based speech recognition has the advantage of being implemented in a server capable of storing more entity names such as geographical names, human names, and brand names than devices.

Therefore, according to server-based speech recognition, a higher weight can be given to words related to a new buzzword or a new song title, and when speech recognition is not possible, the defect in speech recognition is repaired by adding the word to the dictionary. The advantage is that hot-fix is possible. In addition, there is an advantage in that rescoring of the speech recognition result is possible by using a language model and dictionary information optimized for a third-party application operating in a server.

Therefore, speech recognition for general purposes such as dictation, general commands, subtitle generation, etc., is performed by the device, but when speech recognition is to be performed using a language model and dictionary information corresponding to a specific domain, a hybrid ( hybrid) method of speech recognition is required.

In this case, when the device and the server separately perform the entire speech recognition process, a dependency may occur between the device and the server.

As an example, a method in which a calculation for applying an acoustic model to speech is performed in a device, and a decoding calculation in which a language model and dictionary information are applied to an intermediate value extracted from the acoustic model is performed in a server may be used. According to this method, since there is a dependency between the device and the server, there is a problem that it cannot be used between a device and a server that are not compatible with each other.

As another example, in an end-to-end speech recognition method including an encoding calculation and a decoding calculation process, only the encoding calculation is performed in the device and the decoding calculation for the encoded data is performed in the server. Can be used. In order to perform the decoding calculation, prior information on the encoding method is required, so there is a dependency between the device performing the encoding and the server performing the decoding. Accordingly, there is a problem that such a method cannot be used between devices and devices that are not compatible with each other.

In order to solve the above-described problem, a speech recognition system according to an embodiment of the present disclosure illustrated in FIG. 2A is proposed.

The device 100 according to an embodiment of the present disclosure may perform on-device voice recognition for converting a voice signal into a first character string. The device 100 may determine whether the on-device voice recognition has failed based on the reliability of the first character string. When the on-device voice recognition fails, the device 100 may transmit a first character string, which is a result of voice recognition, to the server 200.

According to an embodiment of the present disclosure, the device 100 transmits information on the voice signal to the server 200 in the form of a character string, so that the relationship between the type of on-device voice recognition that the device 100 uses. There is an advantage that the server 200 can process a character string without it.

According to an embodiment of the present disclosure, the first character string transmitted from the device 100 to the server 200 may be a frame synchronized character string.

A'frame' may mean a unit in which a voice signal is divided at regular time intervals for processing a voice signal, or a divided voice signal itself. "Frame-synchronized character string" means a character string including characters individually corresponding to each of the frames of the speech signal when the speech signal is converted into a character string by the speech recognition model and output.

The device 100 according to an embodiment may generate a frame-synchronized character string as a result of speech recognition using an algorithm such as RNN-T or CTC.

However, the present disclosure is not limited thereto, and the device 100 according to an embodiment performs a forced alignment even when the result of the voice recognition of the device 100 is not frame-synchronized, Can be created. A specific method of generating a frame-synchronized character string and a frame-synchronized character string by forced alignment will be described later with reference to FIG. 6.

The device 100 according to an embodiment of the present disclosure may use the result of performing the voice recognition as it is when the confidence score for the result of performing voice recognition using on-device voice recognition is sufficiently high.

On the other hand, if the device 100 according to an embodiment of the present disclosure determines that the reliability of the result of performing the speech recognition using on-device speech recognition is not sufficiently high, the server 200 ).

Therefore, when it is determined that the reliability of the result of performing speech recognition using on-device speech recognition is not sufficiently high, the device 100 according to an embodiment of the present disclosure transmits a speech signal to the server 200 Since the speech recognition process is not restarted in the server 200 from the beginning, there is an advantage in that the processing time can be reduced.

When it is determined that the reliability of a result of performing speech recognition using on-device speech recognition is not sufficiently high, the device 100 according to an embodiment of the present disclosure may perform a sentence unit, a word unit, a phrase unit, or a frame unit. A character string resulting from speech recognition may be transmitted to the server 200.

The device 100 according to an embodiment transmits all characters included in the sentence or phrase to the server 200 when acquiring a string constituting a sentence or phrase by performing voice recognition, or Some of the characters included in the sentence or phrase may be transmitted to the server 200. The device 100 may transmit some characters with low reliability to the server 200 based on the reliability of the character string.

The device 100 according to an embodiment may receive a corrected character string from the server 200 and may combine a character string that has not been transmitted to the server 200 because it is determined that there is no need for calibration, with the corrected character string. The device 100 according to an embodiment may output a combined character string or provide a voice assistant service based on a result of analyzing the combined character string.

The server 200 according to an embodiment of the present disclosure may receive a character string, which is a result of speech recognition, from the device 100 in units of sentences, words, phrases, or frames.

The server 200 according to an embodiment of the present disclosure may correct an error in the received first character string by using the language model and dictionary information stored in the server 200. The server 200 may obtain a second character string from the first character string by using a language model in the server 200 including a larger amount of information than the language model stored in the device 100. The server 200 may obtain the second character string by replacing at least one character included in the first character string with another character. The second character string may be a character string in which an error included in the first character string is corrected.

In the present disclosure, the server 200 according to an embodiment corrects the first character string by replacing at least one character included in the first character string received from the device 100 with another character, and corrects the first character string. May be transmitted to the device 100.

The “operation of correcting the first character string” may refer to an operation of recommending and outputting a second character string having a higher reliability than the first character string. Accordingly, in the present disclosure,'correction of a character string','correction of a character','replace a predetermined character with another character','recommend another character instead of a predetermined character','replace a predetermined character string with another character string', and ' The expression'recommending another string instead of a predetermined string' may be used interchangeably.

The server 200 according to an embodiment, when a character string constituting a sentence or phrase is obtained from the device 100, corrects characters included in the sentence or phrase, or Some of these can be calibrated. The server 200 may correct some characters with low reliability based on the reliability of the character string.

The server 200 according to an embodiment may combine a text string that has not undergone a calibration process and a corrected text string because it is determined that calibration is not required. The server 200 according to an embodiment may transmit the combined character string to the device 100.

The server 200 according to an embodiment of the present disclosure may perform decoding on a received character string using different dictionary information and language models for each domain. According to an embodiment of the present disclosure, since dictionary information is stored in the server 200, there is an advantage that a new word or a new entity name can be easily hot-fixed.

The server 200 according to an embodiment may receive a character string from the device 100 and select a domain related to the received character string. As an example, the server 200 may receive domain information related to the character string together with the character string from the device 100, and determine a domain to perform decoding on the character string based on the received information. As another example, the server 200 may determine a domain related to the received character string based on the character string received from the device 100. The server 200 according to an embodiment may include advance information corresponding to the determined domain. And using the language model, it is possible to perform decoding on the received character string.

Accordingly, the server 200 according to an embodiment of the present disclosure may output a speech recognition result with improved speech recognition accuracy through re-decoding of a character string received from the device 100. For example, the server 200 receives at least one character included in the first character string by receiving a first character string from the device 100 and performing decoding using the language model and dictionary information in the server 200 The corrected second character string may be output.

The server 200 may transmit the second character string to the device 100. The device 100 may improve speech recognition performance by receiving and using a second character string having higher reliability than the first character string from the server 200.

When the server 200 according to an embodiment obtains a character string including characters constituting a sentence from the device 100, an error is corrected for the entire sentence or an error is corrected for some of the characters constituting the sentence. Can be corrected. The server 200 may correct errors for some characters with low reliability based on the reliability of the character string. The server 200 according to an embodiment may obtain the second character string by combining the uncorrected character string with the corrected character string because it is determined that there is no need for calibration.

As shown in FIG. 2A, the server 200 according to an embodiment of the present disclosure may transmit the second character string itself to the device 100 as a result of voice recognition. However, the embodiment is not limited to the example shown in FIG. 2A.

As shown in FIGS. 2B and 2C, the server 200 according to an exemplary embodiment of the present disclosure recognizes the user's utterance intention for the second character string, and thereby transmits information related to the voice assistant service based on the second character string to the device ( 100).

The server 200 according to an embodiment of the present disclosure may provide various types of voice assistant services to the device 100 by using the second character string obtained from the first character string. The voice assistant service may be a service that provides a conversation with a user. The voice assistant service may provide a response message to a user as if a person had a direct conversation with the user in consideration of the user's situation and the device's situation. In addition, the voice assistant service can appropriately generate and provide information required by the user, like the user's personal assistant.

In this case, the server 200 uses a Natural Language Understanding (NLU) model, a Dialog Mananer (DM) model, and a Natural Language Generating (NLG) model in the server 200 to provide a voice assistant service based on a character string. Thus, information for conducting a conversation with the user may be provided to the device 100.

As an example, the server 200 may control the device 100 or another device (eg, a smart home appliance, a wearable device, etc.) based on a result of analyzing the second character string.

As shown in FIG. 2B, the server 200 according to an embodiment provides a control command for controlling the device 100 or a control command for controlling the device 100 to control other devices based on the result of analyzing the character string. A control command may be generated and the generated control command may be provided to the device 100.

In addition, as shown in FIG. 2C, the server 200 according to an embodiment of the present disclosure may provide a voice assistant service related to various services. For example, broadcast service, content sharing service, content providing service, power management service, game providing service, chat service, document writing service, search service, call service, photo shooting service, transportation recommendation service, or video playback service. Various services and voice assistant services are linked, so that the voice assistant service can provide the information or functions required by the user.

The server 200 according to an embodiment of the present disclosure may transmit information related to the voice assistant service to the device 100 based on the second character string. The information related to the voice assistant service may include a response message provided to the user or information required by the user as if a person directly communicates with the user in consideration of the situation of the user and the device.

In addition, the server 200 may determine the user's utterance intention based on the second character string, and may request the service providing server 201 to provide a service required by the user. The service providing server 201 is a broadcasting service, a content sharing service, a content providing service, a power management service, a game providing service, a chat service, a document creation service, a search service, a call service, a photo shooting service, a transportation method recommendation service, or a video. At least one of the reproduction services may be provided.

In FIG. 2C, it is illustrated that the server 200 providing the voice assistant service is connected to one service providing server 201, but the present disclosure is not limited to that shown in FIG. 2C. For example, according to an embodiment of the present disclosure, the server 200 is connected to a plurality of service providing servers, and may determine a service required by the user according to the user's speech intention. The server 200 may select a service providing server corresponding to the determined service and transmit a service provision request to the selected service providing server.

The service providing server 201 according to an embodiment may provide information related to the requested service based on a service request received from the voice assistant service providing server 200. For example, the service providing server 201 may provide broadcast, content, application, transportation means recommendation information, and search results, as information related to the requested service. The service providing server 201 may provide information related to the requested service to the voice assistant service providing server 200 or the device 100.

Hereinafter, according to an embodiment of the present disclosure, the device 100 for requesting correction of the string by selectively transmitting the string as a result of voice recognition to the server 200 and the server 200 for performing correction on the received string Each configuration and operation method will be described in detail.

3 is a block diagram of a device according to an embodiment.

The device 100 according to an embodiment of the present disclosure may be a fixed terminal implemented as a computer device or a mobile terminal. The device 100 includes, for example, a smart phone, a mobile phone, a navigation device, a computer, a notebook computer, a digital broadcasting terminal, an artificial intelligence speaker, a speaker, a personal digital assistant (PDA), a portable multimedia player (PMP), and It may be at least one of the tablet PCs, but is not limited thereto. The device 100 may communicate with other devices and/or servers through a network using a wireless or wired communication method.

Referring to FIG. 3, the device 100 may include a reception unit 110, a processor 120, a communication unit 130, a memory 140, and an output unit 150. Not all of the components shown in FIG. 3 are essential components of the device 100. The device 100 may be implemented by more components than the components shown in FIG. 3, or the device 100 may be implemented by fewer components than the components shown in FIG. 3. For example, as shown in FIG. 19, the device 100 according to some embodiments may further include a user input unit 2100, a sensing unit 2400, and an A/V input unit 2600.

The receiver 110 according to an embodiment of the present disclosure may receive a voice signal from a user. For example, the receiving unit 110 may receive a voice signal by converting external sound into electrical sound data using a microphone. In FIG. 3, the receiving unit 110 is shown to be included in the device 100, but the receiving unit 110 according to another embodiment is included in a separate device and is connected to the device 100 by wire or wirelessly. It can be implemented in the form of

The memory 140 according to an embodiment of the present disclosure may store instructions for performing speech recognition, various models used for speech recognition, neural networks, dictionary information, and the like.

The processor 120 according to an embodiment of the present disclosure may perform speech recognition by executing one or more instructions stored in the memory 140.

The processor 120 according to an embodiment of the present disclosure may obtain the first character string as a result of speech recognition for a speech signal.

For example, the first character string may be a frame-synchronized character string including characters corresponding to each of the audio signal frames in which the audio signal is divided at predetermined time intervals. Alternatively, the first character string may be a character string obtained through a label synchronization method so as to include one character for each character pronounced by a voice signal.

Next, the processor 120 according to an embodiment of the present disclosure determines whether to replace the first character string with another character string, and sends the first character string to the server 200 through the communication unit 130 based on the determination. Can be transmitted. The processor 120 according to an embodiment may transmit the first character string to the server 200 in a sentence unit, a word unit, a phrase unit, or a frame unit. The processor 120 according to an embodiment transmits all the characters included in the sentence or phrase to the server 200 or includes them in the sentence or phrase when acquiring a string constituting a sentence or phrase by performing voice recognition. Some of the characters may be transmitted to the server 200. The processor 120 may transmit some characters with low reliability to the server 200 based on the reliability of the character string.

Determining whether to replace the first character string with another character string may mean determining that speech recognition has failed and determining that the first character string is replaced with another character string. Alternatively, determining whether to replace the first character string with another character string may mean that the server determines whether to replace the first character string with another character string obtained by additionally performing voice recognition.

As an example, the processor 120 may determine the reliability of the first character string, and determine whether to replace the first character string with another character string based on the reliability.

The reliability of the first character string may be calculated based on at least one of a likelihood of a plurality of estimated character strings obtained from the first character string and posterior probabilities in which at least one character in the first character string is replaced with another character.

For example, the processor 120 may calculate the reliability based on the likelihood output as a result of Viterbi decoding. Alternatively, the processor 120 may calculate the reliability based on posterior probabilities output from the softmax layer in the end-to-end speech recognition model.

Alternatively, the processor 120 according to an embodiment may determine a plurality of estimated character strings estimated during a speech recognition process for a speech signal, and calculate the reliability of the first character string based on the correlation between the plurality of estimated character strings. have. The higher the correlation between the plurality of estimated strings including the first string, the higher the reliability of the first string.

As another example, the processor 120 may determine whether to replace the first character string with another character string based on a result of comparing the first character string with keywords previously stored in the device 100. For example, when keywords previously stored in the first string are not included, the processor 120 may determine to replace the first string with another string.

As another example, the processor 120 may determine whether to replace the first character string with another character string, based on whether a domain to which the first character string is related or whether an entity name is included in the first character string. For example, when it is determined that the first character string is related to a domain based on an entity name or is not related to an open domain, the processor 120 may determine to replace the first character string with another character string.

When it is determined that the first character string should be replaced with another character string, the processor 120 according to an embodiment of the present disclosure controls the communication unit 130 to transmit the first character string to the server 200 based on this determination. can do.

Meanwhile, the communication unit 130 according to an embodiment of the present disclosure may communicate with an external device, device, or server through wired communication or wireless communication. The communication unit 130 may include a short-range communication module, a wired communication module, a mobile communication module, and a broadcast receiving module.

The processor 120 according to an embodiment generates a frame-synchronized character string by performing a forced alignment on the first character string and transmits it to the server 200 when the result of the speech recognition for the speech signal is not a frame-synchronized character string. I can.

The processor 120 according to an embodiment may identify a voice signal section in which each character included in the first character string is pronounced, and identify a plurality of voice frames included in the identified voice signal section. The processor 120 may obtain a frame-synchronized character string by sequentially arranging the corresponding character a plurality of times according to the identified voice frames.

For example, when the pronunciation time of a predetermined character included in the first character string is n frames (n is a natural number), the processor 120 may obtain a frame synchronized character string by sequentially arranging n predetermined characters. .

The communication unit 130 may receive a second character string from the server 200. The second character string is a character string obtained by replacing at least one character in the first character string with another character by the server 200. In addition, the communication unit 130 may receive a response message generated by the server 200 based on the interpretation of the second character string from the server 200.

When it is determined that the correction of the first character string is not required, the processor 120 according to an embodiment of the present disclosure may determine not to replace the first character string with another character string. The processor 120 according to an embodiment of the present disclosure may output the first character string through the output unit 150 when the first character string is not replaced with another character string.

On the other hand, when it is determined that the correction of the first character string is necessary, the processor 120 may determine to replace the first character string with another character string. When it is determined that the first character string should be replaced with another character string, the output unit 150 may output the second character string received from the server 200 instead of the first character string.

According to an embodiment of the present disclosure, the first character string acquired by the device 100 may be a character string acquired based on the first dictionary information and the first language model. According to an embodiment of the present disclosure, the second character string acquired by the server 200 may be a character string acquired based on second dictionary information and a second language model stored in the server 200.

The second dictionary information and the second language model stored in the server 200 may include a larger amount of information than the first dictionary information and the first language model. Accordingly, the second string received from the server 200 may have a higher reliability than the first string. The device 100 may improve speech recognition performance by receiving and using a second character string having higher reliability than the first character string from the server 200.

The output unit 150 according to an embodiment of the present disclosure may output a first character string or a second character string as it is, or may output a word sequence obtained from the first character string or the second character string. For example, when the first character string is a frame-synchronized character string, the output unit 150 may output a word sequence obtained from the first character string.

The output unit 150 according to an embodiment of the present disclosure may output a result of performing speech recognition based on the first character string or the second character string. The output unit 150 may notify a user of a result of performing voice recognition or transmit it to an external device (eg, a smart phone, a home appliance, a wearable device, a server, etc.). For example, the output unit 150 may include a speaker capable of outputting an audio signal or a display capable of outputting a video signal.

Alternatively, the output unit 150 may perform an operation corresponding to a result of performing speech recognition. For example, the device 100 may analyze a first character string or a second character string and determine a function of the device 100 corresponding to the analysis result. The device 100 may output a screen that performs a corresponding function through the output unit 150. Alternatively, the device 100 may transmit a keyword corresponding to the analysis result to an external server, receive information related to the transmitted keyword from the server, and output it on the screen through the output unit 150. Alternatively, the device 100 may generate a response message for the voice signal based on the analysis result, and may output the response message through the output unit 150.

The device 100 according to an embodiment may output information related to the voice assistant service through the output unit 150 by identifying the user's utterance intention through natural language processing on the first character string or the second character string. . The device 100 includes a Natural Language Understanding (NLU) model, a Dialog Mananer (DM) model, and a Natural Language Generating (NLG) model in the device 100 in order to provide a voice assistant service based on a first character string or a second character string. Etc. can be used.

Alternatively, the output unit 150 may receive information related to the voice assistant service based on the second character string from the server 200 and output the received information. For example, information related to the voice assistant service based on the second character string is generated based on the result of analyzing the user’s speech intention through natural language processing on the second character string. May contain control commands. Or, for example, information related to the voice assistant service based on the second character string is provided based on the result of interpreting the user’s speech intention through natural language processing on the second character string, or the user May contain information required by

On the other hand, when the processor 120 according to an embodiment transmits only some of the characters included in the sentence or phrase to the server 200, the processor 120, the corrected character string received from the server 200 and It is determined that calibration is not necessary, and a character string that has not been transmitted to the server 200 may be combined. The processor 120 may output a combined character string, output a result of performing speech recognition based on the combined character string, or provide a voice assistant service based on a result of interpreting the combined character string.

Hereinafter, a method of operating the device 100 will be described in detail with reference to FIGS. 4A and 4B.

4A is a detailed block diagram of a device according to an embodiment.

As shown in FIG. 4A, the ASR module 121 of the processor 120 may receive the voice signal obtained from the receiver 110 and perform voice recognition on the voice signal.

The ASR module 121 of FIG. 4A may perform voice recognition on a voice signal in an end-to-end manner. The end-to-end scheme is a speech recognition scheme using a deep neural network trained to directly map speech signals to strings or word strings. Compared with other speech recognition methods using multiple models such as acoustic models and language models, the end-to-end method can simplify the speech recognition process by using a single trained deep neural network. As a sub-example of an end-to-end speech recognition model, there are an RNN-T model, a CTC model, and the like.

The ASR module 121 can extract a feature vector from the speech signal. The ASR module 121 may output a first character string from a feature vector using a deep neural network (DNN) 144 stored in the memory 140.

The determination unit 125 of the processor 120 according to an embodiment of the present disclosure determines whether to replace the first string with another string based on the reliability of the first string output from the ASR module 121. I can. The determination unit 125 may receive reliability information regarding the first character string from the ASR module 121.

The determiner 125 according to an embodiment may receive a posterior probability value output from the softmax layer in the ASR module 121 as reliability information about the first character string. The determiner 125 may calculate a reliability based on a posterior probability value related to the first character string.

For example, if the reliability is higher than or equal to the threshold value, the determination unit 125 may determine that the correction of the first character string is not required and output the first character string through the output unit 150. On the other hand, if the reliability is less than the threshold value, the determination unit 125 may determine that calibration of the first character string is necessary and transmit the first character string to the server 200 through the communication unit 130.

In FIG. 4A, for convenience of explanation, a case in which the first character string is output through the output unit 150 is illustrated as an example, but the present disclosure is not limited thereto. The device 100 according to an embodiment may output information related to the voice assistant service through the output unit 150 by identifying the user's utterance intention through natural language processing on the first character string.

The device 100 may use a Natural Language Understanding (NLU) model, a Dialog Mananer (DM) model, and a Natural Language Generating (NLG) model in the device 100 to provide a voice assistant service based on the first character string. have.

For example, the processor 120 of the device 100 generates a response message for the first character string to the user and generates a response message to the user as if a person directly communicates with the user in consideration of the situation of the user and the device 150) can be printed. Alternatively, for example, the processor 120 may generate information required by the user based on the first character string and provide it to the user through the output unit 150. Alternatively, for example, the processor 120 may determine the user's utterance intention based on the first character string, and request a service providing server to provide a service required by the user. The output unit 150 may output information received from the service providing server.

In addition, the output unit 150 of the device 100 according to an embodiment may receive information related to the voice assistant service from the server 200 and output the received information. The information related to the voice assistant service may be information generated by the server 200 based on the first character string or the second character string in which the first character string has been corrected. For example, information related to the voice assistant service may include a response message to a user's voice signal, a service required by the user, or information required by the user.

Meanwhile, FIG. 4B is a detailed block diagram of a device according to another exemplary embodiment.

As shown in FIG. 4B, the ASR module 121 of the processor 120 may receive the voice signal obtained from the receiver 110 and perform voice recognition on the voice signal. The phoneme sequence acquisition unit 122 may acquire a phoneme sequence from a speech signal using the acoustic model 141 stored in the memory 140. The acoustic model 141 may divide a waveform of a speech signal and estimate a phoneme sequence including phonemes using a hidden Markov model, a Gaussian mixture model, Bayesian inference, or a multilayer neural network.

The character string acquisition unit 123 of the processor 120 estimates words from the phoneme sequence and outputs a character string including the estimated words based on the dictionary information 142 and the language model 143 stored in the memory 140 can do.

The determination unit 125 of the processor 120 according to an embodiment of the present disclosure calculates the reliability of the first character string output from the ASR module 121, and converts the first character string into another character string based on the calculated reliability. You can decide whether to replace it. The determination unit 125 may receive reliability information regarding the first character string from the ASR module 121.

The determination unit 125 according to an embodiment may calculate the reliability based on a partial likelihood of the first string output from the Viterbi decoder in the ASR module 121 as reliability information on the first string. I can.

If the reliability is higher than or equal to the threshold value, the determiner 125 according to an embodiment may determine that calibration of the first character string is not required and output the first character string through the output unit 150. On the other hand, if the reliability is less than the threshold value, the determination unit 125 may determine that calibration of the first character string is necessary and transmit the first character string to the server 200 through the communication unit 130. In FIG. 4B For convenience of explanation, a case in which the first character string is output through the output unit 150 is illustrated as an example, but the present disclosure is not limited thereto. The device 100 according to an embodiment may output information related to the voice assistant service through the output unit 150 by identifying the user's utterance intention through natural language processing on the first character string.

The device 100 may use a Natural Language Understanding (NLU) model, a Dialog Mananer (DM) model, and a Natural Language Generating (NLG) model in the device 100 to provide a voice assistant service based on the first character string. have.

For example, the processor 120 of the device 100 generates a response message for the first character string to the user and generates a response message to the user as if a person directly communicates with the user in consideration of the situation of the user and the device 150) can be printed. Alternatively, for example, the processor 120 may generate information required by the user based on the first character string and provide it to the user through the output unit 150. Alternatively, for example, the processor 120 may determine the user's utterance intention based on the first character string, and request a service providing server to provide a service required by the user. The output unit 150 may output information received from the service providing server.

In addition, the output unit 150 of the device 100 according to an embodiment may receive information related to the voice assistant service from the server 200 and output the received information. The information related to the voice assistant service may be information generated by the server 200 based on the first character string or the second character string in which the first character string has been corrected. For example, information related to the voice assistant service may include a response message to a user's voice signal, a service required by the user, or information required by the user.

As described above, the device 100 according to an embodiment of the present disclosure may determine whether to replace the first character string with another character string, based on the reliability of the result of speech recognition for the speech signal. However, the present disclosure is not limited to this embodiment, and according to another embodiment, the device 100 converts the first character string to another character string based on a result of comparing the character string with keywords previously stored in the device 100. You can decide whether to replace it with. Alternatively, the device 100 according to another embodiment may determine whether to replace the first character string with another character string based on a domain to which the first character string is associated. Alternatively, the device 100 according to another embodiment may interpret the meaning of the first character string through natural language understanding processing, and determine whether to replace the first character string with another character string based on the analyzed result.

5A is a diagram for describing a method of determining that a device performs on-device voice recognition, according to an exemplary embodiment.

As an example, the determination unit 125 of the processor 120 of the device 100 according to an embodiment of the present disclosure, based on a result of comparing the keywords previously stored in the device 100 with a first character string, It is possible to determine whether to replace the first character string with another character string.

The determiner 125 according to an embodiment may determine not to replace the first string with another string when at least one of keywords previously stored in the first string is included. Accordingly, the device 100 may use the result of performing the speech recognition performed by the ASR module 121 of the device 100 as it is without the intervention of the server 200.

For example, if the first character string output from the ASR module 121 is “Read me my text”, the determination unit 125 determines that the first character string includes a pre-stored keyword “text”, It may be determined not to replace the first string with another string.

As another example, the determination unit 125 of the processor 120 of the device 100 according to an embodiment of the present disclosure may be based on whether the domain to which the first character string is related or the entity name is included in the first character string. You can decide whether to replace 1 string with another string.

The determination unit 125 according to an embodiment may determine not to replace the first string with another string when it is determined that the first string is not related to an entity name-oriented domain and is related to an open domain. . Accordingly, the device 100 may use the result of performing the speech recognition performed by the ASR module 121 of the device 100 as it is without the intervention of the server 200.

For example, when the first character string output from the ASR module 121 is “Take a picture”, the determination unit 125 determines that the first character string is related to the open domain, and sets the first character string to another character string. You can decide not to replace it with.

When it is determined that the object name is included in the first string, the determiner 125 according to an embodiment may determine to replace the first string with another string.

The determiner 125 according to an embodiment of the present disclosure may determine whether at least one of the object names stored in the memory 140 is included in the first character string. Alternatively, without prior information on the entity name, the determination unit 125 according to an embodiment may determine whether the entity name is included in the first string. For example, the determination unit 125 may identify an entity name included in the first string by performing POS Tagging (Part-Of-Speech Tagging) of words identified from the first string.

For example, when the first character string output from the ASR module 121 is “Take a picture”, the determination unit 125 determines that the first character string does not include an entity name, and sets the first character string to another character string. You can decide not to replace it with.

As another example, the determination unit 125 of the processor 120 of the device 100 according to an embodiment of the present disclosure interprets the meaning of the first character string through natural language understanding processing, and based on the analyzed result It is possible to determine whether to replace the first character string with another character string.

When determining that the voice signal is a general command related to the operation of the device 100 as a result of analyzing the first string, the determiner 125 according to an embodiment determines not to replace the first string with another string. I can. Accordingly, the device 100 may use the result of performing the speech recognition performed by the ASR module 121 of the device 100 as it is without the intervention of the server 200.

For example, when the first string output from the ASR module 121 is “Do I have any new voice mail?”, the determination unit 125 determines that the first string is a general command related to the confirmation of the text message. And, it may be determined not to replace the first character string with another character string.

Meanwhile, FIG. 5B is a diagram illustrating a method of determining that a device performs server-based voice recognition, according to an exemplary embodiment.

As shown in FIG. 5B, the determination unit 125 of the processor 120 of the device 100 according to an embodiment determines that the first character string should be replaced with another character string, and the first character string is The character string may be transmitted to the server 200.

5B is an example of a case where the ASR module 121 of the device 100 receives a voice signal in which the user pronounces “The Cardinals baseball team” and acquires the first string [the cat and deers baseball team] Shown as.

As an example, the determining unit 125 of the processor 120 of the device 100 according to an embodiment of the present disclosure may replace the first string with another string because the first string does not include pre-stored keywords. Can decide.

As another example, when the determination unit 125 of the processor 120 of the device 100 according to an embodiment of the present disclosure determines that the first character string is related to a sports domain or is determined to include an entity name, You can decide to replace one string with another string.

The determiner 125 according to an embodiment of the present disclosure may determine whether at least one of the object names stored in the memory 140 is included in the first character string. Alternatively, without prior information on the entity name, the determination unit 125 according to an embodiment may determine whether the entity name is included in the first string. For example, the determination unit 125 may identify an entity name included in the first string by performing POS Tagging (Part-Of-Speech Tagging) of words identified from the first string. However, embodiments are not limited thereto, and various methods of Named Entity Recognition (NER) methods may be used.

As another example, the determination unit 125 of the processor 120 of the device 100 according to an embodiment of the present disclosure determines that the voice signal is not a general command as a result of analyzing the first character string, and the first You can decide to replace the string with another string.

As shown in FIG. 5B, the determination unit 125 of the device 100 according to an embodiment determines that the first character string should be replaced with another character string, and based on this determination, the first character string is replaced by the server 200. ). The server 200 may receive the first character string from the device 100 and perform decoding using a language model and dictionary information (eg, dictionary information of a sports domain) in the server 200. As a result of decoding, the server 200 may obtain a second character string in which at least one character included in the first character string has been modified. The device 100 may increase the accuracy of speech recognition by receiving and using the second character string from the server 200.

Meanwhile, when the device 100 according to an embodiment of the present disclosure acquires a string constituting a sentence or phrase by performing voice recognition, all characters included in the sentence or phrase are transmitted to the server 200, or Some of the characters included in the sentence or phrase may be transmitted to the server 200. The determination unit 125 of the processor 120 of the device 100 may determine to transmit some characters with low reliability to the server 200 based on the reliability of the character string.

The device 100 according to an embodiment may receive a corrected character string from the server 200 and may combine a character string that has not been transmitted to the server 200 because it is determined that there is no need for calibration, with the corrected character string. The device 100 according to an embodiment may output a combined character string, output a result of performing speech recognition based on the combined character string, or provide a voice assistant service based on a result of interpreting the combined character string. I can.

In addition, the device 100 according to an embodiment may provide the server 200 with domain information related to the first string of the device 100 while requesting the server 200 to calibrate the first string. The domain information is information for identifying a domain, and may include, for example, a name of a domain and an identifier of a domain, but is not limited thereto. The device 100 may identify a domain related to the first character string based on the domain reliability of the first character string output from the ASR model of the device 100. The domain reliability may be a number indicating to what extent at least a part of the first character string is related to a specific domain. For example, the device 100 may calculate a confidence score indicating to what extent the first character string output from the ASR model is related to a domain previously registered in the device 100. Also, the device 100 may identify a domain related to the first character string based on the calculated domain reliability. The device 100 may identify a domain related to the first character string based on a rule, or obtain a domain reliability level related to the first character string by using an artificial intelligence model trained for domain identification.

6 is a diagram for describing a frame-synchronized character string according to an exemplary embodiment.

As shown in FIG. 6, the ASR module 121 of the device 100 according to an embodiment of the present disclosure corresponds to each of the voice signal frames F in which the voice signal 601 is divided at predetermined time intervals. A frame-synchronized character string 603 including characters to be displayed may be output.

For example, the ASR module 121 receives a voice signal pronounced by the user as "baseball", and frame-synchronized strings [b, b, a,, a, a, s, s, e,, b , b, a, a, l] can be printed.

However, the present disclosure is not limited thereto, and the ASR module 121 according to an embodiment may output a text string that is not frame synchronized (ie, a label synchronization text string) as a result of speech recognition. Even in this case, the device 100 may generate a frame-synchronized character string by forcibly aligning the character string obtained from the voice signal.

The processor 120 of the device 100 according to an embodiment may identify a voice signal section in which each character included in the first string is pronounced, and identify a plurality of voice frames included in the identified voice signal section. have. The processor 120 may obtain a frame-synchronized character string by sequentially arranging the corresponding character a plurality of times according to the identified voice frames.

For example, the ASR module 121 may output a first character string [b, a, s, e, b, a, l, l], which is a character string that is not frame synchronized. In this case, the processor 120 may consecutively arrange each character a plurality of times based on a time when each character included in the first character string is pronounced. As a result, the processor 120 may acquire frame-synchronized character strings [b, b, a,, a, a, s, s, e,, b, b, a, a, l].

The device 100 according to an embodiment of the present disclosure may output the frame-synchronized character string 603 to the server 200. The server 200 may decode the frame-synchronized character string 603 received from the device 100 and transmit the obtained second character string to the device 100 based on a result of performing the decoding.

7 is a block diagram of a server according to an embodiment.

The server 200 according to an embodiment of the present disclosure may be connected to the device 100 by wire or wirelessly.

Referring to FIG. 7, the server 200 may include a communication unit 210, a processor 220, and a memory 230. Not all of the components shown in FIG. 7 are essential components of the server 200. The server 200 may be implemented by more components than the components shown in FIG. 7, or the server 200 may be implemented by fewer components than the components shown in FIG. 7.

The memory 230 of the server 200 according to an embodiment of the present disclosure may store instructions for performing speech recognition, various models used for speech recognition, neural networks, dictionary information, and the like.

The processor 220 according to an embodiment of the present disclosure may perform speech recognition by executing one or more instructions stored in the memory 230.

Meanwhile, the communication unit 210 according to an embodiment of the present disclosure may communicate with an external device or an apparatus through wired communication or wireless communication. The communication unit 210 may include a short-range communication module, a wired communication module, a mobile communication module, a broadcast receiving module, and the like.

First, the communication unit 210 of the server 200 according to an embodiment of the present disclosure may receive a first character string from the device 100. The first character string may be output through a voice recognition process by the device 100 from a voice signal input to the device 100.

As an example, the first character string received from the server 200 may be a frame synchronization character string including characters corresponding to each of the audio signal frames in which the audio signal is divided at predetermined time intervals. As another example, the first character string received from the server 200 may be a character string that is not frame synchronized.

The processor 220 according to an embodiment of the present disclosure may obtain a frame-synchronized character string from the first character string when the first character string received from the device 100 is a character string that is not frame synchronized. The processor 220 may obtain a frame-synchronized character string by continuously arranging at least one character included in the first character string in units of frames at a predetermined time interval.

The processor 220 of the server 200 according to an embodiment of the present disclosure may obtain a second character string from the first character string by replacing at least one character in the first character string with another character.

According to an embodiment of the present disclosure, the processor 220 identifies replacement characters having a similar pronunciation to each character in the first string, and at least one character in the first string is converted into a different character based on the identified replacement characters. Corrected estimated character strings can be determined. The processor 220 may select the most suitable estimated character string from among the determined estimated character strings based on pre-stored information such as language model and dictionary information, and obtain it as the second character string.

Hereinafter, a method of obtaining a second character string by the processor 220 according to an exemplary embodiment will be described in more detail.

First, the processor 220 may identify a plurality of estimated character strings from the first character string. The processor 220 may calculate likelihood matrices for replacement characters to be substituted for each character in the first character string. The processor 220 may identify a plurality of estimated character strings in which at least one character in the first character string is replaced with another character based on likelihood values in the likelihood matrices.

The processor 220 according to an embodiment may calculate the likelihood of a plurality of estimated character strings from the first character string. The processor 220 may calculate the likelihood of a plurality of estimated character strings based on likelihood values in likelihood matrices for replacement characters to which each character is to be replaced with respect to each character in the first character string.

The likelihood obtained from the first character string may mean a probability that the first character string is estimated as a result of speech recognition, assuming that each of the plurality of estimated character strings is a true value character string. According to an embodiment of the present disclosure, the processor 220 identifies replacement characters having a similar pronunciation to each character in the first string, and at least one character in the first string is different based on the identified replacement characters. In order to determine the estimated character strings corrected to, the likelihood obtained from the first character string can be used.

The processor 220 may obtain a second character string, which is one of a plurality of estimated character strings, based on the likelihood, dictionary information, and language model. The processor 220 may determine whether to replace the first character string with the second character string based on the calculated likelihood. Based on the determination, the processor 220 may obtain a second character string from the first character string by replacing at least one character in the first character string with another character.

The processor 220 according to an embodiment may obtain a likelihood from the first character string through a process described later.

As an example, the processor 220 may calculate posterior probabilities of each character based on previous characters of each character in the first character string. The posterior probabilities of the predetermined character in the first character string may include probabilities in which the predetermined character is replaced by a plurality of other characters when previous characters are considered. That is, the posterior probabilities of a predetermined character are the probability that the ASR module of the processor 120 of the device 100 correctly predicts the predetermined character when considering the previous characters of the predetermined character in the character string, and the other character incorrectly as a predetermined character It can contain the probability of predicting.

Next, the processor 220 may calculate a character arrangement probability of the first character string. The character arrangement probability of a character string may mean a probability that characters are arranged according to a corresponding character string. The character arrangement probability may be calculated based on characters accumulated before each character of the character string. The processor 220 may calculate the likelihood of a plurality of estimated character strings obtained from the first character string based on the posterior probabilities of each character and the character arrangement probability.

In calculating the posterior probabilities, the processor 220 according to an embodiment comprises a recurrent neural network (RNN) including a plurality of long-short term memory (LSTM) layers and a softmax layer. Can be used. A regression neural network used to calculate posterior probabilities will be described in more detail later with reference to FIG. 10A.

As another example, the processor 220 may calculate posterior probabilities of each character in the first character string, based on a predetermined confusion matrix. The processor 220 may calculate the likelihood of a plurality of estimated character strings obtained from the first character string based on the posterior probabilities of each character. The error matrix used to calculate posterior probabilities will be described in more detail later with reference to FIG. 10B.

As another example, the processor 220 may calculate posterior probabilities of each character in the first string based on predetermined probability values. The processor 220 may determine a probability that the first character included in the first character string is actually the first character as P. P may be a predetermined value. P may be a value of 0 or more and 1 or less. Further, the processor 220 may determine a probability that the first character included in the first character string is actually a character other than the first character as (1-P)/(N-1). N means the number of characters, and may be a natural number. That is, the processor 220 determines the probability that the ASR module of the processor 120 of the device 100 correctly predicts the first character in the first character string as P, and the probability that another character is incorrectly predicted as the first character. Can be determined as (1-P)/(N-1).

For example, the processor 220 determines the probability that the first character included in the first character string is actually the first character as 0.9, and the probability that the first character is actually another character is 0.1/(N-1) Can be determined by

The processor 220 according to an embodiment may include a likelihood calculator that calculates the likelihood of a plurality of estimated strings obtained from the first string. Further, the processor 220 may include a decoder that obtains a second character string from the likelihood by using dictionary information and a language model. The processor 220 may obtain the second character string by performing re-decoding of the likelihood obtained from the first character string using the dictionary information and the language model.

As an example, the decoder of the processor 220 may obtain the second character string based on dictionary information and a language model stored in the server 200. The decoder may output the second character string as the likelihood of the plurality of estimated character strings obtained from the first character string is input. For example, the decoder of the processor 220 may include a weighted finite state transducer (WFST) decoder.

When the processor 220 performs WFST decoding, the server 200 according to an embodiment includes a relationship between characters (T), dictionary information (L) including mapping information of words and characters, and a specific word string. Based on the language model (G) that estimates the probability of the next word given when is, it is possible to construct and decode a search space with WFST.

As another example, the decoder of the processor 220 may recalculate the likelihood of a plurality of estimated character strings obtained from the first character string based on the dictionary information and the language model. The decoder may determine a second character string that maximizes the recalculated likelihood from among the plurality of estimated character strings. For example, the decoder of the processor 220 may include a Viterbi decoder. The Viterbi decoder may find the most likely character string for the first character strings as the second character string in consideration of the dictionary information and the language model.

The communication unit 210 according to an embodiment of the present disclosure may transmit the second character string to the device 100. Alternatively, the communication unit 210 may transmit a response message to the voice signal generated by the processor 220 to the device 100. The processor 220 may interpret the second character string using a natural language understanding model, and generate a response message for the speech signal based on the result of the analysis.

The processor 220 may determine the type of the response message by applying a dialog manager (DM) model to the result of analyzing the second character string. The processor 220 may generate a response message of a determined type using a Natural Language Generation (NLG) model and transmit it to the device 100.

Alternatively, the communication unit 210 may transmit information related to the voice assistant service generated based on the second character string to the device 100. In order to provide a voice assistant service based on the second character string, the processor 220 uses the NLU model, the DM model, and the NLG model in the server 200 to provide information for conducting a conversation with the user. ) Can be provided. Also, the processor 220 may generate a control command for controlling the device 100 or another device based on the result of analyzing the second character string, and provide the generated control command to the device 100.

Hereinafter, with reference to FIG. 8A, a method of supporting voice recognition of the device 100 in each component of the server 200 according to an embodiment will be described. 8A illustrates an example in which the user of the device 100 utters “The Cardinals baseball team”.

First, the device 100 may estimate a first character string called [The cat and deers baseball team] by performing voice recognition on the user's voice signal.

The device 100 converts the first string to another string based on the reliability of the first string, the domain to which the first string is related, the result of analyzing the meaning of the first string, or whether the object name is included in the first string. You can decide whether to replace it with. A detailed method of determining whether or not the device 100 performs server-based voice recognition to replace the first character string has been described above with reference to FIGS. 4A to 5B, and thus a duplicate description will be omitted.

In FIG. 8A, the device 100 may determine that it is necessary to replace the first character string with another character string, and transmit the first character string [The cat and deers baseball team] to the server 200.

In transmitting the first character string to the server 200, the device 100 according to an embodiment may transmit information related to a voice signal together with the first character string. The device 100 according to an embodiment may transmit information related to the length of a voice signal frame represented by each character in the first character string together with the first character string. For example, the device 100 may transmit a first character string synchronized to the voice signal frame to the server 200. The character string synchronized with the audio signal frame may mean a character string including characters corresponding to each of the audio signal frames in which the audio signal is divided at predetermined time intervals.

However, the present disclosure is not limited to an embodiment in which the device 100 transmits a frame-synchronized character string to the server 200. The device 100 according to an embodiment may transmit a first character string that is not frame-synchronized to the server 200. The first character string that is not frame-synchronized may mean a character string obtained through a label synchronization method to include one character for each character pronounced by a voice signal.

The device 100 according to an embodiment may transmit a first string that is not frame-synchronized to the server 200, but may transmit information related to a voice signal together with the first string. The server 200 may generate a frame-synchronized character string by forcibly aligning the first character string based on information related to the voice signal. For example, the information related to the speech signal may include information on a speech signal section in which the speech recognition model of the device 100 has obtained the first character string.

The server 200 according to an embodiment of the present disclosure may receive the first character string from the device 100 through the communication unit 210. The server 200 according to an embodiment may receive a frame-synchronized first character string. However, as described above, the server 200 may receive a first character string that is not frame synchronized. In this case, the server 200 may receive information related to the voice signal together with the first character string obtained from the voice signal by the device 100 from the device 100. The server 200 may generate a frame-synchronized first character string by forcibly aligning the first character string based on information related to the voice signal.

The processor 220 may identify a plurality of estimated character strings from the first character string and obtain a second character string based on the plurality of estimated character strings.

According to an embodiment, the processor 220 identifies replacement characters whose pronunciation is similar to that of each character in the first string, and at least one character in a predetermined string is corrected to another character based on the identified replacement characters. You can decide. The processor 220 may select the most suitable estimated character string from among the determined estimated character strings based on pre-stored information such as language model and dictionary information, and obtain it as the second character string.

Hereinafter, a method of obtaining a second character string by the processor 220 according to an exemplary embodiment will be described in more detail.

First, the processor 220 may calculate the likelihood of a plurality of estimated character strings obtained from the first character string.

The first character string estimated from the speech signal by the device 100 is obtained by considering a language model and dictionary information stored in the device 100 with respect to a probability distribution in which speech signal frames correspond to each of arbitrary characters. . The server 200 removes the bias related to the language model and dictionary information of the device 100 from the first character string estimated by the device 100, and re-reads the language model and dictionary information stored in the server 200. Decoding can be performed.

The server 200 may calculate the likelihood of a plurality of estimated character strings obtained from the first character string in order to remove a bias based on the language model and dictionary information of the device 100 from the first character string.

The processor 220 may obtain the second character string by performing decoding by applying dictionary information and a language model stored in the memory 230 on the likelihood obtained from the first character string. When the processor 220 performs decoding by applying the dictionary information and language model stored in the memory 230 of the server 200, it is possible to use dictionary information and language models including a vast amount of object names, and thus speech recognition The accuracy of the can be increased.

For example, the entity name "Cardinals" may not be stored in the language model in the memory of the device 100. Accordingly, the device 100 may erroneously estimate the first character string [The cat and deers baseball team] from the voice signal “The Cardinals baseball team”.

On the other hand, as shown in FIG. 8A, in the memory 230 of the server 200, the entity name “Cardinals” of the sports domain may be stored. Accordingly, the processor 220 of the server 200 may determine that the probability that the'cat and deers' estimated by the device 100 is actually the baseball team name'Cardinals' is higher.

The processor 220 identifies replacement characters having a similar pronunciation to each character in the first string, and obtains a second string in which at least one character in the first string is corrected to another character based on the identified replacement characters. I can. Accordingly, the processor 220 may acquire a second string [The Cardinals baseball team] in which'cat and deers' in the first string is replaced with'Cardinals'. For example, a specific method of acquiring the second string [The Cardinals baseball team] from the first string [The cat and deers baseball team] using the WFST decoding method will be described later with reference to FIG. 17.

The server 200 may transmit the second character string to the device 100. The device 100 may replace and output the first character string estimated by the device 100 with the second character string received from the server 200. As shown in FIG. 8A, for example, the reliability of the first string [The cat and deers baseball team] may be 0.1 and the reliability of the second string [The Cardinals baseball team] may be 0.5. The device 100 according to an embodiment may increase speech recognition performance by receiving and using a second character string having higher reliability than the first character string from the server 200.

As described above, the server 200 according to an embodiment may receive a frame-synchronized character string from the device 100 or generate a frame-synchronized character string from the character string received from the device 100. The server 200 may determine a replacement character string by acquiring the likelihood of each character corresponding to each voice signal frame. The server 200 may simultaneously receive the entire character string including a plurality of characters, or may sequentially receive at least some characters included in the character string.

Hereinafter, a method of determining a replacement character string by obtaining a likelihood for each character corresponding to each voice signal frame by the server 200 according to an embodiment will be described in more detail with reference to FIG. 8B.

The server 200 according to an embodiment may receive a frame-synchronized first character string from the device 100 or may generate a frame-synchronized first character string from the character string received from the device 100.

For example, the communication unit 210 of the server 200 may receive information related to the voice signal from the device 100 together with a character string acquired from the voice signal by the device 100. The server 200 may generate a frame-synchronized first character string by forcibly aligning character strings based on information related to the voice signal.

The character string evaluation unit 221 of the server 200 may calculate a likelihood matrix 813 for replacement characters to be replaced with each character for each character in the frame-synchronized first character string.

According to an embodiment, the likelihood matrix for a predetermined character calculated by the character string evaluation unit 221 may mean a matrix including likelihood values for replacement characters to which the predetermined character is to be replaced. The likelihood value for a replacement character to be replaced by a predetermined character may mean a probability that a predetermined character is estimated as a result of speech recognition when the replacement character is assumed to be a ground truth character.

For example, for a character “a” included in a character string obtained as a result of speech recognition, a probability value of “a” as a ground truth character, a probability value of “b”, a probability value of “c”,… , And likelihood matrix containing probability values of “z” [0.4 0.01 0.01 0.01 0.2. 0.01] can be obtained. In obtaining a likelihood matrix including likelihood values for replacement characters corresponding to each character included in a character string, a high likelihood value may be assigned to replacement characters having a similar pronunciation to each character.

The decoder 223 of the server 200 selects one estimated character string from among a plurality of estimated character strings in which at least one character in the frame-synchronized first character string has been replaced, based on the likelihood matrices 813, The selected estimated character string may be obtained as a second character string.

For example, the decoder 223 may re-compute the likelihood matrices 813 based on the dictionary information and the language model. The decoder 223 may determine a second character string that maximizes the recalculated likelihood from among the plurality of estimated character strings. For example, the decoder 223 may include a Viterbi decoder. The Viterbi decoder may find the most likely character string for the first character string as the second character string in consideration of dictionary information and a language model.

The decoder 223 of the server 200 selects the character string 817 with the highest reliability among the plurality of estimated character strings 815 based on the likelihood of the plurality of estimated character strings, dictionary information, and a language model. Can be obtained as. The server 200 may transmit the second character string to the device 100. The device 100 may improve speech recognition performance by receiving and using a second character string having higher reliability than the first character string from the server 200.

Hereinafter, various embodiments of a method for calculating the likelihood in the server 200 will be described in detail with reference to FIGS. 9 to 11B.

9 is a detailed block diagram of a server according to an embodiment.

As illustrated in FIG. 9, the communication unit 210 of the server 200 may receive a first character string from the device 100.

The character string evaluation unit 221 of the processor 220 may output evaluation information on the first character string that enables the decoder 223 to recommend and output a second character string having a higher reliability than the first character string. For example, evaluation information on the first character string may include a likelihood calculated from the first character string.

The character string evaluation unit 221 may calculate the likelihood of a plurality of estimated character strings obtained from the first character string. The character string evaluation unit 221 may obtain a plurality of estimated character strings by replacing each character in the first character string with a different character. The likelihood of the plurality of estimated character strings may mean a probability that the first character string is estimated from the speech recognition module, assuming that each of the plurality of estimated character strings obtained from the first character string is a true value character string.

The likelihood obtained from the first character string output from the character string evaluation unit 221 identifies replacement characters having a similar pronunciation to each character in the first character string, and at least one character in a predetermined character string based on the identified substitution characters. Can be used to determine estimated strings corrected to other characters.

The character string evaluation unit 221 calculates likelihood matrices for replacement characters to be replaced for each character in the first character string, and identifies a plurality of estimated character strings based on likelihood values in the likelihood matrices. can do. The character string evaluation unit 221 may output likelihood matrices obtained from each character as the likelihood of a plurality of estimated character strings.

The character string evaluation unit 221 may calculate the likelihood from the first character string by using the likelihood calculation data 231 stored in the memory 230. For example, the likelihood calculation data 231 may include a neural network trained for likelihood calculation or an error matrix.

As an example, the character string evaluation unit 221 may calculate posterior probabilities of each character based on previous characters of each character in the first character string. The character string evaluation unit 221 may calculate a character arrangement probability from the first character string. The character string evaluation unit 221 may calculate the likelihood of a plurality of estimated character strings obtained from the first character string, based on the posterior probabilities of each character and the character arrangement probability.

As another example, the character string evaluation unit 221 may calculate posterior probabilities of each character in the first character string based on a predetermined error matrix. The character string evaluation unit 221 may calculate the likelihood of a plurality of estimated character strings obtained from the first character string, based on posterior probabilities of each character in the first character string.

After calculating the likelihood in the character string evaluation unit 221, the decoder 223 may obtain the second character string based on the calculated likelihood using dictionary information and a language model. The decoder 223 may obtain a second character string having a maximum likelihood from among a plurality of estimated character strings obtained by replacing at least one in the first character string with another character.

The decoder 223 may obtain a second character string in which at least one character in the first character string is replaced with another character using the dictionary information 232 and the language model 233. For example, the decoder 223 may include a WFST decoder that takes likelihood as an input or a Viterbi decoder that uses traditional token passing.

According to an embodiment of the present disclosure, the dictionary information stored in the server 200 may be dictionary information storing a relationship between a word and a character string, not general dictionary information storing a relationship between a phoneme sequence and a word. In addition, the language model may be an artificial intelligence model in which a relationship between words is learned so as to estimate the probability of words that appear next when a specific word sequence is given. For example, the language model may be a neural network such as an RNN or a statistical n-gram.

The communication unit 210 may transmit the second character string to the device 100. However, the embodiment of the present disclosure is not limited to the embodiment of transmitting the second character string to the device 100. The server 200 according to an embodiment of the present disclosure recognizes the user's utterance intention through natural language processing on the second character string, and thereby transmits information related to the voice assistant service based on the second character string to the device ( 100).

Information related to the second character string transmitted from the server 200 to the device 100 according to various embodiments of the present disclosure has been described above with reference to FIGS. 2B and 2C, and thus a detailed description thereof will be omitted.

Meanwhile, the decoder 223 of the server 200 according to an embodiment of the present disclosure may decode a first character string using different dictionary information and language models for each domain. Accordingly, the server 200 according to an embodiment of the present disclosure may output a speech recognition result with improved speech recognition accuracy through re-decoding of the first character string from the device 100.

The processor 220 of the server 200 according to an embodiment may receive a first character string from the device 100 and determine a domain related to the first character string. The decoder 223 of the server 200 may decode the first character string using dictionary information and a language model corresponding to the determined domain.

As an example, the processor 220 of the server 200 receives domain information related to the first character string together with the first character string from the device 100, and performs decoding on the first character string based on the received information. You can decide which domain to do. For example, the processor 220 may determine a domain that is identical or similar to a domain identified from information received from the device 100 as a domain to perform decoding.

As another example, the processor 220 of the server 200 may determine a domain related to the received first character string based on the first character string received from the device 100. Although not shown in FIG. 9, the server 200 may store a domain identification model, which is an artificial intelligence model trained for domain identification, in the memory 230. The processor 220 may output the domain reliability using the first character string as an input value using the domain identification model. The processor 220 may determine a domain related to the first character string based on the domain reliability. According to an embodiment, the string evaluation unit 221 or the decoder 223 of the server 200 may determine a domain related to the received first string based on the first string received from the device 100.

For example, the decoder 223 of the server 200 may determine a domain related to the first character string based on the first character string received from the device 100. The decoder 223 according to an embodiment may decode the received first character string using a dictionary and a language model specialized for the determined domain.

The decoder 223 according to an embodiment may be a two-pass decoder. The two-pass decoder may perform primary decoding on the evaluation information for the first character string received from the character string evaluation unit 221 and then perform secondary decoding using the first decoding result.

In this case, the decoder 223 according to an embodiment of the present disclosure may perform decoding using a general dictionary and a language model by a first pass decoder. In addition, the decoder 223 may decode the received first character string using a dictionary and a language model specialized for a domain determined by a second pass decoder.

As another example, the communication unit 210 of the server 200 according to an embodiment may receive information for determining a domain related to the first string together with the first string from the device 100. For example, information for determining a domain received from the device 100 may include context information. For example, the context information may include at least one of information on an application currently being used by the user on the device 100 or the server 200, conversation history information, situation information around the device 100, or trend information. I can. The processor 220 of the server 200 may determine a domain for decoding the first character string based on context information. Hereinafter, a specific method of determining a domain based on context information will be described.

For example, the processor 220 may determine the domain based on the application currently being used by the user. The processor 220, when the user is using the map application on the device 100 or the server 200, determines the domain for the character string obtained from the user's utterance, the domain to decode the domain related to the map It can be determined as For example, the processor 220 may determine a domain to perform decoding by assigning a higher weight to the map domain, or determine the map domain as a domain to perform decoding.

Alternatively, for example, the processor 220 may determine a domain based on conversation history information. When it is determined that the user's conversation history is related to'music', the processor 220 may determine a domain related to music as a domain to perform decoding in determining a domain for a character string obtained from the user's speech. For example, the processor 220 may determine a domain to perform decoding by assigning a higher weight to the music domain, or determine the music domain as a domain to perform decoding.

Alternatively, for example, the processor 220 may determine the domain based on context information around the device 100 detected by a sensor mounted on the device 100. The processor 220 may determine the domain based on the location of the device 100 identified using GPS information of the device 100. When the user wants to search for a restaurant, the processor 220 may determine a domain related to the location of the device 100 as a domain to perform decoding. When the location of the device 100 is near a movie theater, the processor 220 may determine a domain related to a movie as a domain to perform decoding.

Alternatively, for example, the processor 220 may determine a domain based on trend information. The processor 220 may determine a domain related to major news or a real-time search word through a portal site as a domain to perform decoding.

Hereinafter, a case in which the character string evaluation unit 221 of the server 200 according to an exemplary embodiment acquires a likelihood based on previously accumulated characters of each character in the first character string will be described in detail.

The communication unit 210 of the server 200 according to an embodiment may receive a frame-synchronized first string y _o [0:l+1] from the device 100. Regarding the frame-synchronized character string, since it has been described above with reference to FIG. 6, redundant descriptions are omitted.

In the following description, y _o [L] may be a frame-synchronized character estimated from a speech signal by the on-device speech recognition module. The frame-synchronized text may mean a text estimated from one voice frame included in the voice signal. y _o [L] is included in V, the set of all characters.

y _o [0:L+1] means the sequence of _{y o [L'] when 0≤L'≤L.} L and L'are indexes of the character string.

The communication unit 210 may simultaneously receive the entire character string including a plurality of characters, or may sequentially receive at least some characters included in the character string.

y _p [L] refers to a frame-synchronized character estimated to post-process the character string acquired by the device in the server. y _p [L] is included in V, which is a set of characters. W _i is a string of words. W _i is a word included in D, which is a set of words.

The character string evaluation unit 221 of the server 200 according to an embodiment, a _{character arrangement probability P(y o} [0:L+1] in which characters are to be listed according to _{the first character string y o [0:L+1])} ) Can be calculated. The character arrangement probability P(y _o [0:L+1]) can be calculated from a character-level language model.

_{When the first character string y o} [0:L+1] is estimated by the device 100, the character string evaluating unit 221 according to an exemplary embodiment may have a _{posterior probability that the L-th character is actually y p} [L]. S P(y _p [L]|y _o [0:L+1]) can be calculated. The character string evaluation unit 221, based on the first character string y _o [0:L+1], _{the posterior probabilities of the character y o} [L] P(y _p [L]|y _o [0:L+1] ]) can be calculated. That is, the posterior probability calculation unit 225, a first character string _{y o [0: L + 1} ] on the basis of, character by a device (100) y _o [L] is the probability and the letter is correctly estimate y _o [L We can calculate the probabilities that] is incorrectly estimated.

The string evaluation unit 221 according to an embodiment may calculate posterior probabilities of each character of the first string from the first string using a neural network.

The string evaluation unit 221 according to an embodiment may calculate the posterior probabilities of each character in the first string using a regression neural network including the LSTM layer 1010 and the softmax layer 1030 shown in FIG. 10A. have.

The LSTM layer 1010 illustrated in FIG. 10A may include a plurality of stacked LSTM layers. In FIG. 10A, a first character string is input to the LSTM layer 1010, data output from the LSTM layer 1010 is input to the softmax layer 1030, and the softmax layer 1030 is each character of the first character string. We can output the posterior probabilities of.

According to an embodiment, a neural network that calculates posterior probabilities of each character in a character string may be trained by learning a true value character string and an error character string output from a speech recognition module. Specifically, when the neural network receives an input of an error string output from a speech recognition module, the neural network may be trained so that the output value approaches the true value string.

The artificial intelligence model used by the character string evaluation unit 221 according to an embodiment to obtain posterior probabilities is, in order to prevent overfitting with the speech recognition result of a specific speech recognition module, the speech of a plurality of speech recognition modules It can be trained based on recognition results.

The character string evaluation unit 221 of the processor 220 includes the posterior probabilities P(y _p [L]|y _o [0:L+1]) and the character arrangement probability P(y _o [0:L+1]) Based on, the likelihood P(y _o [0:L+1]|y _p [L]) can be calculated.

The likelihood P(y _o [0:L+1]|y _p [L]) may be calculated by Equation 1 below based on a posterior probability and a character arrangement probability.

[Equation 1]

In [Equation 1], P(y _p [L]) means a prior probability _{of y p [L].} The prior probability of the predetermined character y _p [L] may be a statistically pre-calculated value based on the frequency at which the predetermined character is used.

The decoder 223 of the server 200 according to an embodiment uses the dictionary information 232 and the language model 233, the likelihood P(y _o [0:L+1]|y _p [L] ), the second character string W _i can be estimated. The second character string may be a character string in which at least one character of the first character string is replaced with another character. The communication unit 210 may transmit the second character string W _i to the device 100. Although the server 200 receives the frame-synchronized character string y _o [0:L+1] _{from the device 100, the server 200 may transmit the second character string W i} having the form of a word string to the device 100.

Meanwhile, the string evaluation unit 221 of the server 200 according to another embodiment may calculate a likelihood by considering only each character itself, without considering the previously accumulated characters of each character. The string evaluation unit 221 according to another embodiment may calculate the likelihood by considering only the _{characters y o [L] instead of the string y o} [0:L+1]. _{When only the character y o} [L] is considered instead of the string y _o [0:L+1], the structure of the server 200 becomes very simple, and since only the character-level error matrix is stored and used instead of a neural network, the calculation process is reduced. It can be simplified.

The communication unit 210 of the server 200 may receive a frame-synchronized first character string y _o [0:L+1] from the device 100. Regarding the frame-synchronized character string, since it has been described above with reference to FIG. 6, redundant descriptions are omitted. The communication unit 210 may simultaneously receive the entire character string including a plurality of characters, or may sequentially receive at least some characters included in the character string.

_{According to another embodiment, when the first character y o} [L] in the first character string is estimated by the device 100, the character string evaluation unit 221 of the server 200 according to another exemplary embodiment is _{actually the L-th character y p} [L] days posterior probabilities P(y _p [L]|y _o [L]) can be obtained. String evaluation unit 221, the character y _o based on [L], the character y _o the posterior probability of the [L] P | may obtain the _{(y p [L] y o} [L]). That is, the character string evaluation unit 221, based on the first character y _o [L], the probability that the character y _o [L] was accurately estimated _{by the device 100 and the character y o} [L] was incorrectly estimated. You can get probabilities.

The character string evaluation unit 221 according to an embodiment may obtain posterior probabilities of each character from the first character string using an error matrix.

10B shows an example of an error matrix 1001 used to calculate posterior probabilities according to an embodiment.

The error matrix 1001 includes a probability that the speech recognition module of the device 100 correctly predicts a predetermined character included in a character string and a probability that another character incorrectly predicts a predetermined character.

For example, since the pronunciation of the letter “a” and the letter “e” are similar, the probability that the speech recognition module incorrectly estimates the actual letter “a” as “e” may be relatively high. On the other hand, since the pronunciation of the letter "a" and the letter "b" are very different, the probability that the speech recognition module incorrectly estimates the actual letter "a" as "b" may be relatively low.

Therefore, as illustrated in FIG. 10B, the probability that the speech recognition module of the device 100 incorrectly estimates the actual character “a” as the character “e” is 0.23, and incorrectly estimates the actual character “a” as the character “b”. The probability of estimation may be 0.01.

When the character estimated by the device 100 is a first character y _o [L], the character string evaluating unit 221 according to an embodiment includes the posterior probabilities P(y _p) when the actual character is y _{p [L].} [L]|y _o [L]) may be obtained by searching from the error matrix 1001 in FIG. 10B.

The character string evaluation unit 221 may calculate the likelihood _{P(y o} [L]|y _p [L]) _{based on the acquired posterior probabilities P(y p} [L]|y _o [L]). have.

The likelihood P(y _o [L]|y _p [L]) can be calculated by the following [Equation 2] based on posterior probabilities.

[Equation 2]

In [Equation 2], P(y _p [L]) denotes _{a prior probability of y p} [L]. The prior probability of the predetermined character y _p [L] may be a statistically pre-calculated value based on the frequency at which the predetermined character is used.

The decoder 223 of the server 200 according to another embodiment uses the dictionary information 232 and the language model 233 to determine the likelihood P(y _o [L]|y _p [L]). 2 Can estimate the string W _i. The second character string may be a character string in which at least one character of the first character string is replaced with another character. The communication unit 210 may transmit the second character string W _i to the device 100. The server 200 receives the frame synchronization string y _o [0:l+1] from the device 100, but may output _{the second string W i having the form of a word string.}

As described above, the character string evaluation unit 221 of the server 200 according to an embodiment may receive a frame-synchronized character string from the device 100 and obtain a probability for each character corresponding to each voice signal frame. have. For example, the character string evaluation unit 221 may have the likelihood P(y _o [0:L+1]|y _p [L]) _{for the character y o [L] of the index L corresponding to the speech signal frame.} P(y _o [L]|y _p [L]) can be calculated.

Hereinafter, referring to FIGS. 11A and 11B, a method of obtaining a likelihood for each character corresponding to each voice signal frame from a character string received from the device 100 by the character string evaluation unit 221 according to an exemplary embodiment. It will be described in detail.

As shown in FIG. 11A, the character string evaluation unit 221 according to an embodiment may receive a frame-synchronized character string 1101. The character string evaluation unit 221 may calculate a likelihood matrix for replacement characters to be replaced with each character for each character.

As shown in FIG. 11B, the likelihood matrix for replacement characters calculated by the string evaluation unit 221 according to an embodiment is expressed as a matrix including likelihood values where a predetermined character is each of arbitrary characters. Can be. As shown in table 1105 of FIG. 11B, each of the arbitrary characters may be mapped to each index of the likelihood matrix.

For example, the value of the _{index a 1} in the likelihood matrix 1103 may represent a likelihood value in which a predetermined character is replaced with the character “a” corresponding to the _{index a 1.} The value of the _{index a 2} in the likelihood matrix 1103 may represent a likelihood value in which a predetermined character is replaced with a character “b” corresponding to the _{index a 2.} Possible value of the index is also a ₃ in matrix 1103, a certain character can represent the value can also be replaced with the letter "c" corresponding to the index ₃ a.

The character string evaluating unit 221 according to an embodiment may calculate likelihood matrices 1107 regarding replacement characters to which each character is to be replaced for each character in the character string. The character string evaluation unit 221 may output the likelihood matrices 1107 calculated as the likelihood of a plurality of estimated character strings in which at least one character in the first character string has been replaced, to the decoder 223.

Based on the likelihood received from the character string evaluation unit 221, the decoder 222 according to an embodiment uses dictionary information and a language model to select a character string having the highest reliability among a plurality of estimated character strings. Can be obtained as a string.

As described above, the speech recognition system according to various embodiments of the present disclosure may selectively perform on-device speech recognition and server-based speech recognition. However, the present disclosure is not limited thereto, and the device 300 according to an embodiment of the present disclosure includes a plurality of speech recognition modules, and a case in which speech recognition is performed in the first speech recognition module in a manner similar to that described above, The second voice recognition module may selectively perform a case where voice recognition is performed.

12 is a block diagram of a device that selectively uses two voice recognition modules according to an embodiment.

Referring to FIG. 12, the device 300 may include a receiving unit 310, a processor 320, a memory 340, and an output unit 350. Not all of the components shown in FIG. 12 are essential components of the device 300. The device 300 may be implemented by more components than the components shown in FIG. 12, or the device 300 may be implemented by fewer components than the components shown in FIG. 12. For example, as shown in FIG. 19, the device 300 according to some embodiments may further include a user input unit 2100, a sensing unit 2400, and an A/V input unit 2600.

The receiving unit 310 according to an embodiment of the present disclosure may receive a voice signal from a user. For example, the receiving unit 310 may receive a voice signal by converting external sound into electrical sound data using a microphone. In FIG. 12, the receiver 310 is shown to be included in the device 300, but the receiver 310 according to another embodiment is included in a separate device and is connected to the device 300 by wire or wirelessly. It can be implemented in the form of

The memory 340 according to an embodiment of the present disclosure may store instructions for performing speech recognition, various models used for speech recognition, neural networks, dictionary information, and the like.

The memory 340 may store various models, neural networks, dictionary information, and the like used for speech recognition. The first data 341 stored in the memory 340 may include at least one of a model, a neural network, and dictionary information used by the first ASR module 321 to perform speech recognition. The second data 342 stored in the memory 340 may include at least one of a model, a neural network, and dictionary information used by the second ASR module 322 to perform speech recognition.

The processor 320 according to an embodiment of the present disclosure may perform speech recognition by executing one or more instructions stored in the memory 340. The processor 320 according to an embodiment of the present disclosure may include a first ASR module 321 and a second ASR module 322.

The first ASR module 321 of the processor 320 according to an embodiment of the present disclosure receives the voice signal obtained from the receiving unit 310, and receives the first data 341 (eg, an acoustic model, a neural network, Speech recognition on a speech signal may be performed based on a language model or dictionary information. The first ASR module 321 may obtain a first character string from the voice signal. The first character string may be a frame synchronized character string.

Since the first ASR module 321 of FIG. 12 may correspond to the ASR module 121 of FIG. 4A or the ASR module 121 of FIG. 4B, a duplicate description will be omitted.

The determination unit 323 of the processor 320 according to an embodiment of the present disclosure may determine whether to replace the first character string output from the first ASR module 321 with another character string.

As an example, the determination unit 323 of the processor 320 may determine the reliability of the first character string and determine whether to replace the first character string with another character string based on the reliability.

For example, if the reliability of the first character string is higher than or equal to the threshold value, the determination unit 323 of the processor 320 according to an embodiment determines that calibration of the first character string is not required and selects the first character string. It may be output through the output unit 350. On the other hand, if the reliability is less than the threshold value, the determination unit 323 of the processor 320 may determine that calibration of the first character string is necessary and transmit the first character string to the second ASR module 322.

As another example, the determination unit 323 of the processor 320 determines whether to replace the first string with another string based on a result of comparing the first string with keywords stored in the device 100. I can. As another example, the determination unit 323 of the processor 320 determines whether to replace the first string with another string, based on whether the domain to which the first string is related or whether the object name is included in the first string. I can.

Regarding a specific method of determining whether the determination unit 323 of the processor 320 according to an embodiment determines whether to replace a first character string with another character string, the device according to an embodiment with reference to FIGS. 3 to 5B A method in which the processor 120 of 100 determines whether to replace the first character string with another character string may be used. Redundant descriptions are omitted.

The determination unit 323 of the processor 320 according to an embodiment of the present disclosure may determine not to replace the first string with another string when it is determined that the correction of the first string is not required. The determination unit 323 of the processor 320 according to an embodiment of the present disclosure may output the first character string through the output unit 350 when not replacing the first character string with another character string.

When it is determined that the first character string should be replaced with another character string, the determination unit 323 of the processor 320 according to an embodiment of the present disclosure replaces the first character string with the second ASR module 322 based on this determination. Can be transferred to.

The determiner 323 of the processor 320 according to an embodiment may transmit the first character string to the second ASR module 322 in units of sentences, words, phrases, or frames. When the first ASR module 321 of the processor 320 according to an embodiment performs speech recognition to obtain a character string constituting a sentence or phrase, the determination unit 323 may select characters included in the sentence or phrase. All of them may be transmitted to the second ASR module 322 or some of the characters included in the sentence or phrase may be transmitted to the second ASR module 322. The determination unit 323 may transmit some characters with low reliability to the second ASR module 322 based on the reliability of the character string.

The second ASR module 322 of the processor 320 according to an embodiment of the present disclosure may receive the first character string and process the first character string. The second ASR module 322 performs re-decoding of the first character string based on the language model and dictionary information included in the second data 342, thereby replacing the second character string with at least one character in the first character string. Can be obtained.

The second ASR module 322 may calculate the likelihood of a plurality of estimated character strings from the first character string. The second ASR module 322 may determine whether to replace the first character string with the second character string based on the calculated likelihood. The second ASR module 322 may obtain the second character string from the first character string by replacing at least one character in the first character string with another character based on the determination. The second ASR module 322 may acquire a second character string, which is one of a plurality of estimated character strings, based on the likelihood, dictionary information, and language model.

Since the second ASR module 322 of FIG. 12 may correspond to the processor 220 of FIGS. 7 and 9, a redundant description will be omitted.

The second ASR module 322 may output the second character string through the output unit 350.

The output unit 350 according to an embodiment of the present disclosure may output a speech recognition result corresponding to the first character string or the second character string. The output unit 350 may notify a user of a result of performing voice recognition or transmit it to an external device (eg, a smart phone, a home appliance, a wearable device, a server, etc.). For example, the output unit 350 may include a speaker capable of outputting an audio signal or a display capable of outputting a video signal.

Alternatively, the device 300 according to an embodiment may perform an operation corresponding to a result of analyzing the first character string or the second character string. For example, the device 300 may determine a function of the device 300 corresponding to a result of performing speech recognition, and may output a screen performing the function through the output unit 350. Alternatively, the device 300 may transmit a keyword corresponding to a result of performing speech recognition to an external server, receive information related to the transmitted keyword from the server, and output it on the screen through the output unit 350. .

Alternatively, the device 300 according to an embodiment may output information related to the voice assistant service through the output unit 350 by identifying the user's utterance intention through natural language processing on the first character string or the second character string. I can. The device 300 may use a Natural Language Understanding (NLU) model, a Dialog Mananer (DM) model, and a Natural Language Generating (NLG) model in the device 300 to provide a voice assistant service.

As an example, the device 300 may generate and output a response message based on the first character string or the second character string as if a person directly communicates with the user in consideration of the user's situation, the device situation, and the like. As another example, the device 300 may generate and output information required by the user based on the first character string or the second character string. As another example, the device 300 may determine the user's speech intention based on the first character string or the second character string, and request the service providing server to provide a service that the user needs. The device 300 may transmit information received from the service providing server through the output unit 350.

The second ASR module 322 according to an embodiment of the present disclosure includes a larger amount of language model and second data 342 than the first data 341 used by the first ASR module 321. ) Can be used. In addition, compared to the first data 341, the second data 342 may include more entity names such as a place name, a person's name, and a brand name. Therefore, according to the speech recognition by the second ASR module 322, it is possible to use dictionary information including a vast amount of entity names and a language model, so that speech recognition with high accuracy can be performed.

Therefore, the device 300 of FIG. 12 may perform voice recognition for general purposes such as dictation, general commands, and caption generation in the first ASR module 321 in order to minimize the delay time. On the other hand, when it is determined that the reliability of the first character string output from the first ASR module 321 is insufficient, the second ASR module 322 performs additional processing on the first character string. I can. The second ASR module 322 may increase the accuracy of speech recognition by using the second data 342 including a larger amount of information than the first data 341.

On the other hand, the processor 320 of the device 300 according to an embodiment acquires the corrected character string in the second ASR module 322, determines that calibration is not necessary, and has not been transmitted to the second ASR module 322. Strings can be combined with corrected strings. The device 300 according to an embodiment may output a combined character string, output a result of performing speech recognition based on the combined character string, or provide a voice assistant service based on a result of interpreting the combined character string. I can.

In addition, the determination unit 323 of the processor 320 according to an embodiment may request the second ASR module 322 to correct the first string and provide domain information related to the first string together. The domain information is information for identifying a domain, and may include, for example, a name of a domain and an identifier of a domain, but is not limited thereto.

The determination unit 323 of the device 300 may identify a domain related to the first character string based on the domain reliability of the first character string output from the first ASR module 321. The domain reliability may be a number indicating to what extent at least a part of the first character string is related to a specific domain. For example, the determination unit 323 may calculate a confidence score indicating how relevant the first character string output from the first ASR module 321 is to a domain previously registered in the first data 341. have. Also, the device 300 may identify a domain related to the first character string based on the calculated domain reliability. The device 300 may identify a domain related to the first character string based on a rule, or obtain a domain reliability level related to the first character string using an artificial intelligence model trained for domain identification.

Meanwhile, the second ASR module 322 according to an embodiment of the present disclosure may perform decoding on a first character string using different dictionary information and language models for each domain included in the second data 342. I can. Accordingly, the second ASR module 322 according to an embodiment of the present disclosure may output a speech recognition result with improved speech recognition accuracy through re-decoding of the first character string.

The second ASR module 322 according to an embodiment may receive a first character string from the determiner 323 and determine a domain related to the first character string. The second ASR module 322 may decode the first character string by using dictionary information and language model corresponding to the determined domain.

As an example, the second ASR module 322 receives domain information related to the first character string together with the first character string from the determination unit 323, and performs decoding on the first character string based on the received information. You can decide the domain. For example, the second ASR module 322 may determine a domain that is identical to or similar to the domain identified from the information received from the determination unit 323 as a domain to perform decoding.

As another example, the second ASR module 322 may determine a domain related to the received first character string based on the first character string received from the determination unit 323. The device 300 may store a domain identification model, which is an artificial intelligence model trained for domain identification, in the memory 340. The second ASR module 322 may output the domain reliability using the first character string as an input value using the domain identification model. The second ASR module 322 may determine a domain related to the first character string based on the domain reliability.

As another example, the second ASR module 322 according to an embodiment may receive information for determining a domain related to the first character string together with the first character string from the determination unit 323. Information for determining the domain received from the determination unit 323 may include context information. For example, the context information may include at least one of information about an application currently being used by the user on the device 300, information about a conversation history, information about a situation around the device 300, or information about a trend. The second ASR module 322 may determine a domain for decoding the first character string based on context information. For a specific method of determining a domain based on context information, since the operating method of the processor 220 of FIG. 9 may be applied, a redundant description will be omitted.

Hereinafter, a method of operating the device 100 according to an exemplary embodiment will be described. Each step of the method of operating the device 100 described below may be performed by the configurations shown in FIGS. 3, 4A, and 4B. Redundant descriptions are omitted.

13 is a flowchart illustrating a method of performing voice recognition by a device according to an exemplary embodiment.

In step S1310, the device 100 according to an embodiment of the present disclosure may acquire a first character string by performing voice recognition on a voice signal.

The device 100 according to an embodiment of the present disclosure may estimate the first character string by performing speech recognition using various speech recognition methods.

As an example, the device 100 may obtain a character string from a speech signal using an acoustic model, dictionary information, and a language model. First, the device 100 may acquire a phoneme sequence included in a speech signal using an acoustic model. For example, the device 100 may estimate a phoneme sequence including phonemes using a hidden Markov model, a Gaussian mixed model, Bayesian inference, and a multilayer neural network. The device 100 may estimate words from a phoneme sequence and obtain a first character string including the estimated words based on dictionary information and a language model.

As another example, the device 100 may extract a feature vector from a speech signal and obtain a first character string from the feature vector using a deep neural network (DNN).

For example, the first character string may be a frame-synchronized character string including characters corresponding to each of the audio signal frames in which the audio signal is divided at predetermined time intervals. Alternatively, for example, the first character string may be a character string obtained through a label synchronization method so as to include one character for each character pronounced by a voice signal.

The device 100 according to an embodiment may obtain a frame-synchronized character string by performing forced alignment when the first character string is not frame-synchronized. With regard to a frame-synchronized character string and a specific method of generating a frame-synchronized character string by forced alignment, the above description with reference to FIG. Redundant descriptions are omitted.

In step S1330, the device 100 according to an embodiment of the present disclosure may determine whether to replace the first character string with another character string.

As an example, the device 100 according to an embodiment may determine the reliability of the first character string, and determine whether to replace the first character string with another character string based on the reliability. For example, if the reliability is higher than or equal to the threshold value, the device 100 may determine that it is not necessary to replace the first character string with another character string. On the other hand, when the reliability is less than the threshold value, the device 100 may determine to replace the first character string with another character string.

The reliability of the first character string may be calculated based on at least one of a likelihood of a plurality of estimated character strings obtained from the first character string and posterior probabilities in which at least one character in the first character string is replaced with another character.

For example, the device 100 may calculate the reliability based on the likelihood output as a result of Viterbi decoding. Alternatively, the processor 120 may calculate the reliability based on posterior probabilities output from the softmax layer in the end-to-end speech recognition model.

Alternatively, the device 100 according to an embodiment may determine a plurality of estimated character strings estimated during a speech recognition process for a speech signal, and calculate the reliability of the first character string based on the correlation between the plurality of estimated character strings. have. The higher the correlation between the plurality of estimated strings including the first string, the higher the reliability of the first string.

As another example, the device 100 may determine whether to replace the first character string with another character string based on a result of comparing the previously stored keywords with the first character string. For example, when keywords previously stored in the first string are not included, the device 100 may determine to replace the first string with another string.

As another example, the device 100 may determine whether to replace the first character string with another character string, based on whether the domain to which the first character string is related or whether an entity name is included in the first character string. For example, when it is determined that the first character string is related to a domain based on an entity name, the device 100 may determine to replace the first character string with another character string.

When it is determined to replace the first string with another string, the device 100 according to an embodiment of the present disclosure may transmit the first string to the server 200 in step S1340. The device 100 according to an embodiment may transmit the frame-synchronized first character string to the server 200. The device 100 may simultaneously transmit an entire character string including a plurality of characters, or may sequentially transmit at least some characters included in the character string. In addition, the device 100 according to an embodiment may transmit the first character string in units of words or sentences.

Meanwhile, when it is determined not to replace the first character string with another character string, the device 100 according to an embodiment of the present disclosure may output the first character string in step S1370. The device 100 according to an embodiment may output the first character string as it is or may output a word sequence obtained from the first character string.

In step S1350, the device 100 according to an embodiment of the present disclosure may receive a second character string from the server 200. The second character string may be a character string obtained by replacing at least one character in the first character string with another character by the server 200.

In step S1360, the device 100 according to an embodiment of the present disclosure may output a second character string. The device 100 according to an exemplary embodiment may output the second character string as it is or may output a word sequence obtained from the second character string.

The present disclosure is not limited to an embodiment in which the device 100 outputs the first character string or the second character string as it is, as shown in FIG. 13. The device 100 according to an embodiment may output information related to a voice assistant service by identifying a user's utterance intention through natural language processing on a first character string or a second character string.

The device 100 may use an NLU model, a DM model, and an NLG model in the device 100 to provide a voice assistant service based on the first character string or the second character string.

For example, the device 100 may generate and output a response message based on the first character string or the second character string, as if a person directly communicates with the user in consideration of the user's situation, the device situation, and the like. Alternatively, for example, the device 100 may generate and output information required by the user based on the first character string or the second character string. Alternatively, for example, the device 100 may determine the user's speech intention based on the first character string or the second character string, and request a service providing server to provide a service that the user needs. The device 100 may output information received from the service providing server.

Meanwhile, instead of receiving the second character string from the server 200, the device 100 according to an embodiment receives information related to the voice assistant service generated based on the second character string, and outputs the received information. can do. The information related to the voice assistant service may be information generated by the server 200 on the basis of the second character string in which the first character string has been corrected. For example, information related to the voice assistant service may include a response message to a user's voice signal, a service required by the user, or information required by the user.

As shown in FIG. 13, the device 100 according to an embodiment determines whether to replace the first string output from the on-device voice recognition module with another string, and based on the determination result, the server- Based post-processing can be used selectively.

The device 100 according to an embodiment calculates the reliability of the first string output from the speech recognition module in units of words (or sentences) spoken by the user, and replaces the first string based on the reliability. You can decide whether or not.

14 is a flowchart illustrating step S1310 of FIG. 13 in a specific method of performing voice recognition by a device according to an embodiment.

In step S1411, the device 100 according to an embodiment may receive a voice signal and determine whether a word boundary has been detected in step S1413. The device 100 according to an embodiment may continuously receive a speech signal including speech frames until a word boundary is detected.

For example, the device 100 may detect the boundary of a word based on a pause detected from a voice signal or prosodic information including intonation and stress.

When the boundary of the word is detected, in step S1415, the device 100 according to an embodiment may obtain the first character string from the voice signal.

In step S1431, the device 100 according to an embodiment of the present disclosure may calculate the reliability of the first character string. The reliability of the first character string may be calculated based on at least one of a likelihood of a plurality of estimated character strings obtained from the first character string and posterior probabilities in which at least one character in the first character string is replaced with another character.

For example, the device 100 may calculate the reliability based on the likelihood output as a result of Viterbi decoding. Alternatively, the device 100 may calculate the reliability based on posterior probabilities output from the softmax layer in the end-to-end speech recognition model.

Alternatively, the device 100 according to an embodiment may determine a plurality of estimated character strings estimated during a speech recognition process for a speech signal, and calculate the reliability of the first character string based on the correlation between the plurality of estimated character strings. have. The higher the correlation between the plurality of estimated strings including the first string, the higher the reliability of the first string.

In step S1433, the device 100 according to an embodiment of the present disclosure may determine whether the reliability is less than a threshold value.

When the reliability of the first character string is less than the threshold value, in operation S1340, the device 100 according to an embodiment of the present disclosure may transmit the first character string to the server 200. In response to the transmitted first character string, the device 100 may receive a second character string in which at least one character in the first character string is replaced by another character by the server 200. The device 100 may output the received second character string.

On the other hand, when the reliability of the first character string is greater than or equal to the threshold value, in step S1370, the device 100 according to an embodiment of the present disclosure may output the first character string.

Meanwhile, the device 100 according to an embodiment recognizes the user's speech intention through natural language processing on the first string or the second string, instead of outputting the first string or the second string as it is, Service-related information can be displayed.

The device 100 may use an NLU model, a DM model, and an NLG model in the device 100 to provide a voice assistant service based on the first character string or the second character string.

For example, the device 100 may generate and output a response message based on the first character string or the second character string, as if a person directly communicates with the user in consideration of the user's situation, the device situation, and the like. Alternatively, for example, the device 100 may generate and output information required by the user based on the first character string or the second character string. Alternatively, for example, the device 100 may determine the user's speech intention based on the first character string or the second character string, and request a service providing server to provide a service that the user needs. The device 100 may output information received from the service providing server.

On the other hand, the device 100 according to an embodiment, instead of receiving the second string from the server 200 in response to the transmitted first string, provides information related to the voice assistant service generated based on the second string. Can receive. The device 100 may output information received from the server 200. The information related to the voice assistant service may be information generated by the server 200 on the basis of the second character string in which the first character string has been corrected.

For example, information related to the voice assistant service may include a response message to a user's voice signal, a service required by the user, or information required by the user.

As shown in FIG. 14, the device 100 according to an embodiment of the present disclosure may determine whether to replace the first character string with another character string based on the reliability of the first character string. When the reliability of the first character string is less than the threshold value, the device 100 may transmit the first character string to the server 200. The device 100 may obtain a second character string obtained by replacing at least one character of the first character string with another character based on dictionary information and a language model in the server 200 from the server 200. Accordingly, the device 100 according to an embodiment may increase speech recognition accuracy by receiving and using a second character string having a higher reliability than the first character string from the server 200.

14 illustrates an embodiment in which the reliability of the speech recognition result is calculated in units of words spoken by the user and whether to replace the first character string is determined. However, the present disclosure is not limited to the example shown in FIG. 14, and the device 100 according to an embodiment calculates the reliability of the speech recognition result in units of sentences spoken by the user, and replaces the first character string. You can decide whether or not. As a method of detecting the end of a sentence uttered by a user, various conventional methods may be used, and a detailed description thereof will be omitted in the present disclosure.

15 is a flowchart of a method of operating the server 200 according to an embodiment. Each step of the method of operating the server 200 described below may be performed by the configurations shown in FIGS. 7 and 9. Redundant descriptions are omitted.

In operation S1510, the server 200 according to an embodiment of the present disclosure may receive a first character string from the device 100. The first character string may be output from a voice signal by the device 100 through voice recognition processing.

As an example, the first character string received from the server 200 may be a frame synchronization character string including characters corresponding to each of the audio signal frames in which the audio signal is divided at predetermined time intervals. As another example, the first character string received from the server 200 may be a character string that is not frame synchronized.

The processor 220 according to an embodiment of the present disclosure may obtain a frame-synchronized character string from the first character string when the first character string received from the device 100 is a character string that is not frame synchronized. The processor 220 may obtain a frame-synchronized character string by continuously arranging at least one character included in the first character string in units of frames at a predetermined time interval.

In step S1520, the server 200 according to an embodiment of the present disclosure may calculate the likelihood of a plurality of estimated character strings from the first character string. The server 200 according to an embodiment may obtain a plurality of estimated character strings by replacing each character in the first character string with a different character. The likelihood of the plurality of estimated character strings may mean a probability that the first character string is estimated from the speech recognition module, assuming that each of the plurality of estimated character strings obtained from the first character string is a true value character string.

According to an embodiment of the present disclosure, the server 200 identifies replacement characters having a similar pronunciation to each character in the first string, and at least one character in the first string is different based on the identified replacement characters. In order to determine the estimated character strings corrected to, the likelihood obtained from the first character string may be obtained.

The server 200 according to an embodiment calculates likelihood matrices for replacement characters to be replaced for each character in the first character string, and estimates a plurality of likelihood values in the likelihood matrices. Strings can be identified. The server 200 may obtain likelihood matrices obtained from each character as likelihood for a plurality of estimated character strings.

As an example, the server 200 may calculate the likelihood from the first character string based on previously accumulated characters of each character in the first character string. The server 200 according to an embodiment may calculate posterior probabilities of each character based on previously accumulated characters of each character in the first character string. The server 200 may calculate a character arrangement probability based on previously accumulated characters of each character in the first character string. The server 200 may calculate the likelihood of a plurality of estimated character strings obtained from the first character string, based on the posterior probabilities of each character and the character arrangement probability.

As another example, the server 200 may calculate the likelihood from the first character string by considering only each character without considering the previously accumulated characters of each character in the first character string. The server 200 according to an embodiment may calculate posterior probabilities of each character in the first character string based on a predetermined error matrix. The server 200 may calculate the likelihood of a plurality of estimated character strings obtained from the first character string, based on posterior probabilities of each character in the first character string.

In step S1530, the server 200 according to an embodiment of the present disclosure replaces at least one character in the first character string with another character based on the likelihood calculated in step S1520 to replace the second character string from the first character string. Can be obtained.

The server 200 according to an embodiment may identify a plurality of estimated character strings in which at least one character in the first character string has been replaced with another character based on the calculated likelihood. The server 200 may obtain a second character string from among the plurality of estimated character strings, based on the likelihood of the identified plurality of estimated character strings, a language model, and dictionary information.

The server 200 according to an embodiment may determine whether to replace the first character string with the second character string based on the calculated likelihood. The server 200 may obtain the second character string from the first character string by replacing at least one character in the first character string with another character based on the determination. The server 200 may select an estimated character string that maximizes the likelihood from among a plurality of estimated character strings based on the likelihood, dictionary information, and language model. The server 200 may obtain a second character string in which at least one character in the first character string is replaced with another character according to the selected estimated character string.

As an example, the server 200 may acquire the second character string using a WFST decoder based on dictionary information and language models stored in the server 200. When the server 200 performs WFST decoding, the server 200 according to an embodiment includes a relationship (T) between characters, dictionary information (L) including mapping information of words and characters, and a specific word string. Based on the language model (G) that estimates the probability of the next word given when is, a search space can be constructed and decoded with WFST.

As another example, the server 200 may include a Viterbi decoder that recalculates the likelihood of a plurality of estimated character strings obtained from the first character string based on dictionary information and a language model. The Viterbi decoder may determine a second character string that maximizes the recalculated likelihood from among the plurality of estimated character strings. The Viterbi decoder may find the most likely character string for the first character strings as the second character string in consideration of the dictionary information and the language model.

In step S1540, the server 200 according to an embodiment of the present disclosure may transmit the second character string to the device 100.

In addition, the server 200 according to an embodiment may analyze the second character string using a natural language understanding model, and generate a response message for a user's voice signal based on the result of the analysis. The server 200 may generate a response message and additionally transmit it to the device 100.

The present disclosure is not limited to an embodiment in which the server 200 transmits the second character string to the device 100 as it is, as shown in FIG. 15. The server 200 according to an embodiment may transmit information related to a voice assistant service by identifying the user's speech intention through natural language processing on the second character string.

The server 200 may use an NLU model, a DM model, and an NLG model in the server 200 in order to provide a voice assistant service based on the second character string.

As an example, the server 200 may generate a control command for controlling the device 100 or another device based on the result of analyzing the second character string, and transmit the generated control command to the device 100 have. As another example, the server 200 may generate and transmit a response message based on the second character string, as if a person directly communicates with the user in consideration of the situation of the user and the device. As another example, the server 200 may generate and transmit information required by the user based on the second character string. As another example, the server 200 may determine the user's utterance intention based on the second character string, and may request the service providing server to provide a service required by the user. The server 200 may transmit information received from the service providing server.

16 illustrates in detail a method of obtaining a likelihood from a character string in consideration of characters accumulated before each character in a method of operating a server according to an embodiment.

In step S1510, the server 200 according to an embodiment of the present disclosure may obtain a first character string from the device 100.

In step S1621, the server 200 according to an embodiment may obtain posterior probabilities of each character based on previous characters of each character in the first character string.

For example, the server 200 may calculate the posterior probabilities of each character in the first character string using a pretrained neural network to calculate the posterior probabilities of the character string.

In step S1623, the server 200 according to an embodiment may calculate a character arrangement probability from the first string.

In step S1625, the server 200 according to an embodiment may calculate the likelihood of a plurality of estimated strings obtained from the first string, based on the posterior probabilities calculated in step S1621 and the character arrangement probability calculated in step S1623. I can. The server 200 according to an embodiment calculates likelihood matrices for replacement characters to be replaced with each character for each character in the first character string, and enables a plurality of estimated character strings including the calculated likelihood matrices. You can get degrees.

The server 200 according to an embodiment may determine whether a likelihood matrix has been calculated for all characters included in the first string. The server 200 according to an embodiment may repeatedly perform steps S1621, S1623, and S1625 until likelihood matrices are calculated for all characters included in the first string.

The detailed process of calculating the likelihood from the first character string has been described above with reference to FIG. 9, and therefore, a duplicate description is omitted.

In step S1627, the server 200 according to an embodiment may obtain a second character string from the likelihood calculated in step S1625 using dictionary information and a language model. The second character string may be a character string in which at least one character of the first character string is replaced with another character.

For example, the server 200 uses a WFST decoder that inputs likelihood or a Viterbi decoder using traditional token passing, based on dictionary information, a language model, and a calculated likelihood, Among them, a second character string may be obtained.

In operation S1540, the server 200 according to an embodiment may transmit a second character string to the device 100. The server 200 according to an embodiment transmits information related to the voice assistant service by identifying the user's speech intention through natural language processing on the second string instead of transmitting the second string as it is to the device 100 can do. Redundant descriptions are omitted.

17 is a diagram for explaining weighted finite state transducer (WFST) decoding according to an embodiment.

The server 200 according to an embodiment of the present disclosure may calculate a likelihood from a first character string received from the device 100 and perform WFST decoding by inputting the calculated likelihood. The server 200 according to an embodiment of the present disclosure includes a likelihood (T) of a plurality of estimated strings obtained from a first string, dictionary information (L) including mapping information of words and characters, and a specific word string. Given is given, WFST decoding can be performed by modeling each language model G, which estimates the probability of the next word, using a weighted finite-state transducer (WFST).

Hereinafter, an example of modeling a language model for storing information on a relationship between words'the, cat, and, deer, is, cardinals, baseball, team' using WFST will be described. 17 shows a finite number of character strings that can be created by combining words based on a language model.

Each circle in FIG. 17 represents a state, and words stored in the language model are displayed on the arrow. The WFST decoder may calculate the reliability of a character string from each of a plurality of character strings combined along a plurality of paths. The reliability of each character string may be calculated based on the likelihood of each character string, dictionary information, and a language model. The WFST decoder can select and output a character string with the highest reliability.

For example, as shown in FIG. 8A, the server 200 according to an embodiment of the present disclosure may receive a first character string [The cat and deer is baseball team] from the device 100.

The server 200 may calculate the likelihood of a plurality of estimated character strings from the first character string. As the calculated likelihood is input to the WFST decoder of the server 200, the WFST decoder may output a second character string. The WFST decoder may determine and output a second character string having the highest reliability among a plurality of estimated character strings.

As illustrated in FIG. 8A, in the memory 230 of the server 200, the entity name “Cardinals” of the sports domain may be stored. Accordingly, the processor 220 of the server 200 may determine that the probability that the'cat and deer is' estimated by the device 100 is actually the baseball team name'Cardinals' is higher.

Therefore, referring to FIG. 17, the WFST decoder according to an embodiment includes [The Cardinals baseball team], which is a string having the highest reliability among a plurality of estimated strings [The cat and deers baseball team] and [The Cardinals baseball team] Can be determined as the second character string and printed.

18 illustrates an example of a screen on which a voice recognition result is displayed on a device according to an embodiment.

The device 100 according to an embodiment may perform voice recognition on a voice signal received from a user and output a word string 1811 "Cat and deers baseball team" obtained from an estimated character string. When on-device voice recognition is being performed, the device 100 may display an image 1812 indicating that on-device voice recognition is being performed on the screen.

The device 100 according to an embodiment of the present disclosure may use the result of performing the voice recognition as it is when the reliability of the result of performing voice recognition using on-device voice recognition is sufficiently high.

On the other hand, if the device 100 according to an embodiment of the present disclosure determines that the reliability of the result of performing the speech recognition using on-device speech recognition is not sufficiently high, the server 200 ).

The server 200 according to an embodiment receives a character string from the device 100 and performs decoding using a language model and dictionary information in the server 200, so that at least one character included in the received character string is The corrected string "Caldinals baseball team" can be acquired. The server 200 may transmit "Caldinals baseball team" to the device 100.

The device 100 according to an embodiment may output the string 1821 "Caldinals baseball team" received from the server 200. When server-based voice recognition is being performed, the device 100 may display an image 1832 indicating that server-based voice recognition is being performed on the screen.

19 is a detailed block diagram of a device according to an embodiment.

The device 100 illustrated in FIG. 19 may include the same components as the device 100 described in FIG. 3. For example, among the components illustrated in FIG. 19, the controller 2300 is the same as the processor 120 illustrated in FIG. 3, and the output unit 2220 is the same as the output unit 150 illustrated in FIG. 3. In addition, although not shown in FIG. 19, the memory 2700 of FIG. 19, like the memory 140 of FIG. 3, includes instructions for performing speech recognition, various models used for speech recognition, neural networks, dictionary information, etc. Can be saved. Therefore, redundant descriptions will be omitted.

The device 100 illustrated in FIG. 19 may perform all of the operations and functions of the device 100 described in FIGS. 3 to 18. Accordingly, hereinafter, components of the device 100 that have not been described so far will be described.

Referring to FIG. 19, the device 100 includes a user input unit 2100, an output unit 2200, a control unit 2300, a sensing unit 2400, a communication unit 2500, an A/V input unit 2600, and a memory ( 2700) may be included.

The user input unit 2100 refers to a means for a user to input data for controlling the device 100. For example, the user input unit 2100 includes a key pad, a dome switch, a touch pad (contact type capacitance method, pressure type resistive film method, infrared detection method, surface ultrasonic conduction method, integral type). Tension measurement method, piezo effect method, etc.), a jog wheel, a jog switch, and the like, but are not limited thereto. The user input unit 2100 may receive a user input necessary to generate conversation information to be provided to a user.

The output unit 2200 may output an audio signal, a video signal, or a vibration signal, and the output unit 2200 may include a display unit 2210, an audio output unit 2220, and a vibration motor 2230. .

The vibration motor 2230 may output a vibration signal. For example, the vibration motor 2230 may output a vibration signal corresponding to an output of audio data or video data (eg, a call signal reception sound, a message reception sound, etc.).

The sensing unit 2400 may detect a state of the device 100 or a state around the device 100, and transmit the sensed information to the controller 2300.

The sensing unit 2400 includes a magnetic sensor 2410, an acceleration sensor 2420, a temperature/humidity sensor 2430, an infrared sensor 2440, a gyroscope sensor 2450, and a position sensor. (For example, a GPS) 2460, an atmospheric pressure sensor 2470, a proximity sensor 2480, and an RGB sensor 2490 may be included, but are not limited thereto. Since the function of each sensor can be intuitively inferred by a person skilled in the art from its name, detailed description will be omitted.

The communication unit 2500 may include a component for performing communication with another device. For example, the communication unit 2500 may include a short range communication unit 2510, a mobile communication unit 2520, and a broadcast reception unit 2530.

The short-range wireless communication unit 251 includes a Bluetooth communication unit, a Bluetooth Low Energy (BLE) communication unit, a near field communication unit, a WLAN (Wi-Fi) communication unit, a Zigbee communication unit, and an infrared ( IrDA, infrared data association) communication unit, WFD (Wi-Fi Direct) communication unit, UWB (ultra wideband) communication unit, Ant+ communication unit, etc. may be included, but is not limited thereto.

The mobile communication unit 2520 transmits and receives a radio signal with at least one of a base station, an external terminal, and a server over a mobile communication network. Here, the wireless signal may include a voice call signal, a video call signal, or various types of data according to transmission/reception of text/multimedia messages.

The broadcast receiver 2530 receives a broadcast signal and/or broadcast-related information from outside through a broadcast channel. Broadcast channels may include satellite channels and terrestrial channels. Depending on the implementation example, the device 100 may not include the broadcast receiver 2530.

Also, the communication unit 2500 may transmit/receive information necessary for generating conversation information to be provided to the first user with the second interactive electronic device 3000 and other devices and servers.

The A/V (Audio/Video) input unit 2600 is for inputting an audio signal or a video signal, and may include a camera 2610 and a microphone 2620. The camera 2610 may obtain an image frame such as a still image or a video through an image sensor in a video call mode or a photographing mode. The image captured through the image sensor may be processed by the controller 2300 or a separate image processing unit (not shown).

The image frame processed by the camera 2610 may be stored in the memory 2700 or transmitted to the outside through the communication unit 2500. Two or more cameras 2610 may be provided depending on the configuration aspect of the terminal.

The microphone 2620 receives an external sound signal and processes it as electrical voice data. For example, the microphone 2620 may receive an acoustic signal from an external device or a speaker. The microphone 2620 may use various noise removal algorithms for removing noise generated in a process of receiving an external sound signal.

The memory 2700 may store a program for processing and control of the controller 2300 and may store data input to the device 100 or output from the device 100.

The memory 2700 is a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (for example, SD or XD memory), and RAM. , SRAM, ROM, EEPROM, PROM, magnetic memory, magnetic disk, optical disk may include at least one type of storage medium.

Programs stored in the memory 2700 may be classified into a plurality of modules according to their functions, for example, a UI module 2710, a touch screen module 2720, a notification module 2730, and the like. .

The UI module 2710 may provide a specialized UI, GUI, etc. that are interlocked with the device 100 for each application. The touch screen module 2720 may detect a user's touch gesture on a touch screen and transmit information on the touch gesture to the controller 2300. The touch screen module 2720 according to some embodiments may recognize and analyze a touch code. The touch screen module 2720 may be configured with separate hardware including a controller.

The notification module 2730 may generate a signal to notify the occurrence of an event of the device 100. Examples of events occurring in the device 100 include call signal reception, message reception, key signal input, and schedule notification. The notification module 2730 may output a notification signal in the form of a video signal through the display unit 2210, may output a notification signal in the form of an audio signal through the sound output unit 2220, or the vibration motor 2230 It is also possible to output a notification signal in the form of a vibration signal through.

The disclosed embodiments may be implemented as a S/W program including commands stored in a computer-readable storage media.

The computer, as a device capable of calling a command stored from a storage medium and performing operations according to the disclosed embodiments according to the called command, may include an image transmission device and an image reception device according to the disclosed embodiments.

The computer-readable storage medium may be provided in the form of a non-transitory storage medium. Here, "non-transitory" means that the storage medium does not contain signals and is tangible, but does not distinguish between semi-permanent or temporary storage of data in the storage medium.

Further, the electronic device or method according to the disclosed embodiments may be provided in a computer program product. Computer program products can be traded between sellers and buyers as commodities.

The computer program product may include a S/W program and a computer-readable storage medium storing the S/W program. For example, the computer program product may include a product (e.g., downloadable app) electronically distributed through an electronic device manufacturer or an electronic market (e.g., Google Play Store, App Store). have. For electronic distribution, at least a part of the S/W program may be stored in a storage medium or may be temporarily generated. In this case, the storage medium may be a server of a manufacturer, a server of an electronic market, or a storage medium of a relay server temporarily storing an SW program.

The computer program product may include a storage medium of a server or a storage medium of a terminal in a system consisting of a server and a terminal (eg, an image transmission device or an image reception device). Alternatively, when there is a third device (eg, a smartphone) that is communicatively connected to the server or terminal, the computer program product may include a storage medium of the third device. Alternatively, the computer program product may include a S/W program itself transmitted from a server to a terminal or a third device, or transmitted from a third device to a terminal.

In this case, one of the server, the terminal, and the third device may execute the computer program product to perform the method according to the disclosed embodiments. Alternatively, two or more of a server, a terminal, and a third device may execute a computer program product to distribute and implement the method according to the disclosed embodiments.

For example, a server (eg, a cloud server or an artificial intelligence server) may execute a computer program product stored in the server, and control a terminal connected to the server to perform the method according to the disclosed embodiments.

As another example, by executing a computer program product, the third device may control a terminal connected in communication with the third device to perform the method according to the disclosed embodiment. As a specific example, the third device may remotely control an image transmission device or an image reception device to transmit or receive a packed image.

When the third device executes the computer program product, the third device may download the computer program product from the server and execute the downloaded computer program product. Alternatively, the third device may perform the method according to the disclosed embodiments by executing the computer program product provided in a preloaded state.

Claims (20)

A memory that stores one or more instructions;
A processor that executes the one or more instructions stored in the memory; And
Including a communication unit for receiving the first character string from the device,
The processor,
Identify a plurality of estimated character strings from the first character string, and obtain a second character string based on the plurality of estimated character strings,
Controlling the communication unit to transmit the second character string to the device,
The server, characterized in that the first character string is output through speech recognition processing from a speech signal input to the device. The method of claim 1,
The processor,
Identifying replacement characters corresponding to each character in the first character string, identifying the plurality of estimated character strings based on the identified replacement characters,
Obtaining one estimated character string from among the plurality of estimated character strings as the second character string,
The server, characterized in that the replacement characters are characters having a pronunciation similar to each of the characters. The method of claim 1,
The processor,
For each character in the first character string, likelihood matrices for replacement characters to be substituted for each character are calculated, and the plurality of estimated character strings are identified based on likelihood values in the likelihood matrices. , server. The method of claim 1,
Wherein the first character string includes characters corresponding to each of the voice signal frames in which the voice signal is divided at predetermined time intervals. The method of claim 3,
The processor,
Calculate likelihood of the plurality of estimated character strings based on likelihood values in the likelihood matrices,
Based on the likelihood, dictionary information, and a language model, one of the plurality of estimated character strings is selected,
And obtaining the second character string in which at least one character in the first character string is replaced with another character according to the selected estimated character string. The method of claim 3,
The likelihood matrices obtained for each character are,
The server, characterized in that calculated based on the previously accumulated characters of each character. The method of claim 3,
The likelihood matrices obtained for each character are,
The server, characterized in that calculated based on posterior probabilities calculated based on previously accumulated characters of each character, and character arrangement probabilities calculated based on previously accumulated characters of each character. The method of claim 7,
The posterior probabilities are calculated using an artificial intelligence recurrent neural network (RNN) including a plurality of long-short term memory (LSTM) layers and a softmax layer. The method of claim 3,
The likelihood matrices obtained for each character are,
The server, characterized in that calculated based on a predetermined confusion matrix. The method of claim 1,
The processor,
The server, characterized in that to provide a service related to the voice signal input to the device based on the obtained second character string. A memory that stores one or more instructions;
A processor that executes the one or more instructions stored in the memory; And
Including a communication unit in communication with the server,
The processor,
Perform speech recognition on the speech signal to obtain a first character string,
Determine whether to replace the first character string with another character string,
Controlling the communication unit to transmit the first character string to the server based on the determination,
And controlling the communication unit to receive from the server a second character string obtained by replacing at least one character in the first character string with another character by the server. In the server operation method,
Receiving a first character string from a device;
Identifying a plurality of estimated character strings from the first character string;
Obtaining a second character string based on the plurality of estimated character strings; And
Transmitting the second character string to the device,
The method, characterized in that the first character string is output through speech recognition processing from a speech signal input to the device. The method of claim 12,
Identifying a plurality of estimated character strings from the first character string,
Identifying replacement characters corresponding to each character in the first character string; And
Identifying the plurality of estimated character strings based on the identified replacement characters,
Obtaining a second character string based on the plurality of estimated character strings,
Obtaining one estimated character string from among the plurality of estimated character strings as the second character string,
The method, characterized in that the replacement characters are characters of a pronunciation similar to each of the characters. The method of claim 12,
Identifying the plurality of estimated character strings,
Calculating likelihood matrices for replacement characters to be replaced with each character for each character in the first character string; And
And identifying the plurality of estimated strings based on likelihood values in the likelihood matrices. The method of claim 12,
Wherein the first character string includes characters corresponding to each of the speech signal frames in which the speech signal is divided at predetermined time intervals. The method of claim 14,
Obtaining the second character string,
Calculating a likelihood of the plurality of estimated character strings based on likelihood values in the likelihood matrices;
Selecting one of the plurality of estimated character strings based on the likelihood, dictionary information, and a language model; And
And obtaining the second character string in which at least one character in the first character string has been replaced with another character according to the selected estimated character string. The method of claim 14,
The likelihood matrices obtained for each character are,
Characterized in that it is calculated based on previously accumulated characters of each character. The method of claim 14,
The likelihood matrices obtained for each character are,
Characterized in that it is calculated based on posterior probabilities calculated based on previously accumulated characters of each character, and character arrangement probabilities calculated based on previously accumulated characters of each character. The method of claim 12,
And providing a service related to the voice signal input to the device based on the obtained second character string. In the method of operating the device,
Obtaining a first character string by performing speech recognition on the speech signal;
Determining whether to replace the first character string with another character string;
Transmitting the first character string to a server based on the determination; And
And receiving, from the server, a second character string obtained by replacing at least one character in the first character string with another character by the server.

Images