KR20150116389A - Speech recognition using Electronic Device and Server - Google Patents

Abstract

An electronic device is provided. The electronic device includes a processor configured to perform automatic speech recognition (ASR) on a speech input by using a speech recognition model that is stored in a memory and a communications module configured to provide the speech input to a server and receive a speech instruction, which corresponds to the speech input, from the server. The electronic device may perform different operations according to a confidence score of a result of the ASR. Besides, it may be permissible to prepare other various embodiments speculated through the specification.

Description

[0001] Speech recognition using an electronic device and a server [0002]

Various embodiments of the present invention relate to techniques for recognizing a user's voice input and performing voice commands using a voice recognition model installed on an electronic device and a voice recognition model available in the server.

In addition to traditional input methods using a keyboard or a mouse, recent electronic devices can support an input method using a user's speech. For example, an electronic device such as a smart phone or a tablet may analyze a user's voice input when a specific function (e.g., S-Voice, Siri, etc.) is executed, convert the voice to text, Can be performed. In addition, some electronic devices are always-on, with voice recognition being always awake, awake, unlocked, and functioning like Internet browsing, calling, SMS / email reading Can be performed.

Although various studies and techniques related to speech recognition are known, a method of performing speech recognition in an electronic device is limited. For example, an electronic device may utilize a speech recognition model that is self-contained in an electronic device for quick response to speech recognition. However, the storage space and processing capabilities of electronic devices are limited, thereby limiting the number or types of speech inputs that can be recognized.

In order to obtain a more accurate and reliable result for voice input, the electronic device can send voice input to the server to request voice recognition, provide a result returned from the server, or perform a specific operation based on the returned result . However, this method increases the communication usage of the electronic device and brings about a relatively slow response speed.

The various embodiments disclosed herein utilize two or more different speech recognition capabilities or speech recognition models to improve the inefficiencies that may occur in the various situations described above and provide a fast response time and high accuracy to the user Can be provided.

An electronic device according to various embodiments of the present invention includes a processor for performing automatic speech recognition (ASR) on speech input using a speech recognition model stored in a memory, a processor for providing the speech input to a server, And a communication module for receiving a voice command corresponding to the voice command. Wherein the processor is configured to: (1) perform an operation corresponding to a result of the ASR if the reliability of the ASR is greater than or equal to a first threshold value; (2) It may provide feedback on the reliability.

According to various embodiments of the present invention, speech recognition is performed using a speech recognition model that is built in the electronic device, and based on the speech recognition result, It is possible to provide a voice recognition function with high accuracy.

The results of the speech recognition using the electronic device and the server can be compared and reflected in the speech recognition model or speech recognition algorithm based on the comparison result. As the speech recognition is repeatedly performed, the accuracy and the response speed can be continuously improved.

BRIEF DESCRIPTION OF THE DRAWINGS Figure 1 shows a server connected via a network with an electronic device and an electronic device according to an embodiment of the present invention.
2 shows an electronic device and a server according to another embodiment of the present invention.
3 is a flowchart illustrating a method of performing speech recognition according to an embodiment of the present invention.
4 is a flowchart illustrating a method of performing speech recognition according to another embodiment of the present invention.
5 shows a flow diagram of a method for updating a threshold according to an embodiment of the present invention.
6 is a flowchart of a method for updating a speech recognition model according to an embodiment of the present invention.
Figure 7 illustrates an electronic device in a network environment in accordance with one embodiment of the present invention.
8 shows a block diagram of an electronic device according to an embodiment of the present invention.

Various embodiments of the invention will now be described with reference to the accompanying drawings. It should be understood, however, that the invention is not intended to be limited to the particular embodiments, but includes various modifications, equivalents, and / or alternatives of the embodiments of the invention. In connection with the description of the drawings, like reference numerals may be used for similar components.

In this document, the expressions "have," "may," "include," or "include" may be used to denote the presence of a feature (eg, a numerical value, a function, Quot ;, and does not exclude the presence of additional features.

In this document, the expressions "A or B," "at least one of A and / or B," or "one or more of A and / or B," etc. may include all possible combinations of the listed items . For example, "A or B," "at least one of A and B," or "at least one of A or B" includes (1) at least one A, (2) Or (3) at least one A and at least one B all together.

Expressions such as " first, "second," first, "or" second, " as used in various embodiments, Not limited. For example, the first user equipment and the second user equipment may represent different user equipment, regardless of order or importance. For example, without departing from the scope of the present invention, the first component may be referred to as a second component, and similarly, the second component may also be named as the first component.

(Or functionally or communicatively) coupled with / to "another component (eg, a second component), or a component (eg, a second component) Quot; connected to ", it is to be understood that any such element may be directly connected to the other element or may be connected through another element (e.g., a third element). On the other hand, when it is mentioned that a component (e.g., a first component) is "directly connected" or "directly connected" to another component (e.g., a second component) It can be understood that there is no other component (e.g., a third component) between other components.

As used herein, the phrase " configured to " (or set) to be "adapted to, " To be designed to, "" adapted to, "" made to, "or" capable of ". The term " configured (or set) to "may not necessarily mean " specifically designed to" Instead, in some situations, the expression "configured to" may mean that the device can "do " with other devices or components. For example, a processor configured (or configured) to perform the phrases "A, B, and C" may be a processor dedicated to performing the operation (e.g., an embedded processor), or one or more software programs To a generic-purpose processor (e.g., a CPU or an application processor) that can perform the corresponding operations.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of the other embodiments. The singular expressions may include plural expressions unless the context clearly dictates otherwise. All terms used herein, including technical or scientific terms, may have the same meaning as commonly understood by one of ordinary skill in the art. Commonly used predefined terms may be interpreted to have the same or similar meaning as the contextual meanings of the related art and are not to be construed as ideal or overly formal in meaning unless expressly defined in this document . In some cases, the terms defined in this document can not be construed to exclude embodiments of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An electronic apparatus according to various embodiments will now be described with reference to the accompanying drawings. In this document, the term user may refer to a person using an electronic device or a device using an electronic device (e.g., an artificial intelligence electronic device).

BRIEF DESCRIPTION OF THE DRAWINGS Figure 1 shows a server connected via a network with an electronic device and an electronic device according to an embodiment of the present invention.

Referring to FIG. 1, an electronic device may include the same configuration as the user terminal 100. For example, the user terminal 100 may include a microphone 110, a controller 120, an ASR module 130, an ASR model 140, a transceiver 150, a speaker 170, and a display 180. The configuration of the user terminal 100 shown in FIG. 1 is exemplary and various modifications are possible that can implement various embodiments disclosed in this document. For example, the electronic device may comprise or be modified to suitably utilize configurations such as the user terminal 101 shown in Fig. 2, the electronic device 701 shown in Fig. 7, the electronic device 801 shown in Fig. 8 have. Various embodiments of the present invention will now be described with reference to a user terminal 100. [

The user terminal 100 can acquire voice input from the user via the microphone 110. [ For example, if the user is running an application related to speech recognition, or if speech recognition is always active, the user's speech may be obtained via the microphone 110. The microphone 110 may include an analog-to-digital converter (ADC) that converts an analog signal to a digital signal. However, in some embodiments, an ADC, a Digital-Analog Converter (DAC), and various signal processing or pre-processing circuits may be included in the controller 120.

The controller 120 may provide the ASR module 130 and the transceiver 150 with audio signals (or audio signals) generated based on the audio input or the audio input obtained by the microphone 110. The audio signal provided to the ASR module 130 by the controller 120 may be a preprocessed signal for speech recognition. For example, the audio signal may be a noise filtering or a signal to which an equalizer suitable for a human voice is applied. While the signal provided to the transceiver 150 by the controller 120 may be the speech input itself. The controller 120 can transmit audio signals to the transceiver 150, unlike the ASR module 130, so that the audio signal processing appropriate or better performance can be applied by the server 200.

The controller 120 can control the general operation of the user terminal 100. For example, the controller 120 may control speech input from a user, control speech recognition, and control the performance of speech recognition.

The ASR module 130 may perform speech recognition on the audio signal provided from the controller 120. The ASR module 130 can perform isolated word recognition, connected word recognition, and large vocabulary recognition on a voice input (audio signal). The ASR performed by the ASR module 130 may be implemented in a speaker-independent manner, or may be implemented in a speaker-dependent manner. The ASR module 130 does not necessarily have to be one speech recognition engine, but may be composed of two or more speech recognition engines. Further, when the ASR module 130 includes a plurality of speech recognition engines, each speech recognition engine may have different recognition purposes. For example, one speech recognition engine may recognize a wakeup speech, such as " Hi, Galaxy " to activate the ASR function, and the other speech recognition engine may recognize a speech command speech, , For example, "read a recent e-mail". The ASR module 130 performs speech recognition based on the ASR model 140, and thus the range (e.g., type or number) of speech input that can be recognized by the ASR model 140 can be specified. The above description of the ASR module 130 may be applied to the ASR module 230 of the server described later.

ASR module 130 may convert speech input to text. The ASR module 130 may determine an action or function to be performed by the electronic device for voice input. Also, the ASR module 130 can determine a confidence lever (or a confidence score) for the result of the ASR.

The ASR model 140 may include a grammar. In addition to linguistic grammars, the grammar can include various forms of statistically generated grammars (collected either on user input or on the Web). In various embodiments, the ASR model 140 may include an acoustic model, a language model, and the like. Or the ASR model 140 may be a speech recognition model used for isolated word recognition. In various embodiments, the ASR model 140 may include a recognition model for performing an appropriate level of speech recognition taking into account the computing power and storage capabilities of the user terminal 100. [ For example, the grammar may include grammars for a given command structure, regardless of the linguistic grammar. For example, " call [user name] " may be included in the ASR model 140 as a syntax for issuing a call to a user of [user name].

The transceiver 150 may transmit the voice signal provided from the controller 120 to the server 200 via the network 10. [ And can also receive the result of performing speech recognition corresponding to the transmitted speech signal from the server 200. [

Speaker 170 and display 180 may be used to interact with user input. For example, if a voice input is provided from the user via the microphone 110, the result of performing speech recognition may be displayed on the display 180 and output via the speaker 170. Of course, the speaker 170 and the display 180 can perform general sound output and screen output functions of the user terminal 100, respectively.

The server 200 may include a configuration for performing voice recognition of voice input provided from the user terminal 100 through the network 10. [ Accordingly, some components of the server 200 may correspond to the user terminal 100. For example, the server 200 may include a transceiver 210, a controller 220, an ASR module 230, an ASR model 240, and the like. The server 200 may further include an ASR model converter 250 or a NLP (Natural Language Processing) 260.

The controller 220 can control the function modules for performing speech recognition in the server 200. [ For example, controller 220 may be coupled to ASR module 230 and / or NLP 260. In addition, the controller 220 can perform a function related to the recognition model update in association with the user terminal 100. In addition, the controller 220 may perform preprocessing on the voice signal transmitted through the network 10 and provide it to the ASR module 230. The preprocessing herein may have a different approach or effect from the preprocessing performed in the user terminal 100. In some embodiments, the controller 220 of the server 200 may be referred to as an orchestrator.

The ASR module 230 can perform speech recognition on the voice signal provided from the controller 220. At least a portion of the description of the ASR module 130 may be applied to the ASR module 230. However, the server ASR module 230 and the ASR module 130 for the user terminal perform some similar functions, but may include different functional ranges or algorithms. The ASR module 230 performs speech recognition based on the ASR model 240, thereby generating a result different from the speech recognition result of the ASR module 130 of the user terminal 100. Specifically, the server 200 generates recognition results by speech recognition, natural language understanding (NLU), and dialog management (DM), or a combination thereof, by the ASR module 230 and the NLP 260, The ASR module 130 can generate the recognition result. For example, the first operation information and the first reliability may be determined for the ASR performance result of the ASR module 130, and the second operation information and the second reliability may be determined as a result of performing the voice recognition of the ASR module 230. In some embodiments, the execution results of the ASR module 130 and the ASR module 230 may be consistent, or at least partially different. For example, the first operation information and the second operation information correspond to each other, but the second reliability may be higher than the first reliability. In various embodiments, speech recognition (ASR) performed by the ASR module 130 of the user terminal 100 is referred to as a first speech recognition, and speech recognition (ASR) performed by the ASR module 230 of the server 200 is referred to as second speech recognition .

In various embodiments, when the algorithm of the first speech recognition performed by the ASR module 130 is different from the algorithm of the second speech recognition performed by the ASR module 230, or when the model used for speech recognition is different, the ASR The model converter 250 may be included in the server 200.

In addition, the server 200 may include an NLP 260 for determining a user's intention based on a result recognized by the ASR module 230 and determining a function to be performed. NLP 260 can perform natural language processing that mechanically analyzes the human language uttered by human beings and expresses them in a language that humans can understand.

2 shows an electronic device and a server according to another embodiment of the present invention.

Fig. 2 shows an example of an electronic device implemented in a manner different from Fig. However, the speech recognition method disclosed in this document can be performed by various types of apparatus that can be modified from the electronic device / user terminal of Figs. 1 and 2, or FIGS. 7 and 8 described later.

Referring to FIG. 2, the user terminal 101 may include a processor 121 and a memory 141. The processor 121 may include an ARS engine 131 for performing speech recognition. The memory 141 may store an ASR model 143 used by the ASR engine 131 to perform speech recognition. For example, the processor 121, the ASR engine 131, and the ASR model 143 (or the memory 141) of FIG. 2 correspond to the controller 120, the ASR module 130, and the ASR model 140 of FIG. 1 . ≪ / RTI > Description of corresponding or redundant description will be omitted below.

The user terminal 101 can acquire voice input from the user using the voice recognition module 111 (e.g., the microphone 110). The processor 121 can perform the ASR using the ASR model 143 stored in the memory 141 with respect to the acquired voice input. Also, the user terminal 101 can provide voice input to the server 200 via the communication module 151 and receive voice commands (e.g., second operation information) corresponding to voice input from the server 200. [ The user terminal 101 can output the speech recognition result obtainable by the ASR engine 131 and the server 200 using the display 181 (or the speaker).

Hereinafter, various speech recognition methods will be described based on the user terminal 100 with reference to FIGS. 3 to 6. FIG.

3 is a flowchart illustrating a method of performing speech recognition according to an embodiment of the present invention.

In operation 301, the user terminal 100 may use a voice acquisition module, such as a microphone, to obtain a user's voice input. This operation can be performed in the state where a specific function or application associated with voice recognition is executed by the user. However, in some embodiments, speech recognition of the user terminal 100 may always be always-on (e.g., the microphone is always active), in which case the action 301 may always be performed for the user's utterance. Alternatively, the ASR may be activated by a specific speech input (e.g., Hi, Galaxy) by different speech recognition engines as described above, and an ASR for speech recognition that is subsequently input may be performed.

At operation 303, user terminal 100 may transmit a voice signal (or at least a portion of a voice signal) to server 200. Internally, a voice signal (or an audio signal whose speech input is converted into a (digital) voice signal and preprocessed for the voice signal) may be provided to the ASR module 130 by a processor (e.g., controller 120). In other words, in operation 303, the user terminal 100 can provide a voice signal to be recognized by the ASR module located inside and outside the device capable of performing voice recognition. The user terminal 100 can utilize both its own voice recognition and voice recognition through the server 200. [

At operation 305, the user terminal 100 may perform its own speech recognition. This speech recognition can be referred to as ASR1. For example, the ASR module 130 can perform speech recognition for voice input using the ASR model 140. [ For example, the ASR model 140 may perform ASR1 for at least some of the voice signals. As a result of the execution of ASR1, the result of performing the voice input can be obtained. For example, when the user provides voice input such as " weather tomorrow ", the user terminal 100 can determine the operation information such as " weather application execution, tomorrow weather output " . In addition, the result of performing the speech recognition may include the reliability of the operation information in addition to the operation information. For example, the ASR module 130 may determine 95% confidence that "tomorrow weather" is clear as a result of analyzing a user utterance, but if it is not clear whether the result is "daily weather" or "tomorrow weather" 60% of the reliability can be given to the determined operation information.

At operation 307, the processor may determine if the reliability is above a specified threshold. For example, if the confidence level for the action information determined by the ASR module 130 is greater than or equal to a predetermined level (e.g., 80%), then in operation 309, the user terminal 100 determines ASR1, It is possible to perform an operation corresponding to the command. The operation may include, for example, at least one of at least one function executable by the processor, at least one application, or an input based on the results of the performance of the ASR.

Operation 309 may be performed before the result of speech recognition from server 200 is acquired (e.g., operation 315). In other words, when a voice command having sufficient reliability is recognized as a result of performing voice recognition by itself in the user terminal 100, the user terminal immediately performs the corresponding operation without waiting for the additional voice recognition result obtained from the server 200, It is possible to secure a quick response time for the input.

If the reliability is less than the threshold at operation 307, the user terminal 100 may wait until obtaining speech recognition results from the server 200 at operation 315. During the standby operation, the user terminal 100 may display an appropriate message, an icon, an image, etc. to indicate that voice recognition for voice input is being performed.

In operation 311, voice recognition by the server may be performed on the voice signal transmitted from the operation 303 to the server 200. This speech recognition can be referred to as ASR2. Natural language processing (NLP) may also be performed at operation 313. For example, the NLP 260 of the server 200 may perform NLP on the voice input or recognition result of ASR2. In some embodiments, this process may optionally be performed.

If voice recognition results (e.g., second operation information and second reliability) on which ASR2 or ASR2 and NLP have been performed are obtained from server 200 at operation 315, then voice commands recognized by ASR2 at operation 317 Information) can be performed. In the case of operation 317, in addition to performing speech recognition, it may have a longer response time than performing an operation according to operation 309, because it requires additional time to transmit a speech signal at operation 303 and acquire a speech recognition result at operation 315. However, an operation with relatively high reliability and accuracy can be performed for voice recognition that can not be processed by itself or can be processed by itself in operation 317 but has low reliability.

4 is a flowchart illustrating a method of performing speech recognition according to another embodiment of the present invention.

4, the voice acquisition operation 401, the voice signal transmission operation 403, the ASR1 operation 405, the ASR2 operation 415, and the NLP operation 417 correspond to the operations 301, 303, 305, 311, and 313 described above in FIG. 3 The description thereof will be omitted.

The speech recognition performance method described with reference to FIG. 4 is performed based on two threshold values. (1) when the reliability of the result of the ASR1 operation of the operation 405 is equal to or greater than the first threshold value, based on the first threshold value and the second threshold value corresponding to the reliability lower than the first threshold value, (2) And (3) if they are between the first threshold and the second threshold, different actions (e.g., actions 409, 413, 421, respectively) may be performed.

If the reliability at operation 407 is equal to or greater than the first threshold, the user terminal 100 may perform an operation corresponding to the result of the ASR1 operation at operation 409. [ If the reliability at operation 407 is less than the first threshold, then operation 411 may determine whether the reliability is below a second threshold.

In operation 411, if the reliability is below a second threshold, the user terminal 100 may provide feedback on the reliability. This feedback may include a message or audio output indicating that the user's voice input was not normally recognized by the electronic device or that the recognition result was not reliable. For example, the user terminal 100 may " Please tell me again "can be displayed or output via the speaker. Alternatively, the user terminal 100 may derive a relatively easy to recognize speech input (e.g. yes, no, no, no, no, etc.) through feedback such as, for example, The accuracy of recognition results with low reliability can be confirmed.

If feedback is provided at operation 413, then operation 421 may not be performed even though a speech recognition result is obtained at operation 419 over time thereafter. A new voice input may be generated from the user by feedback, and in this case, it is not appropriate that the operation for the previously generated voice input is performed. However, in some embodiments, no additional input is generated from the user for a period of time despite the feedback of action 413, and the speech recognition results (e.g., second action information and second confidence) received from server 200 at action 419 Operation 421 may be performed after operation 413 if the specified condition is met (e.g., the second reliability is greater than or equal to the first threshold or any third threshold).

At operation 411, if the reliability obtained at operation 405 is greater than or equal to a second threshold (i.e., if the reliability is between the first threshold and the second threshold) The recognition result can be obtained. In operation 421, the user terminal 100 may perform an operation corresponding to the voice command (second operation information) recognized by the ASR2.

In the embodiment of FIG. 4, the reliability of the speech recognition result by the user terminal 100 is divided into usable levels by referring to the usable level and the unusable level and the ASR result of the server 200 so that proper operation is performed according to the reliability can do. In particular, when the reliability is too low, the user terminal 100 provides feedback regardless of whether the result is received from the server 200 to induce re-input to the user, A message can be prevented from being provided.

5 shows a flow diagram of a method for updating a threshold according to an embodiment of the present invention.

5, the voice acquisition operation 501, the voice signal transmission operation 503, the ASR1 operation 505, the ASR2 operation 511, and the NLP operation 513 correspond to the operations 301, 303, 305, 311, and 313 described above in FIG. 3 The description thereof will be omitted.

If the reliability for the result of the execution of ASR1 is equal to or greater than the threshold value (e.g., the first threshold value) at operation 507, the operation proceeds to operation 509 to perform an operation corresponding to the voice command recognized by ASR1 (e.g., first operation information) . If the reliability of the result of the ASR1 is less than or equal to the threshold value in operation 507, the operation after operation 315 in FIG. 3 or the operation after operation 411 in FIG. 4 may be performed.

In the embodiment of FIG. 5, operations 515 through 517 may be performed without the process terminating even after operation 509 is performed. At operation 515, the user terminal 100 may obtain a speech recognition result from the server 200. [ For example, the user terminal 100 may obtain the second reliability information and the second operation information that is the result of the performance of the ASR2 with respect to the voice signal transmitted in operation 503. [

At operation 517, the user terminal 100 may compare the recognition results of ASR1 and ASR2. For example, the user terminal can determine whether the recognition results of ASR1 and ASR2 are the same (or correspond) or different from each other. For example, when a voice is recognized as in the case of "recognition of tomorrow" as a result of recognition of ASR1, and voice is recognized as a result of recognition of ASR2 such as "what is the weather tomorrow?", In both cases, the operation information is " Output " In such a case, it can be understood that the recognition results of ASR1 and ASR2 correspond to each other. However, when different operations are performed as a result of speech recognition, it can be determined that two (or more) speech recognition results do not correspond to each other.

In operation 519, the user terminal 100 may change the threshold value by comparing the result of performing its own ASR with the voice command received from the server. For example, the user terminal 100 may reduce the first threshold if the first operation information and the second operation information include the same or corresponding voice commands. For example, if a voice input has a reliability of 80% or more in the past, if the result of performing the speech recognition of the user terminal 100 itself is utilized without waiting for a response from the server 200, only the reliability of 70% The speech recognition result of the user terminal 100 itself can be utilized. The threshold update can be repeatedly performed each time the user uses the voice recognition function, and as a result, a low threshold value is set for the voice recognition frequently used by the user, so that a quick response time can be obtained.

However, if the execution result of ASR1 differs from the execution result of ASR2, the threshold value may be increased. In some embodiments, the threshold update operation may occur after the specified condition is accumulated for a specified number of times. For example, the threshold value may be updated (lowered) when the number of times the ASR1 execution result and the ASR2 execution result match for any voice input occurs five times or more.

6 is a flowchart of a method for updating a speech recognition model according to an embodiment of the present invention.

6, the voice acquisition operation 601, the voice signal transmission operation 603, the ASR1 operation 605, the ASR2 operation 611, and the NLP operation 613 correspond to the operations 301, 303, 305, 311, and 313 described above in FIG. 3 The description thereof will be omitted.

If the reliability for the result of the execution of ASR1 is less than or equal to the threshold value (e.g., the first threshold value) in operation 607, operations after operation 309 in FIG. 3, operation 409 in FIG. 4, and operation 509 in FIG. 5 may be performed.

If the confidence in the result of the performance of ASR1 at operation 607 is less than the threshold, at operation 609, the user terminal 100 obtains a speech recognition result from the server 200 and performs an operation corresponding to the voice command recognized by ASR2 at operation 615 can do. Operations 609 and 615 may correspond to operations 315 and 317 in FIG. 3, or operations 419 and 421 in FIG.

At operation 617, the user terminal 100 may compare the speech recognition results of ASR1 and ASR2. Operation 617 may correspond to operation 517 of FIG.

At operation 619, user terminal 100 may update the speech recognition model (e.g., ASR model 140) of user terminal 100 based on the result of the comparison of operation 517. For example, the user terminal 100 may add a speech recognition result (e.g., second operation information, or second operation information and second reliability) of ASR2 to speech input to the speech recognition model. For example, when the first operation information and the second operation information do not correspond to each other, the user terminal 100 determines, based on the first reliability and the second reliability (e.g., when the second reliability is higher than the first reliability) The second operation information (and the second reliability) can be added to the speech recognition model used for the first speech recognition. Similar to the embodiment of Fig. 5, the speech recognition model update operation may occur after the specified condition is accumulated for the specified number of times.

Figure 7 illustrates an electronic device in a network environment in accordance with one embodiment of the present invention.

Referring to FIG. 7, an electronic device 701 in a network environment 700 in various embodiments is described. The electronic device 701 may include a bus 710, a processor 720, a memory 730, an input / output interface 750, a display 760, and a communication interface 770. In some embodiments, the electronic device 701 may omit at least one of the components or additionally include other components.

The bus 710 may include circuitry, for example, to connect components 710-170 to each other and to communicate communications (e.g., control messages and / or data) between the components.

Processor 720 may include one or more of a central processing unit (CPU), an application processor (AP), or a communication processor (CP). Processor 720 may perform computations or data processing related to, for example, control and / or communication of at least one other component of electronic device 701. [

Memory 730 may include volatile and / or nonvolatile memory. Memory 730 may, for example, store instructions or data related to at least one other component of electronic device 701. [ According to one embodiment, memory 730 may store software and / or program 740. The program 740 may include, for example, a kernel 741, a middleware 743, an application programming interface (API) 745, and / or an application program (or "application") 747. At least a portion of the kernel 741, middleware 743, or API 745 may be referred to as an operating system (OS).

The kernel 741 may include, for example, system resources (e.g., bus 710, processor 720, or 710) used to execute an operation or function implemented in other programs (e.g., middleware 743, API 745, or application program 747) Memory 730, etc.). In addition, the kernel 741 may provide an interface that can control or manage system resources by accessing individual components of the electronic device 701 in the middleware 743, API 745, or application program 747.

The middleware 743, for example, can perform an intermediary role so that the API 745 or the application program 747 can communicate with the kernel 741 to exchange data.

In addition, the middleware 743 can process one or more task requests received from the application program 747 according to the priority order. For example, the middleware 743 may prioritize the use of the system resources (e.g., bus 710, processor 720, or memory 730, etc.) of the electronic device 701 in at least one of the application programs 747. For example, the middleware 743 can perform the scheduling or load balancing of the one or more task requests by processing the one or more task requests according to the priority assigned to the at least one task.

The API 745 is, for example, an interface for an application 747 to control the functions provided by the kernel 741 or the middleware 743, and includes at least one interface for controlling, for example, file control, window control, image processing, Interfaces or functions (e.g., commands).

The input / output interface 750 may serve as an interface through which commands or data input from, for example, a user or other external device can be communicated to another component (s) of the electronic device 701. Output interface 750 can output commands or data received from other component (s) of the electronic device 701 to a user or other external device.

The display 760 can be, for example, a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, or a microelectromechanical systems (MEMS) . ≪ / RTI > Display 760 may display various content (e.g., text, image, video, icon, or symbol, etc.) to a user, for example. Display 760 may include a touch screen and may receive touch, gesture, proximity, or hovering input using, for example, an electronic pen or a portion of the user's body.

Communication interface 770 may establish communication between, for example, electronic device 701 and an external device (e.g., first external electronic device 702, second external electronic device 704, or server 706). For example, the communication interface 770 may be connected to the network 762 via wireless or wired communication to communicate with the external device (e.g., the second external electronic device 704 or the server 706).

The wireless communication may use at least one of, for example, LTE, LTE-A, CDMA, WCDMA, UMTS, WiBro, or GSM as the cellular communication protocol. The wireless communication may also include, for example, local communication 764. The near field communication 764 may include at least one of, for example, Wi-Fi, Bluetooth, near field communication (NFC), or global positioning system (GPS) The wired communication may include at least one of, for example, a universal serial bus (USB), a high definition multimedia interface (HDMI), a recommended standard 832 (RS-232), or plain old telephone service (POTS). Network 762 may include at least one of a telecommunications network, e.g., a computer network (e.g., LAN or WAN), the Internet, or a telephone network.

Each of the first and second external electronic devices 702, 704 may be the same or a different kind of device as the electronic device 701. According to one embodiment, the server 706 may include one or more groups of servers. According to various embodiments, all or a portion of the operations performed on the electronic device 701 may be performed on one or more other electronic devices (e.g., electronic device 702, 704, or server 706). According to one embodiment, in the case where the electronic device 701 has to perform some function or service automatically or upon request, the electronic device 701 may perform at least some functions associated therewith (E.g., electronic device 702, 704, or server 706). Other electronic devices (e.g., electronic device 702, 704, or server 706) may execute the requested function or additional function and deliver the result to electronic device 701. The electronic device 701 can directly or additionally process the received result to provide the requested function or service. For this purpose, for example, cloud computing, distributed computing, or client-server computing technology may be used.

8 shows a block diagram of an electronic device according to an embodiment of the present invention.

Referring to Fig. 8, the electronic device 801 may include all or part of the electronic device 701 shown in Fig. 7, for example. The electronic device 801 includes one or more processors (e.g., an application processor (AP)) 810, a communication module 820, a subscriber identification module 824, a memory 830, a sensor module 840, an input device 850, a display 860, an interface 870, 891, a power management module 895, a battery 896, an indicator 897, and a motor 898.

The processor 810 may, for example, operate an operating system or an application program to control a plurality of hardware or software components coupled to the processor 810, and may perform various data processing and operations. The processor 810 may be implemented as a system on chip (SoC), for example. According to one embodiment, the processor 810 may further include a graphics processing unit (GPU) and / or an image signal processor. Processor 810 may include at least some of the components shown in FIG. 8 (e.g., cellular module 821). Processor 810 may load and process instructions or data received from at least one of the other components (e.g., non-volatile memory) into volatile memory and store the various data in non-volatile memory.

The communication module 820 may have the same or similar configuration as the communication interface 770 of FIG. The communication module 820 may include, for example, a cellular module 821, a Wi-Fi module 823, a Bluetooth module 825, a GPS module 827, an NFC module 828 and a radio frequency (RF) module 829.

The cellular module 821 can provide voice calls, video calls, text services, or Internet services, for example, over a communication network. According to one embodiment, the cellular module 821 can perform identification and authentication of the electronic device 801 within the communication network using a subscriber identity module (e.g., SIM card) 824. According to one embodiment, the cellular module 821 may perform at least some of the functions that the processor 810 may provide. According to one embodiment, the cellular module 821 may include a communications processor (CP).

Each of the Wi-Fi module 823, the Bluetooth module 825, the GPS module 827, or the NFC module 828 may include, for example, a processor for processing data transmitted and received through a corresponding module. According to some embodiments, at least some (e.g., two or more) of the cellular module 821, the Wi-Fi module 823, the Bluetooth module 825, the GPS module 827 or the NFC module 828 may be included in one integrated chip (IC) .

The RF module 829 can, for example, send and receive communication signals (e.g., RF signals). The RF module 829 may include, for example, a transceiver, a power amplifier module (PAM), a frequency filter, a low noise amplifier (LNA), or an antenna. According to another embodiment, at least one of the cellular module 821, the Wi-Fi module 823, the Bluetooth module 825, the GPS module 827, or the NFC module 828 can transmit and receive RF signals through separate RF modules.

The subscriber identity module 824 may include, for example, a card containing a subscriber identity module and / or an embedded SIM and may include unique identification information (e.g., an integrated circuit card identifier (ICCID) (E.g., international mobile subscriber identity (IMSI)).

Memory 830 (e.g., memory 730) may include, for example, internal memory 832 or external memory 834. The internal memory 832 may be a volatile memory such as a dynamic RAM (DRAM), a static random access memory (SRAM), or a synchronous dynamic RAM (SDRAM), a non-volatile memory such as an OTPROM one time programmable ROM), programmable ROM (PROM), erasable and programmable ROM (EPROM), electrically erasable and programmable ROM (EEPROM), mask ROM, flash ROM, flash memory (e.g., NAND flash, or NOR flash), a hard drive, or a solid state drive (SSD).

The external memory 834 may be a flash drive, for example, a compact flash (CF), a secure digital (SD), a micro-SD, a mini-SD, an extreme digital (xD), a multi- a memory stick, and the like. The external memory 834 may be functionally and / or physically connected to the electronic device 801 through various interfaces.

The sensor module 840 may, for example, measure a physical quantity or sense the operating state of the electronic device 801 and convert the measured or sensed information into an electrical signal. The sensor module 840 may include, for example, a gesture sensor 840A, a gyro sensor 840B, an air pressure sensor 840C, a magnetic sensor 840D, an acceleration sensor 840E, a grip sensor 840F, a proximity sensor 840G, a color sensor 840H , An on / humidity sensor 840J, an illuminance sensor 840K, or an ultraviolet (UV) sensor 840M. Additionally or alternatively, the sensor module 840 may include, for example, an E-nose sensor, an EMG (electromyography) sensor, an EEG (electroencephalogram) sensor, an ECG (electrocardiogram) sensor, an IR And / or a fingerprint sensor. The sensor module 840 may further include a control circuit for controlling at least one sensor belonging to the sensor module 840. In some embodiments, the electronic device 801 may further include a processor configured to control the sensor module 840, either as part of the processor 810 or separately, to control the sensor module 840 while the processor 810 is in a sleep state .

The input device 850 may include, for example, a touch panel 852, a (digital) pen sensor 854, a key 856, or an ultrasonic input device 858. The touch panel 852 can employ, for example, at least one of an electrostatic type, a pressure sensitive type, an infrared type, and an ultrasonic type. Further, the touch panel 852 may further include a control circuit. The touch panel 852 may further include a tactile layer to provide a tactile response to the user.

 The (digital) pen sensor 854 may be, for example, part of a touch panel or may include a separate recognition sheet. Key 856 may include, for example, a physical button, an optical key, or a keypad. The ultrasonic input device 858 can detect the ultrasonic wave generated by the input tool through the microphone (e.g., the microphone 888) and confirm the data corresponding to the ultrasonic wave detected.

Display 860 (e.g., display 760) may include panel 862, hologram device 864, or projector 866. Panel 862 may include the same or similar configuration as display 760 of FIG. The panel 862 can be embodied, for example, flexible, transparent, or wearable. The panel 862 may be composed of a touch panel 852 and one module. The hologram device 864 can display stereoscopic images in the air using the interference of light. The projector 866 can display images by projecting light onto the screen. The screen may be located, for example, inside or outside the electronic device 801. According to one embodiment, the display 860 may further include control circuitry for controlling the panel 862, the hologram device 864, or the projector 866.

The interface 870 may include, for example, an HDMI 872, a USB 874, an optical interface 876, or a D-sub (D-subminiature) 878. The interface 870 may be included in the communication interface 770 shown in Fig. 7, for example. Additionally or alternatively, interface 870 may include, for example, a mobile high-definition link (MHL) interface, an SD card / MMC interface, or an infrared data association (IrDA) interface.

Audio module 880 can, for example, bidirectionally convert sound and electrical signals. At least some of the components of the audio module 880 may be included, for example, in the input / output interface 750 shown in FIG. The audio module 880 can process sound information input or output through, for example, a speaker 882, a receiver 884, an earphone 886, a microphone 888, or the like.

The camera module 891 is, for example, a device capable of capturing still images and moving images, and according to one embodiment, one or more image sensors (e.g., a front sensor or a rear sensor), a lens, an image signal processor And may include a flash (e.g., LED or xenon lamp).

The power management module 895 can manage the power of the electronic device 801, for example. According to one embodiment, the power management module 895 may include a power management integrated circuit (PMIC), a charger integrated circuit (PMIC), or a battery or fuel gauge. The PMIC may have a wired and / or wireless charging scheme. The wireless charging system may include, for example, a magnetic resonance system, a magnetic induction system, or an electromagnetic wave system, and may further include an additional circuit for wireless charging, for example, a coil loop, a resonant circuit, have. The battery gauge can measure, for example, the remaining amount of the battery 896, the voltage during charging, the current, or the temperature. The battery 896 may include, for example, a rechargeable battery and / or a solar battery.

The indicator 897 may indicate a particular state of the electronic device 801 or a portion thereof (e.g., processor 810), such as a boot state, a message state, or a state of charge. The motor 898 can convert an electrical signal to mechanical vibration and can generate vibration, haptic effects, and the like. Although not shown, the electronic device 801 may include a processing unit (e.g., a GPU) for mobile TV support. The processing device for mobile TV support can process media data conforming to standards such as digital multimedia broadcasting (DMB), digital video broadcasting (DVB), or MediaFlo (TM), for example.

Each of the components described in this document may be composed of one or more components, and the name of the component may be changed according to the type of the electronic device. In various embodiments, the electronic device may comprise at least one of the components described herein, some components may be omitted, or may further include additional other components. In addition, some of the components of the electronic device according to various embodiments may be combined into one entity, so that the functions of the components before being combined can be performed in the same manner.

As used in this document, the term "module" may refer to a unit comprising, for example, one or a combination of two or more of hardware, software or firmware. A "module" may be interchangeably used with terms such as, for example, unit, logic, logical block, component, or circuit. A "module" may be a minimum unit or a portion of an integrally constructed component. A "module" may be a minimum unit or a portion thereof that performs one or more functions. "Modules" may be implemented either mechanically or electronically. For example, a "module" may be an application-specific integrated circuit (ASIC) chip, field-programmable gate arrays (FPGAs) or programmable-logic devices And may include at least one.

At least a portion of a device (e.g., modules or functions thereof) or a method (e.g., operations) according to various embodiments may include, for example, computer-readable storage media in the form of program modules, As shown in FIG.

For example, the storage medium may include instructions that, when executed, cause the processor of the electronic device to obtain a speech input from a user to generate a speech signal, perform a first speech recognition on at least a portion of the speech signal, Receiving at least a portion of the speech signal to a server for performing a second speech recognition, receiving second operation information for the transmitted signal from the server, (1) performing a function corresponding to the first operation information when the first reliability is equal to or greater than a first threshold value, (2) performing a function corresponding to the first reliability if the first reliability is less than a second threshold value, (3) to perform a function corresponding to the second operation information when the first reliability is between the first threshold value and the second threshold value, You may be saving your words.

Modules or program modules according to various embodiments may include at least one or more of the elements described above, some of which may be omitted, or may further include additional other elements. Operations performed by modules, program modules, or other components in accordance with various embodiments may be performed in a sequential, parallel, iterative, or heuristic manner. Also, some operations may be performed in a different order, omitted, or other operations may be added.

And the embodiments disclosed in this document are presented for the purpose of explanation and understanding of the disclosed technical contents, and do not limit the scope of the present invention. Accordingly, the scope of this document should be interpreted to include all modifications based on the technical idea of the present invention or various other embodiments.

Claims (20)

In an electronic device,
A processor for performing automatic speech recognition (ASR) on speech input using a speech recognition model stored in a memory,
And a communication module for providing the voice input to a server and receiving voice commands corresponding to the voice input from the server,
(1) when the reliability of the ASR is greater than or equal to a first threshold, performing an operation corresponding to the result of the ASR; (2) when the reliability of the ASR is less than a second threshold And to provide feedback on the reliability. The method according to claim 1,
And the processor is configured to (3) perform the voice command received from the server if the reliability is between the first threshold and the second threshold. The method according to claim 1,
And when the reliability is equal to or greater than the first threshold value, the operation is set to be performed regardless of whether or not the voice command is received. The method of claim 3,
Wherein the operation is set to include at least one of an input based on at least one function executable by the processor, at least one application, or an execution result of the ASR. The method according to claim 1,
Wherein the feedback is set to include a message or audio output indicating that the speech input is not recognized or that the result of the performance is unreliable. The method according to claim 1,
Wherein the voice command received from the server corresponds to a result of performing voice recognition on the voice input provided by the server based on a voice recognition model different from the voice recognition model stored in the memory. The method of claim 6,
Wherein the speech recognition performed at the server comprises NLP (Natural Language Processing). The method according to claim 1,
Wherein the processor is configured to provide an audio signal to the ASR engine performing the ASR and to provide the audio input itself to the server via the communication module, Device. The method according to claim 1,
Wherein the processor is configured to compare the result of the performance of the ASR with the voice command received from the server and to change the first threshold based on the comparison result if the confidence is above the first threshold. The method of claim 9,
Wherein the processor is configured to decrease the first threshold value if the result of the ASR and the voice command received from the server correspond, and to increase the first threshold value if the voice command does not correspond to each other. The method according to claim 1,
Wherein the processor is configured to compare the result of performing the ASR with the voice command received from the server when the reliability is less than the first threshold and to update the speech recognition model based on a result of the comparison. The method of claim 11,
Wherein the communication module receives reliability from the server with the voice command for the voice command,
Wherein the processor is configured to add confidence for the speech command and the speech command to the speech recognition model. A method of performing speech recognition of an electronic device,
An operation of acquiring a voice input from a user to generate a voice signal,
Performing a first speech recognition on at least a portion of the speech signal to obtain first operational information and a first confidence score,
Transmitting at least a portion of the speech signal to a server for performing a second speech recognition,
Receiving second operation information on the transmitted signal from the server, and
(1) perform a function corresponding to the first operation information when the first reliability is equal to or greater than a first threshold value, and (2) provide feedback on the first reliability when the first reliability is less than a second threshold value And (3) performing a function corresponding to the second operation information when the first reliability is between the first threshold value and the second threshold value. 14. The method of claim 13,
Wherein the operation of performing the function corresponding to the first operation information when the first reliability is equal to or greater than the first threshold value is performed before the operation of receiving the second operation information. 14. The method of claim 13,
Further comprising: increasing the first threshold value when the function corresponding to the first operation information matches the function corresponding to the second operation information. 14. The method of claim 13,
Further comprising decreasing the first threshold value when the function corresponding to the first operation information does not match the function corresponding to the second operation information. 14. The method of claim 13,
And wherein the act of receiving the second action information further comprises receiving a second confidence together with the second action information. 18. The method of claim 17,
Further comprising adding to the speech recognition model used for the first speech recognition, the second operation information and the second reliability information for the speech input when the first reliability is less than the first threshold value. How to perform recognition. 18. The method of claim 17,
When the first operation information and the second operation information do not correspond to each other, the second operation information and the second reliability based on the first reliability and the second reliability, Further comprising adding to the model. 21. A storage medium storing computer-readable instructions, the computer-readable instructions, when executed by a processor of an electronic device,
An operation of acquiring a voice input from a user to generate a voice signal,
Performing a first speech recognition on at least a portion of the speech signal to obtain first operational information and a first confidence score,
Transmitting at least a portion of the speech signal to a server for performing a second speech recognition,
Receiving second operation information on the transmitted signal from the server, and
(1) perform a function corresponding to the first operation information when the first reliability is equal to or greater than a first threshold value, and (2) provide feedback on the first reliability when the first reliability is less than a second threshold value And (3) when the first reliability is between the first threshold value and the second threshold value, perform a function corresponding to the second operation information.

Images