KR20180116731A - Device for performing a task corresponding to user utterance - Google Patents

Abstract

The present invention relates to an electronic device performing a task corresponding to user′s utterance. According to an embodiment of the present invention, the electronic device comprises: a housing; a touch screen display which is arranged in the housing and is exposed through a first portion of the housing; a microphone which is arranged in the housing and is exposed through a second portion of the housing; at least one speaker which is arranged in the housing and is exposed through a third portion of the housing; a wireless communication circuit arranged in the housing; a processor which is arranged in the housing and is electrically connected to the touch screen display, the microphone, the speaker, and the wireless communication circuit; and a memory which is arranged in the housing and is electrically connected to the processor. In the present invention, provided is the electronic device performing a task in accordance with user′s intent when ambiguous user utterance is received.

Description

[0001] The present invention relates to an electronic device for performing a task corresponding to a user's utterance,

The embodiments disclosed herein relate to techniques for performing tasks corresponding to user utterances.

In addition to traditional input methods using a keyboard or a mouse, modern electronic devices can support a variety of input methods such as voice input. For example, electronic devices such as a smart phone or a tablet may recognize a user's voice input while the voice recognition service is running, perform an action corresponding to the voice input, or provide search results.

Recently, speech recognition service has been developed based on technology for processing natural language. The technology for processing natural language is a technique for grasping the intention of user utterance and providing the user with an intentional result.

The speech recognition service can suggest only one task using user utterance. Accordingly, when an existing service uses an ambiguous user utterance, it proposes a task that does not match the intention of the user, thereby inconveniencing the user.

Various embodiments of the present invention seek to propose an electronic device that performs tasks that are consistent with a user's intent when ambiguous user utterances are received.

An electronic device according to an embodiment disclosed in this document includes a housing; A touch screen display disposed within the housing and exposed through a first portion of the housing; A microphone disposed within the housing and exposed through a second portion of the housing; At least one speaker disposed within the housing and exposed to a third portion of the housing; A wireless communication circuit disposed within the housing; A processor disposed within the housing and electrically coupled to the touch screen display, the microphone, the at least one speaker, and the wireless communication circuitry; And a memory disposed in the housing and electrically coupled to the processor, wherein the memory, in execution, causes the processor to receive, in a first example, a first user utterance via the microphone, Wherein said first request comprises a first request to perform said first task with said electronic device and transmits first data associated with said first user utterance to an external server via said wireless communication circuit, Receiving a first response from an external server, the first response including information about sequences of first states of the electronic device for performing the first task, and wherein the electronic device is operable to receive sequences of first states To perform the first task, and in a second example, to receive a second user utterance via the microphone Wherein the second user utterance includes a second request to perform the second task with the electronic device and transmits second data associated with the second user utterance to the external server via the wireless communication circuit, Receiving a second response from the external server via a wireless communication circuit, the second response including at least one sample speech indicating the intent of the user utterance, and the at least one sample speech being associated with the second user utterance For displaying the at least one sample speech on the touch screen display and for selecting one of the at least one sample speech through the microphone or the touch screen display, Wherein the electronic device is adapted to receive the selected one of the at least one sample speech To perform the second task by having a sequence of second states associated with the second task.

Further, an electronic device according to an embodiment disclosed in this document includes: a housing; A touch screen display disposed within the housing and exposed through a first portion of the housing; A microphone disposed within the housing and exposed through a second portion of the housing; At least one speaker disposed within the housing and exposed to a third portion of the housing; A wireless communication circuit disposed within the housing; A processor disposed within the housing and electrically coupled to the touch screen display, the microphone, the at least one speaker, and the wireless communication circuitry; And a memory disposed in the housing and electrically coupled to the processor, wherein the memory, in execution, causes the processor to receive a user utterance through the microphone, The method comprising: sending a request for performing a user utterance to the external server via the wireless communication circuit; receiving a response from the external server via the wireless communication circuit; Wherein the plurality of sample utterances comprises a first sample utterance corresponding to a first pass rule and a second sample utterance corresponding to a second pass rule, Wherein the touch screen display is generated by the external server based on data associated with an utterance, Receiving a first user input for selecting one of the at least one sample speech via the microphone or the touch screen display and the electronic device displaying a selected one of the at least one sample speech And to have the task have a sequence of states associated with it.

Also, a storage medium according to one embodiment disclosed herein is a storage medium storing computer-readable instructions, which when executed by a processor of an electronic device causes the electronic device to receive a user utterance Wherein the user utterance comprises a request to perform a task with the electronic device, the method comprising: sending data associated with the user utterance to an external server; receiving a response from the external server; Wherein the at least one sample utterance is selected by the external server based on the user utterance and displaying the at least one sample utterance, the at least one sample utterance indicating the at least one sample A first user input for selecting one of the utterances; And performing the task by causing the electronic device to have a sequence of states associated with a selected one of the at least one sample utterances.

According to various embodiments of the present invention, it is possible to display a candidate group corresponding to a task expected to meet a user's intention from an ambiguous user utterance, and receive an input for selecting a candidate, thereby enabling a user to execute a desired task.

According to various embodiments of the present invention, the reliability of the intelligent service can be improved by executing a task that matches the intention of the user.

1 is a diagram of an integrated intelligence system in accordance with various embodiments of the present invention.
2 is a block diagram illustrating a user terminal of an integrated intelligent system according to an embodiment of the present invention.
FIG. 3 is a diagram illustrating execution of an intelligent app of a user terminal according to an embodiment of the present invention.
4 is a block diagram illustrating an intelligent server of an integrated intelligence system according to an embodiment of the present invention.
FIG. 5 is a diagram illustrating a method of generating a path rule by a natural language understanding (NLU) module according to an embodiment of the present invention.
Figure 6 is a diagram illustrating the context module of the processor collecting current status according to an embodiment of the present invention.
7 is a diagram illustrating that a persona module according to an embodiment of the present invention manages user information.
8 is a flowchart illustrating a method of performing a task based on a user utterance request according to one embodiment.
9 is a block diagram illustrating a natural language understanding module of an intelligent server according to an embodiment.
10 illustrates an operation that a pass rule according to an embodiment includes.
FIG. 11A is a diagram showing that a user utterance is input when a gallery app according to an embodiment is executed. FIG.
11B is a diagram illustrating a screen in which a plurality of sample utterances are displayed according to an embodiment.
12 is a flowchart illustrating a method for transmitting a plurality of sample utterances by an intelligent server according to an embodiment.
13 is a flowchart illustrating a method for transmitting a plurality of sample utterances by an intelligent server according to an embodiment.
FIG. 14A is a diagram illustrating input of a user utterance in a state where a gallery app according to an embodiment is executed. FIG.
14B is a diagram illustrating a screen in which a plurality of sample utterances are displayed according to an embodiment.
14C is a diagram illustrating a screen in which information related to a selected sample speech according to an embodiment is displayed.
FIG. 14D is a diagram illustrating a screen to which an intention of a user according to an embodiment is input
15 is a diagram showing a screen in which an image related to a sample speech according to an embodiment is further displayed.
16A is a diagram showing that a user utterance is input in a state where a gallery app according to an embodiment is executed.
16B is a diagram showing a screen in which a sample speech is displayed according to a determined order according to an embodiment.
16C is a view showing a screen in which an image related to the sample speech corresponding to the priority according to the embodiment is further displayed.
16D is a diagram showing a screen in which the size of the sample speech is displayed differently according to the determined order according to an embodiment.
FIG. 17 is a flowchart illustrating a method for an electronic device according to an embodiment to perform a pass rule and then receive a feedback input to perform another pass rule.
18 is a diagram showing a screen on which an electronic device according to an embodiment receives a feedback input.
19 is a diagram showing a screen on which a sample speech corresponding to the remaining pass rule according to the embodiment is displayed.
20 is a flowchart illustrating a method for an electronic device according to an embodiment to perform a pass rule and then receive a feedback input to perform another pass rule.
21 is a diagram showing a screen on which an electronic device according to an embodiment receives a feedback input.
FIGS. 22A and 22B show a screen displaying a UI related to a function provided by an electronic device when negative feedback is received according to an embodiment.
23A shows a screen on which a text to be modified is displayed according to an embodiment.
23B shows a screen in which the text to be modified is modified according to an embodiment.
23C shows a result screen in which a text corresponding to a user utterance is modified according to an embodiment.
FIG. 23D shows a screen including a result of modifying a text corresponding to a user utterance according to an embodiment and a UI for updating a speech model corresponding to the corrected text.
FIG. 23E shows a screen including a pop-up window in response to a network connection failure according to an exemplary embodiment.
24A shows a screen for searching a path rule according to an embodiment.
24B shows a path rule search result screen according to a search word input according to an embodiment.
FIG. 24C shows a screen after a pass rule is selected according to an embodiment.
FIG. 25 shows a screen after a UI for feedback that a domain that does not match a user's intention according to an embodiment is selected.
26 shows a screen for performing a search in an electronic device according to an embodiment.
FIG. 27 shows a screen for performing a search on the Internet according to an embodiment.
28 is a block diagram of an electronic device in a network environment in accordance with various embodiments.
In the description of the drawings, the same or similar reference numerals may be used for the same or similar components.

Various embodiments of the invention will now be described with reference to the accompanying drawings. It should be understood, however, that the invention is not intended to be limited to the particular embodiments, but includes various modifications, equivalents, and / or alternatives of the embodiments of the invention.

Before describing an embodiment of the present invention, an integrated intelligent system to which an embodiment of the present invention can be applied will be described.

1 is a diagram of an integrated intelligence system in accordance with various embodiments of the present invention.

Referring to FIG. 1, the integrated intelligent system 10 may include a user terminal 100, an intelligent server 200, a personalization information server 300, or a proposal server 400.

The user terminal 100 can receive a service required by the user through an app stored in the user terminal 100 (or an application program) (e.g., an alarm app, a message app, a photo (gallery) Can be provided. For example, the user terminal 100 may execute and operate another application through an intelligent app (or a voice recognition app) stored in the user terminal 100. [ The user terminal 100 can receive the user input for executing the other application and executing the operation through the intelligent app. The user input may be received via, for example, a physical button, a touchpad, a voice input, a remote input, or the like. According to one embodiment, the user terminal 100 may correspond to various terminal devices (or electronic devices) such as a mobile phone, a smart phone, a personal digital assistant (PDA), or a notebook computer.

According to one embodiment, the user terminal 100 may receive the user's utterance as a user input. The user terminal 100 may receive a user's utterance and generate an instruction to operate the app based on the utterance of the user. Accordingly, the user terminal 100 can operate the application using the command.

The intelligent server 200 can receive user voice input from the user terminal 100 through a communication network and change the text data into text data. In another embodiment, the intelligent server 200 may generate (or select) a path rule based on the text data. The pass rule may include information about an action (or an operation) for performing the function of the app or information about a parameter necessary for executing the action. In addition, the pass rule may include a sequence of the actions of the app. The user terminal 100 may receive the pass rule, select an app according to the pass rule, and execute the action included in the pass rule in the selected app.

For example, the user terminal 100 may execute the operation and display a screen corresponding to the state of the user terminal 100 that has performed the operation on the display. As another example, the user terminal 100 may execute the operation and may not display the result of performing the operation on the display. The user terminal 100 may, for example, execute a plurality of operations and display only some of the results of the operations on the display. The user terminal 100 can display, for example, only the result of executing the last sequence of operations on the display. As another example, the user terminal 100 may receive a user's input and display the result of performing the operation on the display.

The personalization information server 300 may include a database storing user information. For example, the personalization information server 300 may receive user information (e.g., context information, application execution, etc.) from the user terminal 100 and store the information in the database. The intelligent server 200 can be used when receiving the user information from the personalization information server 300 through a communication network and generating a pass rule for user input. According to one embodiment, the user terminal 100 can receive user information from the personalization information server 300 via a communication network and use the information as information for managing the database.

The proposal server 400 may include a database in which information about an introduction of a function or an application or a function to be provided is stored in the terminal. For example, the proposal server 400 may receive a user information of the user terminal 100 from the personalization information server 300 and include a database of functions that the user can use. The user terminal 100 may receive information on the function to be provided from the proposal server 400 through a communication network and provide information to the user.

2 is a block diagram illustrating a user terminal of an integrated intelligent system according to an embodiment of the present invention.

2, a user terminal 100 may include an input module 110, a display 120, a speaker 130, a memory 140, or a processor 150. The user terminal 100 may further include a housing, and the configurations of the user terminal 100 may be placed on the inside of the housing or on the housing.

According to one embodiment, input module 110 may receive user input from a user. For example, input module 110 may receive user input from a connected external device (e.g., keyboard, headset). In another example, input module 110 may include a touch screen (e.g., a touch screen display) coupled with display 120. As another example, the input module 110 may include a hardware key (or physical key) located at the user terminal 100 (or the housing of the user terminal 100).

According to one embodiment, input module 110 may include a microphone (e.g., microphone 111 of FIG. 3) capable of receiving a user's utterance as a speech signal. For example, the input module 110 may include a speech input system, and may receive a speech of a user via the speech input system.

In accordance with one embodiment, the display 120 may display an image of an image, video, and / or application executable. For example, the display 120 may display a graphical user interface (GUI) of the app.

According to one embodiment, the speaker 130 may output a voice signal. For example, the speaker 130 may output a voice signal generated inside the user terminal 100 to the outside.

According to one embodiment, the user terminal 100 may further comprise a wireless communication circuit (not shown). The wireless communication circuit can support various communications and can support establishment of a communication channel between the user terminal 100 and the intelligent server 200 and wireless communication through the established communication channel.

According to one embodiment, the memory 140 may store a plurality of apps 141 and 143. The plurality of apps 141 and 143 stored in the memory 140 may be selected and executed and operated according to user input.

According to one embodiment, the memory 140 may include a database capable of storing information necessary to recognize user input. For example, the memory 140 may include a log database capable of storing log information. As another example, the memory 140 may include a persona database capable of storing user information.

According to one embodiment, the memory 140 stores a plurality of apps 141 and 143, and a plurality of apps 141 and 143 can be loaded and operated. For example, a plurality of applications 141 and 143 stored in the memory 140 can be loaded and operated by the execution manager module 153 of the processor 150. [ The plurality of apps 141 and 143 may include execution services 141a and 143a or a plurality of operations (or unit operations) 141b and 143b that perform functions. The execution services 141a and 143a are generated by the execution manager module 153 of the processor 150 and can execute a plurality of operations 141b and 143b.

According to one embodiment, when the operations 141b and 143b of the applications 141 and 143 are executed, the execution status screen according to the execution of the operations 141b and 143b may be displayed on the display 120. [ The execution status screen may be, for example, a screen in which operations 141b and 143b are completed. The execution status screen may be, for example, a screen in which partial execution of operations 141b and 143b is stopped (e.g., when parameters necessary for operations 141b and 143b are not input) .

According to one embodiment, execution services 141a and 143a may execute operations 141b and 143b in accordance with a path rule. For example, the execution services 141a and 143a are activated by the execution manager module 153, receive an execution request from the execution manager module 153 according to the path rule, and perform operations 141b, 143b can execute the operations of the apps 141, The execution services 141a and 143a can transmit completion information to the execution manager module 153 when the operations 141b and 143b are completed.

According to one embodiment, when a plurality of operations 141b and 143b are executed in the applications 141 and 143, the plurality of operations 141b and 143b can be executed sequentially. The execution services 141a and 143a can open the next operation (operation 2) and transmit the completion information to the execution manager module 153 when the execution of one operation (operation 1) is completed. Here, opening an arbitrary operation can be understood as transitioning an arbitrary operation to an executable state or preparing for execution of an arbitrary operation. In other words, if an operation is not opened, the operation can not be executed. When the completion information is received, the execution manager module 153 can transmit an execution request for the next operations 141b and 143b to the execution service (e.g., operation 2). According to one embodiment, when a plurality of apps 141 and 143 are executed, a plurality of apps 141 and 143 can be sequentially executed. For example, when the execution of the last operation of the first application 141 is completed and the completion information is received, the execution manager module 153 sends a request to execute the first operation of the second application 143 to the execution service 143a Can be transmitted.

According to one embodiment, when a plurality of operations 141b and 143b are executed in the apps 141 and 143, a result screen corresponding to execution of each of the plurality of executed operations 141b and 143b is displayed on the display 120 Can be displayed. According to one embodiment, only a part of a plurality of result screens resulting from the execution of the plurality of executed operations 141b and 143b may be displayed on the display 120. [

According to one embodiment, the memory 140 may store an intelligent app (e.g., a voice recognition app) associated with the intelligent agent 151. An app linked with the intelligent agent 151 can receive and process a user's utterance as a voice signal. According to one embodiment, an application associated with the intelligent agent 151 is operated by a specific input (e.g., input via a hardware key, input via a touch screen, specific voice input) input via the input module 110 .

According to one embodiment, the processor 150 may control the overall operation of the user terminal 100. For example, the processor 150 may control the input module 110 to receive user input. The processor 150 may control the display 120 to display an image. The processor 150 may control the speaker 130 to output a voice signal. The processor 150 may control the memory 140 to retrieve or store the necessary information.

According to one embodiment, the processor 150 may include an intelligent agent 151, a performance manager module 153, or an intelligent service module 155. In one embodiment, processor 150 may execute instructions stored in memory 140 to drive intelligent agent 151, execution manager module 153, or intelligent service module 155. The various modules referred to in various embodiments of the present invention may be implemented in hardware or in software. The operations performed by the intelligent agent 151, the execution manager module 153, or the intelligent service module 155 in various embodiments of the present invention may be understood as operations performed by the processor 150. [

According to one embodiment, the intelligent agent 151 may generate an instruction to operate the app based on the voice signal received at the user input. According to one embodiment, the execution manager module 153 may receive the generated commands from the intelligent agent 151 and select and execute and operate the apps 141 and 143 stored in the memory 140. According to one embodiment, the intelligent service module 155 can be used to manage user information and process user input.

The intelligent agent 151 may transmit the user input received through the input module 110 to the intelligent server 200 for processing.

According to one embodiment, the intelligent agent 151 may pre-process the user input before transmitting the user input to the intelligent server 200. [ According to one embodiment, the intelligent agent 151 may include an adaptive echo canceller (AEC) module, a noise suppression (NS) module, an end-point detection (EPD) module or an automatic gain control (AGC) module. The adaptive echo canceller may remove an echo included in the user input. The noise suppression module may suppress background noise included in the user input. The end point detection module detects an end point of the user voice included in the user input to find a portion where the user's voice exists. The automatic gain control module may adjust the volume of the user input to be suitable for recognizing and processing the user input. According to one embodiment, the intelligent agent 151 may include all of the preprocessing configurations for performance, while in other embodiments, the intelligent agent 151 may include some of the preprocessing configurations to operate at low power .

According to one embodiment, the intelligent agent 151 may include a wake up recognition module that recognizes the user's call. The wakeup recognition module can recognize the user's wakeup command through the voice recognition module and activate the intelligent agent 151 to receive the user input when the wakeup command is received. According to one embodiment, the wake-up recognition module of the intelligent agent 151 may be implemented in a low-power processor (e.g., a processor included in an audio codec). According to one embodiment, the intelligent agent 151 may be activated upon user input via a hardware key. When the intelligent agent 151 is activated, an intelligent app (e.g., a voice recognition app) associated with the intelligent agent 151 may be executed.

According to one embodiment, the intelligent agent 151 may include a speech recognition module for performing user input. The voice recognition module may recognize a user input for causing the application to perform an operation. For example, the speech recognition module may include a limited user (voice) input (e.g., a "click" to perform a capture operation when the camera app is running) and a " The same utterance, etc.). The voice recognition module that assists the intelligent server 200 and recognizes the user input can recognize and process the user command that can be processed in the user terminal 100, for example. According to one embodiment, a speech recognition module for executing user input of the intelligent agent 151 may be implemented in an app processor.

According to one embodiment, the voice recognition module of the intelligent agent 151 (including the voice recognition module of the wakeup module) may recognize the user input using an algorithm for voice recognition. The algorithm used to recognize the speech may be at least one of, for example, a hidden markov model (HMM) algorithm, an artificial neural network (ANN) algorithm, or a dynamic time warping (DTW) algorithm.

According to one embodiment, the intelligent agent 151 may convert the user's speech input into textual data. According to one embodiment, the intelligent agent 151 may pass the user's voice to the intelligent server 200 to receive the converted textual data. Accordingly, the intelligent agent 151 can display the text on the display 120. [

According to one embodiment, the intelligent agent 151 may receive the path rules sent from the intelligent server 200. [ According to one embodiment, the intelligent agent 151 may send the pass rule to the execution manager module 153. [

According to one embodiment, the intelligent agent 151 sends an execution result log according to the path rule received from the intelligent server 200 to the intelligence service module 155, The log may be accumulated and managed in the preference information of the user of the persona manager 155b.

According to one embodiment, the execution manager module 153 receives the path rules from the intelligent agent 151 and executes the apps 141 and 143, and the apps 141 and 143 execute the actions 141b , And 143b. For example, the execution manager module 153 may send command information for executing operations 141b and 143b to the apps 141 and 143, and may receive operations 141b and 143b from the apps 141 and 143, Can be received.

According to one embodiment, the execution manager module 153 transmits and receives command information for executing the operations 141b and 143b of the applications 141 and 143 between the intelligent agent 151 and the applications 141 and 143 . The execution manager module 153 binds the applications 141 and 143 to be executed according to the path rule and transmits command information of the operations 141b and 143b included in the path rule to the apps 141 and 143 can do. For example, the execution manager module 153 sequentially transmits the actions 141b and 143b included in the path rule to the apps 141 and 143, and sends the actions 141b and 143b of the apps 141 and 143 And can be executed sequentially in accordance with the above-mentioned pass rules.

According to one embodiment, the execution manager module 153 can manage the execution states of the operations 141b and 143b of the applications 141 and 143. [ For example, the execution manager module 153 may receive information on the execution status of the operations 141b and 143b from the apps 141 and 143. [ If the execution state of the operations 141b and 143b is, for example, a partial landing (for example, a parameter required for the operations 141b and 143b is not input) To the intelligent agent 151, information about the suspended state. The intelligent agent 151 can request the user to input necessary information (e.g., parameter information) using the received information. The execution manager module 153 can receive the utterance from the user when the execution state of the operations 141b and 143b is in another operation state and the execution manager module 153 can receive the utterances from the executing applications 141 and 143, To the intelligent agent 151, information on the execution state of the devices 141 and 143. The intelligent agent 151 may receive the parameter information of the user's utterance through the intelligent server 200 and may transmit the received parameter information to the execution manager module 153. [ The execution manager module 153 can change the parameters of the operations 141b and 143b to new parameters using the received parameter information.

According to one embodiment, the execution manager module 153 may pass the parameter information contained in the path rule to the apps 141 and 143. [ When a plurality of applications 141 and 143 are sequentially executed according to the path rule, the execution manager module 153 can transmit parameter information included in the path rule from one application to another.

According to one embodiment, the execution manager module 153 may receive a plurality of pass rules. The execution manager module 153 can select a plurality of path rules based on the user's utterance. For example, if the user's utterance identified some apps 141 that would perform some action 141a but did not specify other apps 143 that would run the other action 143b, A different app 143 (e.g., a message app, a telegram app) that can execute the same app 141 (e.g., a gallery app) to execute the action 141a and execute the remaining action 143b A plurality of other path rules can be received. The execution manager module 153 may execute the same operations 141b and 143b (e.g., the same successive operations 141b and 143b) of the plurality of path rules. The execution manager module 153 may display a status screen on the display 120 that allows the user to select different apps 141 and 143 included in the plurality of path rules.

According to one embodiment, the intelligent service module 155 may include a context module 155a, a persona module 155b, or a suggestion module 155c.

The context module 155a may collect the current status of the apps 141 and 143 from the apps 141 and 143. [ For example, the context module 155a may receive context information indicating the current status of the apps 141 and 143 to collect the current status of the apps 141 and 143. [

The persona module 155b can manage the personal information of the user using the user terminal 100. [ For example, the persona module 155b may collect usage information and execution results of the user terminal 100 to manage the user's personal information.

The proposal module 155c predicts the user's intention and can recommend the command to the user. For example, the suggestion module 155c may recommend commands to the user in consideration of the user's current state (e.g., time, place, situation, app).

FIG. 3 is a diagram illustrating execution of an intelligent app of a user terminal according to an embodiment of the present invention.

Referring to FIG. 3, the user terminal 100 receives a user input and executes an intelligent application (e.g., a voice recognition application) linked with the intelligent agent 151.

According to one embodiment, the user terminal 100 may execute an intelligent app for voice recognition via the hardware key 112. [ For example, the user terminal 100 may display the user interface (UI) 121 of the intelligent app on the display 120 when the user input is received via the hardware key 112. The user can touch the voice recognition button 121a to the UI 121 of the intelligent app to input (113b) voice, for example, while the UI 121 of the intelligent app is displayed on the display 120 . The user may press the hardware key 112 to input a voice 113b, for example, to input a voice 113b.

According to one embodiment, the user terminal 100 may execute an intelligent app for recognizing voice through the microphone 111. [ For example, the user terminal 100 displays the UI 121 of the intelligent app on the display 120 when a voice (e.g., wake up!) Designated via the microphone 111 is input 113a can do.

4 is a block diagram illustrating an intelligent server of an integrated intelligence system according to an embodiment of the present invention.

4, the intelligent server 200 includes an automatic speech recognition (ASR) module 210, a natural language understanding (NLU) module 220, a path planner module 220, A dialogue manager (DM) module 240, a natural language generator (NLG) module 250 or a text to speech (TTS) module 260 can do.

The natural language understanding module 220 or the path planner module 230 of the intelligent server 200 may generate a path rule.

According to one embodiment, an automatic speech recognition (ASR) module 210 may convert user input received from the user terminal 100 into text data.

According to one embodiment, the automatic speech recognition module 210 may convert user input received from the user terminal 100 into text data. For example, the automatic speech recognition module 210 may include a speech recognition module. The speech recognition module may include an acoustic model and a language model. For example, the acoustic model may include information related to speech, and the language model may include information on a combination of unit phoneme information and unit phoneme information. The speech recognition module may convert user speech into text data using information on vocalization and information on unit phonemic information. The information about the acoustic model and the language model may be stored in, for example, an automatic speech recognition database (ASR DB)

According to one embodiment, the natural language understanding module 220 can perform a syntactic analysis or a semantic analysis to grasp a user's intention. The grammatical analysis can divide the user input into grammatical units (e.g., words, phrases, morphemes, etc.) and determine what grammatical elements the divided units have. The semantic analysis can be performed using semantic matching, rule matching, formula matching, or the like. Accordingly, the natural language understanding module 220 may obtain a domain (domain), an intent, or a parameter (or a slot) necessary for expressing the intention.

According to one embodiment, the natural language understanding module 220 uses a matching rule that is divided into a domain, an intend, and a parameter (or slot) Lt; RTI ID = 0.0 > and / or < / RTI > For example, the one domain (e.g., an alarm) may include a plurality of intents (e.g., alarm settings, alarm cancellation, etc.), and one intention may include a plurality of parameters Etc.). The plurality of rules may include, for example, one or more mandatory element parameters. The matching rule may be stored in a natural language understanding database (NLU DB)

According to one embodiment, the natural language understanding module 220 grasps the meaning of words extracted from user input using linguistic features (e.g., grammatical elements) such as morphemes, phrases, and the like, And the intention of the user. For example, the natural language understanding module 220 can determine the user intention by calculating how many words extracted from user input are included in each domain and intention. According to one embodiment, the natural language understanding module 220 may determine the parameters of the user input using words that are the basis for grasping the intent. According to one embodiment, the natural language understanding module 220 can determine the intention of the user using the natural language recognition database 221 in which the linguistic characteristic for grasping the intention of the user input is stored. According to another embodiment, the natural language understanding module 220 can determine a user's intention using a personalized language model (PLM). For example, the natural language understanding module 220 may determine a user's intention using personalized information (e.g., a contact list, a music list). The personalized language model may be stored in the natural language recognition database 221, for example. According to one embodiment, not only the natural language understanding module 220 but also the automatic speech recognition module 210 can recognize the user's voice by referring to the personalized language model stored in the natural language recognition database 221.

According to one embodiment, the natural language understanding module 220 may generate a pass rule based on the intent and parameters of the user input. For example, the natural language understanding module 220 may select an app to be executed based on an intention of a user input, and determine an operation to be performed in the selected app. The box salmon understanding module 220 may determine a parameter corresponding to the determined operation to generate a pass rule. According to one embodiment, the pass rules generated by the natural language understanding module 220 may include information about the app to be executed, the action to be performed in the app, and the parameters needed to perform the action.

According to one embodiment, the natural language understanding module 220 may generate one pass rule, or a plurality of pass rules, based on the intent and parameters of the user input. For example, the natural language understanding module 220 receives the path rule set corresponding to the user terminal 100 from the path planner module 230, maps the intention and parameters of the user input to the received path rule set, The rules can be determined.

According to another embodiment, the natural language understanding module 220 determines an app to be executed, an action to be performed in the app, and parameters necessary to perform the action based on the intention and parameters of the user input, You can create a rule. For example, the natural language understanding module 220 may use the information of the user terminal 100 to describe an operation to be executed in the app to be executed and an application in an ontology or a graph model You can create path rules by arranging them. The generated path rule may be stored in a path rule database (PR DB) 231 via the path planner module 230, for example. The generated path rule may be added to the path rule set of the database 231. [

According to one embodiment, the natural language understanding module 220 may select at least one of the generated plurality of pass rules. For example, the natural language understanding module 220 may select an optimal path rule for the plurality of pass rules. As another example, the natural language understanding module 220 may select a plurality of pass rules if only some actions are specified based on user utterances. The natural language understanding module 220 may determine one of the plurality of pass rules by a user's further input.

According to one embodiment, the natural language understanding module 220 may send a pass rule to the user terminal 100 in response to a user input. For example, the natural language understanding module 220 may transmit one pass rule corresponding to user input to the user terminal 100. In another example, the natural language understanding module 220 may transmit a plurality of pass rules corresponding to user input to the user terminal 100. The plurality of pass rules may be generated by the natural language understanding module 220 if, for example, only some actions are specified based on user utterances.

According to one embodiment, the path planner module 230 may select at least one of the plurality of path rules.

According to one embodiment, the path planner module 230 may deliver a set of path rules that includes a plurality of pass rules to the natural language understanding module 220. The plurality of path rules of the path rule set may be stored in a form of a table in the path rule database 231 connected to the path planner module 230. For example, the path planner module 230 transmits a path rule set corresponding to information (e.g., OS information, app information) of the user terminal 100 received from the intelligent agent 151 to the natural language understanding module 220 . The table stored in the path rule database 231 may be stored, for example, for each domain or domain version.

According to one embodiment, the path planner module 230 may select one pass rule or a plurality of pass rules from a pass rule set and transmit the selected pass rules to the natural language understanding module 220. For example, the path planner module 230 selects one pass rule or a plurality of pass rules by matching a user's intention and parameters with the path rule set corresponding to the user terminal 100, .

According to one embodiment, the path planner module 230 may generate one pass rule or a plurality of pass rules using the user intent and parameters. For example, the path planner module 230 may generate one pass rule or a plurality of pass rules by determining an app to be executed and an action to be executed in the app based on the user's intention and parameters. According to one embodiment, the path planner module 230 may store the generated path rule in the path rule database 231. [

According to one embodiment, the path planner module 230 may store the pass rules generated in the natural language understanding module 220 in the pass rule database 231. [ The generated path rule may be added to the path rule set stored in the path rule database 231. [

According to one embodiment, the table stored in the path rule database 231 may include a plurality of path rules or a plurality of path rule sets. The plurality of path rules or the plurality of path rule sets may reflect the type, version, type, or characteristic of the device that executes each pass rule.

According to one embodiment, the conversation manager module 240 may determine whether the intention of the user identified by the natural language understanding module 220 is clear. For example, the conversation manager module 240 may determine whether the intention of the user is clear based on whether or not the information of the parameter is sufficient. The conversation manager module 240 may determine whether the parameters identified in the natural language understanding module 220 are sufficient to perform the task. According to one embodiment, the conversation manager module 240 may perform feedback requesting the user for the necessary information if the intention of the user is unclear. For example, the conversation manager module 240 may perform feedback requesting information about a parameter for grasping a user's intention.

According to one embodiment, the conversation manager module 240 may include a content provider module. If the content providing module can perform an operation based on the intention and parameters recognized in the natural language understanding module 220, the content providing module may generate a result of performing a task corresponding to a user input. According to one embodiment, the conversation manager module 240 may send the results generated in the content provision module to the user terminal 100 in response to user input.

According to one embodiment, the natural language generation module (NLG) 250 may change the specified information into a text form. The information changed in the text form may be in the form of a natural language utterance. The specified information may be, for example, information about additional inputs, information to guide completion of an operation corresponding to the user input, or information (e.g., feedback information on user input) that directs additional input of the user. The information changed in the text form may be transmitted to the user terminal 100 and displayed on the display 120 or transmitted to the text-to-speech conversion module 260 to be changed to a voice form.

According to one embodiment, the text-to-speech module 260 may change textual information to speech-like information. The text-to-speech conversion module 260 receives the information of the text form from the natural language generation module 250, converts the information of the text form into the information of the voice form, and transmits the information to the user terminal 100. The user terminal 100 may output the voice information to the speaker 130.

According to one embodiment, the natural language understanding module 220, the path planner module 230, and the conversation manager module 240 may be implemented as a single module. For example, the natural language understanding module 220, the path planner module 230, and the conversation manager module 240 may be implemented as a module to determine a user's intention and parameters, You can create a response (eg, a path rule). Accordingly, the generated response can be transmitted to the user terminal 100.

5 is a diagram illustrating a method of generating a path rule of a path planner module according to an embodiment of the present invention.

Referring to FIG. 5, the natural language understanding module 220 may divide the functions of the application into unit actions (A to F) and store them in the pass rule database 231 according to an exemplary embodiment. For example, the natural language understanding module 220 generates a plurality of path rules (A-B1-C1, A-B1-C2, A-B1-C3-DF, A-B1-C3-DEF) And store the set of included path rules in the path rule database 231.

According to one embodiment, the path rule database 231 of the path planner module 230 may store a set of path rules for performing the function of the app. The path rule set may include a plurality of pass rules including a plurality of actions. The plurality of path rules may be sequentially arranged in accordance with the parameters inputted to each of the plurality of operations. According to one embodiment, the plurality of pass rules may be configured in an ontology or graph model form and stored in the pass rule database 231. [

According to one embodiment, the natural language understanding module 220 includes a plurality of pass rules A-B1-C1, A-B1-C2, A-B1-C3-DF, (A-B1-C3-DF) can be selected among the optimum path rules (B1-C3-DEF).

According to one embodiment, the natural language understanding module 220 may pass a plurality of rules to the user terminal 100 in the absence of a path rule that perfectly matches the user input. For example, the natural language understanding module 220 may select a pass rule (e.g., A-B1) that is partially corresponding to the user input. The natural language understanding module 220 includes one or more pass rules (e.g., A-B1-C1, A-B1-C2, A-B1-C3 -DF, A-B1-C3-DEF) to the user terminal 100.

According to one embodiment, the natural language understanding module 220 may select one of a plurality of pass rules based on the further input of the user terminal 100 and forward the selected one pass rule to the user terminal 100 have. For example, the natural language understanding module 220 may generate a plurality of pass rules (e.g., A-B1-C1, A-B1, etc.) according to a user input (e.g., (For example, A-B1-C3-DF) among the A-B1-C3-DF and the A-B1-C3-DEF.

According to yet another embodiment, the natural language understanding module 220 may determine a user's intention and parameters corresponding to a user input (e.g., an input to select C3) that is further input to the user terminal 100, And transmit the intention or parameter of the user to the user terminal 100. A-B1-C3-DF, A-B1-C3-DF, and A-B1-C3-DF based on the transmitted intention or the parameter. DEF) (for example, A-B1-C3-DF).

Accordingly, the user terminal 100 can complete the operation of the applications 141 and 143 by the selected one pass rule.

According to one embodiment, the natural language understanding module 220 may generate a pass rule that partially corresponds to the received user input, when the intelligent server 200 receives user input that lacks information. For example, the natural language understanding module 220 may transmit the partially corresponding path rule to the intelligent agent 151 (①). The intelligent agent 151 transmits the partially corresponding path rule to the execution manager module 153 (②), and the execution manager module 153 can execute the first app 141 according to the path rule . The execution manager module 153 can transmit the information on the insufficient parameters to the intelligent agent 151 while executing the first application 141 (3). The intelligent agent 151 may request additional input to the user using the information on the insufficient parameter. The intelligent agent 151 can transmit and process the additional input to the intelligent server 200 when the additional input is received (4) by the user. The natural language understanding module 220 may generate an added path rule based on the intention and parameter information of the further inputted user input and transmit it to the intelligent agent 151 (5). The intelligent agent 151 can execute the second application 143 by sending the pass rule to the execution manager module 153 (6).

According to one embodiment, the natural language understanding module 220 may send a user information request to the personalization information server 300 when user input with some information missing is received by the intelligent server 200. The personalization information server 300 may transmit the information of the user who inputs the user input stored in the persona database to the natural language understanding module 220. [ The natural language understanding module 220 may use the user information to select a path rule corresponding to a user input that is missing some operation. Accordingly, the natural language understanding module 220 may request the missing information to receive the additional input even when the user input with some information is missing, even if it is received by the intelligent server 200, Can be determined.

Figure 6 is a diagram illustrating the context module of the processor collecting current status according to an embodiment of the present invention.

6, when the context module 155a receives a context request from the intelligent agent 151 (①), the context module 155a requests context information indicating the current state of the applications 141 and 143 to the applications 141 and 143 ②) can be done. According to one embodiment, the context module 155a may receive (3) the context information from the applications 141 and 143 and transmit (4) to the intelligent agent 151.

According to one embodiment, the context module 155a may receive a plurality of context information via the apps 141 and 143. [ For example, the context information may be information on the most recently executed applications 141 and 143. In another example, the context information may be information about the current status in the apps 141 and 143 (e.g., information about the corresponding pictures if viewing the pictures in the gallery).

According to one embodiment, the context module 155a may receive context information indicating the current state of the user terminal 100 from the device platform as well as the apps 141 and 143. [ The context information may include general context information, user context information, or device context information.

The general context information may include general information of the user terminal 100. The general context information may be received through a sensor hub or the like of the device platform and confirmed through an internal algorithm. For example, the general context information may include current time and space information. The information on the current time and space may include information on the current time or the current position of the user terminal 100, for example. The current time can be confirmed through time on the user terminal 100, and information on the current position can be confirmed through a global positioning system (GPS). In another example, the general context information may include information about physical motion. The information on the physical movement may include information on, for example, walking, running, driving, and the like. The physical motion information may be identified through a motion sensor. The information on the driving operation can be confirmed not only through the motion sensor but also through the Bluetooth connection in the vehicle to confirm boarding and parking. In another example, the general context information may include user activity information. The user activity information may include information on commute, shopping, trip, and the like, for example. The user activity information can be confirmed using information about a place where a user or an application registers in the database.

The user context information may include information about a user. For example, the user context information may include information on a user's emotional state. The information on the emotional state may include, for example, information about a user's happiness, sadness, anger, etc. [ As another example, the user context information may include information on the current state of the user. The information on the current state may include information on, for example, interest, intention, etc. (e.g., shopping).

The device context information may include information on the status of the user terminal 100. [ For example, the device context information may include information about a path rule executed by the execution manager module 153. [ In another example, the device information may include information about the battery. The information on the battery can be confirmed through, for example, the state of charge and discharge of the battery. In another example, the device information may include information about connected devices and networks. Information about the connected device can be ascertained, for example, via a communication interface to which the device is connected.

7 is a diagram illustrating that a persona module according to an embodiment of the present invention manages user information.

Referring to FIG. 7, the persona module 155b may receive information of the user terminal 100 from the applications 141 and 143, the execution manager module 153, or the context module 155a. The apps 141 and 143 and the execution manager module 153 can store information on the result of executing the app operations 141b and 143b in the action log database. The context module 155a may store information on the current state of the user terminal 100 in the context database. The persona module 155b may receive the stored information from the action log database or the context database. The data stored in the action log database and the context database may be analyzed by, for example, an analysis engine and transmitted to the persona module 155b.

According to one embodiment, the persona module 155b may send information received from the applications 141 and 143, the execution manager module 153, or the context module 155a to the proposal module 155c. For example, the persona module 155b may transmit data stored in the action log database or the context database to the proposal module 155c.

According to one embodiment, the persona module 155b may send information received from the apps 141 and 143, the execution manager module 153, or the context module 155a to the personalization information server 300. [ For example, the persona module 155b may periodically transmit data stored in the action log database or the context database to the personalization information server 300.

According to one embodiment, the persona module 155b may transmit data stored in the action log database or the context database to the suggestion module 155c. The user information generated by the persona module 155b may be stored in a persona database. The persona module 155b may periodically transmit the user information stored in the persona database to the personalization information server 300. [ According to one embodiment, information sent by the persona module 155b to the personalization information server 300 may be stored in a persona database. The personalization information server 300 can deduce user information necessary for generating a pass rule of the intelligent server 200 using the information stored in the persona database.

According to one embodiment, the inferred user information using the information transmitted by the persona module 155b may include profile information or preference information. The profile information or the preference information may be deduced from the user's account and accumulated information.

The profile information may include personal information of a user. For example, the profile information may include demographic information of a user. The demographic information may include, for example, a user's gender, age, and the like. In another example, the profile information may include life event information. The life event information may be inferred by comparing log information with a life event model, for example, and may be reinforced by analyzing a behavior patter. In another example, the profile information may include interest information. The interest information may include, for example, a shopping item of interest, an area of interest (e.g., sports, politics, etc.). As another example, the profile information may include activity area information. The activity area information may include information on, for example, a house, a work place, and the like. The information on the active area may include information on the place where the priority is recorded based on the cumulative stay time and the number of visits as well as information on the location of the place. As another example, the profile information may include activity time information. The activity time information may include information on, for example, a weather time, a commute time, a sleep time, and the like. The information on the commute time can be inferred using the activity area information (e.g., information on the house and work place). Information about the sleeping time can be inferred through the unused time of the user terminal 100. [

The preference information may include user preference information. For example, the preference information may include information about an app preference. The app preference can be deduced, for example, through the usage history of the app (e.g., time and place usage records). The preference of the app may be used to determine an app to be executed according to a user's current state (e.g., time, place). In another example, the preference information may include information about contact preferences. The contact preference can be inferred by analyzing information on the frequency of contact (e.g., frequency of contact by hour or place) of a contact, for example. The contact preference can be used to determine the contact to contact according to the current state of the user (e.g., contact for duplicate names). In another example, the preference information may include setting information. The setting information may be inferred by analyzing information on a setting frequency of a specific setting value (for example, a frequency of setting the setting value by time or place). The setting information may be used to set a specific setting value according to a user's current state (e.g., time, place, situation). As another example, the preference information may include location preference. The place preference can be deduced, for example, through a visit record (for example, an hourly visit record) of a specific place. The location preference can be used to determine a visited location according to the user's current state (e.g., time). As another example, the preference information may include a command preference. The command preference may be inferred through, for example, the frequency of command usage (e.g., time, frequency of use per location). The command preference may be used to determine a command pattern to be used according to the user's current state (e.g., time, location). In particular, the command preference may include information about the menu most frequently selected by the user in the current state of the app being executed by analyzing the log information.

8 is a flowchart illustrating a method of performing a task based on a user utterance request according to one embodiment.

Hereinafter, it is assumed that the user terminal (or electronic device) 100 of FIG. 2 performs the process of FIG. In addition, in the description of FIG. 8, it is understood that the operations described as being performed by the electronic device are controlled by the processor 150 of the electronic device 100. (Or instructions) that may be executed (or executed) by the processor 150 of the electronic device 100. [ The instructions may be stored, for example, in a computer recording medium or in the memory 140 of the electronic device 100 shown in FIG.

Referring to FIG. 8, at operation 801, the processor 150 may receive a user utterance through the microphone 111.

According to one embodiment, the user utterance may include a request to perform the task with electronic device 100. For example, a user utterance may be 'send a Hawaiian picture to a message app to a mother', and the user utterance includes a request to perform a task of sending a picture taken in Hawaii to a mother using a message app.

At operation 802, the processor 150 may transmit data related to user utterance to an external server via a wireless communication circuit.

According to one embodiment, the data associated with a user utterance may include a voice signal corresponding to a user utterance.

According to one embodiment, the processor 150 may transmit context information indicating the current state of the electronic device to an external server via a wireless communication circuit. The context information may include, for example, running application information or state information of an executing application. In one embodiment, the state information of an app may include a state ID, specific screen information of an app, or specific status information in a specific screen

At operation 803, the processor 150 may receive a response from an external server via a wireless communication circuit.

According to one embodiment, the received response includes at least one sample utterance indicative of intent of user utterance, and at least one sample utterance is selected by an external server (e.g., intelligent server 200) based on user utterance Lt; / RTI >

According to one embodiment, the at least one sample utterance may comprise a first sample utterance representing a first intent of user utterance and a second sample utterance representing a second intent other than the first intent. For example, the received response may include a plurality of sample utterances, and each sample utterance may indicate the intent of different user utterances.

In one embodiment, the external server may generate at least one pass rule based on data associated with the user utterance. According to one embodiment, the at least one sample speech may comprise a first sample speech corresponding to a first pass rule and a second sample speech corresponding to a second pass rule. For example, the received response may include a plurality of sample utterances, and each sample utterance may correspond to a different path rule.

At operation 804, the processor 150 may display at least one sample speech on the touch screen display 120. The manner in which the processor 150 displays the sample utterance will be described in detail below.

At operation 805, the processor 150 may receive a first user input for selecting one of the at least one sample utterance via the microphone 111 or the touch screen display 120. The manner in which the processor 150 receives the first user input will be described in detail below.

At operation 806, the processor 150 may perform a task associated with the selected sample utterance.

According to one embodiment, the processor 150 may perform the task by causing the electronic device to have a sequence of states associated with the selected one of the at least one sample utterance.

In one embodiment, after operation 802, if the external server, which may correspond to intelligent server 200, explicitly determines the intent of the user utterance, then the electronic device may perform the task according to the intent of the determined user utterance. However, since the type of user utterance is not limited, an external server may not be able to clearly determine the intention of user utterance.

Hereinafter, a method for an external server to generate a pass rule based on data related to user utterance will be described when the external server can not clearly determine the intention of user utterance.

9 is a block diagram illustrating a natural language understanding module of the intelligent server 200 according to an embodiment.

9, the natural language understanding module 910 may include a domain classifier 911, an intent classifier 912, a slot tagger 913, a criteria handler 914, and the like . According to one embodiment, the natural language understanding module of FIG. 9 may correspond to the natural language understanding module 220 of FIG.

The domain classifier 911 can determine a domain to which the path rule belongs based on at least one of text or context information corresponding to a user utterance. According to one embodiment, the domain may correspond to an app (app) used when the pass rule is executed.

According to one embodiment, the domain classifier 911 may use text corresponding to the user utterance received from the automatic speech recognition module 210. [ According to one embodiment, the domain classifier 911 may use the text directly entered by the user.

According to one embodiment, the context information may include app information (or application information running in the background) that was executed before the user utterance was input, state information of the app, and the like. In one embodiment, the state information of an app may include a state ID, specific screen information of an app, or specific status information in a specific screen

According to one embodiment, the domain classifier 911 may determine a plurality of domains. The domain classifier 911 can further determine a confidence level corresponding to each domain. According to one embodiment, the sum of the plurality of confidence values may be one.

The intention classifier 912 may determine the intent in the domain determined by the domain classifier 911 based on at least one of the text or context information corresponding to the user utterance. According to an embodiment, the intention may correspond to an operation performed using an application corresponding to a domain, and the intention classifier 912 may determine a path rule ID corresponding to the intention.

According to one embodiment, the context information may include app information (or application information running in the background) that was executed before the user utterance was input, state information of the app, and the like.

According to one embodiment, when the domain classifier 911 determines a plurality of domains, the intent classifier 912 may determine intent in each of the plurality of domains.

According to one embodiment, the intent classifier 912 can determine a plurality of intentions. Intention classifier 912 may further determine a confidence value corresponding to each intent. According to one embodiment, the sum of the plurality of confidence values may be one.

Slot tagger 913 may generate a slot necessary for the execution of a path rule based on the intention (or path rule ID) determined by the text and intent classifier 912 corresponding to the user utterance. According to one embodiment, if intent classifier 912 has determined a plurality of intentions, slottener 913 may generate slots for a plurality of intentions.

According to one embodiment, the slot tagger 913 may generate a plurality of slots for each intent. The slot tagger 913 can further determine the reliability value corresponding to each slot.

The format converter 914 can change the parameters generated by the slot tagger 913 into a format that can be used by the path planner module. For example, the format converter 914 may convert a slot called " yesterday " into a data format having a date of " -1 "

According to one embodiment, the natural language understanding module may comprise modules such as a domain classifier 911, an intent classifier 912, a slot tagger 913, and a format converter 914, .

The natural language understanding module can pass the path rule ID and the slot to the path planner module. The path planner module can change the slot to a parameter of the path rule, generate the path rule using the context information of the electronic device, and transmit it to the electronic device.

10 illustrates an operation that a pass rule according to an embodiment includes.

In one embodiment, the pass rules generated by the intelligent server 200 using user utterances may include operations 1 through 5 as shown in FIG.

For example, Actions 1 through 5 may be actions based on a pass rule generated based on a user utterance of " Find a Hawaii photo ", and Actions 1 through 5 use a gallery app to find pictures taken in Hawaii This can be a task.

According to one embodiment, the intelligent server 200 may be configured such that when the electronic device 100 displays a home screen, the electronic device 100, whether the user's utterance is received, And if the user utterance has been received when the action 2 has already been performed, a gallery rule may be used to generate a pass rule for finding a picture taken in Hawaii.

In one embodiment, the natural language understanding module of the intelligent server 200 may further use context information as well as text corresponding to a user utterance. According to one embodiment, the intelligent server 200 may be configured such that when a user utterance is received when the electronic device 100 displays a home screen and when the electronic device 100 has already received operations 1 and 2 The path rule in the case where the user utterance has been received may be generated.

For example, when a user utterance of " Find Hawaii " is received when the electronic device 100 displays a home screen, the intelligent server 200 may generate a pass rule for searching Hawaii in the Internet app.

As another example, if a user utterance " Find Hawaii " is received when the electronic device 100 displays the gallery app screen, the intelligent server 200 may generate a pass rule for finding a Hawaii photo in the gallery app. At this time, although the intention of the user was to search Hawaii on the Internet, it may be that a pass rule is generated in which a user utterance is received while the gallery application is executed and a Hawaii photograph is searched.

The intelligent server 200 may generate and send a sample speech corresponding to a plurality of pass rules and pass rules to the electronic device 100 in a manner to be described below to generate a pass rule conforming to the user's intention. Before describing a specific method, a screen displayed on the electronic device 100 by a plurality of sample utterances generated by the intelligent server 200 will be described with reference to Figs. 11A and 11B.

FIG. 11A is a diagram showing that a user utterance is input when a gallery app according to an embodiment is executed. FIG.

11B is a diagram illustrating a screen in which a plurality of sample utterances are displayed according to an embodiment.

In one embodiment, the electronic device 100 may receive and send to the intelligent server 200 a user utterance 1101 of " Find Hawaii " on the screen where the gallery app is launched, as shown in FIG. 11A.

The intelligent server 200 uses the text corresponding to the user's utterance "Find Hawaii" and the executed application using the context information called the gallery app, and using the gallery application, a pass rule for searching for a picture of Hawaii, A path rule can be generated to search for a path rule. The intelligent server 200 can generate a sample speech corresponding to each pass rule and transmit it to the electronic device 100. [

In one embodiment, the electronic device 100 displays a sample speech 1102 corresponding to the received path rule as shown in FIG. 11B, and executes a pass rule corresponding to the selected sample speech by receiving an input for selecting a sample speech .

Hereinafter, with reference to FIG. 12 and FIG. 13, a specific method for the intelligent server 200 to transmit a plurality of sample utterances will be described.

12 is a flowchart illustrating a method of transmitting a plurality of sample utterances by the intelligent server 200 according to one embodiment.

Referring to FIG. 12, in operation 1201, the domain classifier 911 may determine at least one domain (or an application) based on at least one of text or context information corresponding to a user utterance. According to one embodiment, the domain classifier 911 may further determine a confidence value corresponding to each of the at least one domain.

In operation 1202, the intelligent server 200 may determine whether the reliability value of the domain having the largest reliability value exceeds a specified value.

If the reliability value of the domain having the highest reliability value as a result of the operation 1202 exceeds the designated value, in operation 1203, the intelligent server 200 sets the domain corresponding to the user's utterance and context information to the domain having the highest reliability value You can decide.

If the reliability value of the domain having the highest reliability value as a result of the operation 1202 does not exceed the designated value, at operation 1204, the intelligent server 200 uses the context information to determine whether the user's utterance is root utterance It can be judged. In one embodiment, the root utterance may mean that the intelligent service is executed and the received user utterance when the electronic device 100 is displaying the home screen. According to one embodiment, the context information may include app information that was executed before the intelligent service was executed, and the intelligent server 200 may use this context information to determine whether the user utterance is root utterance.

If the user utterance is not the root utterance as a result of operation 1202, then at operation 1205, the domain classifier 911 may determine at least one domain (or app) based on the user utterance, assuming that the user utterance is root utterance have. According to one embodiment, the domain classifier 911 may further determine a confidence value corresponding to each of the at least one domain.

At operation 1206, the Intent Classifier 912 determines at least one intention in the first domain having the highest confidence value among at least one domain determined in operation 1201, based on at least one of text or context information corresponding to a user utterance May determine at least one intention in the second domain with the highest confidence value among the at least one domain determined in operation 1205. [ According to one embodiment, the intent classifier 912 may determine the path rule ID corresponding to the intent. According to one embodiment, the intent classifier 912 may further determine a confidence value corresponding to each of the at least one intent.

In operation 1207, the intelligent server 200 generates a path rule corresponding to each intention using a predetermined number of intentions and user utterances having a high reliability value among at least one intention (or path rule ID) in the first domain can do. In addition, the intelligent server 200 may generate a path rule corresponding to each intention using a predetermined number of intentions and a user utterance having a high reliability value among at least one intent in the second domain. According to one embodiment, the intelligent server 200 may further generate a sample speech corresponding to a pass rule.

For example, the intelligent server 200 generates two path rules corresponding to each intention using the intention corresponding to the upper two intention and the user utterance in the descending order of the confidence value among the five intentions in the first domain . Also, the intelligent server 200 can generate two path rules corresponding to each intention using the intention corresponding to the top two and the user utterance in the descending order of the reliability value among the four intentions in the second domain . In other words, the intelligent server 200 can generate a total of four pass rules.

In one embodiment, the intelligent server 200 may store a representative sentence (RU) representing the intent of each pass rule. According to one embodiment, the intelligent server 200 may further store representative sentences (RFUs, representative friendly utterances) that are easily understood by the user. The intelligent server 200 can generate a sample speech corresponding to the pass rule using the RFU.

For example, the intelligent server 200 may use the RFU to allow the user to "send a picture taken in Hawaii to Emma via the message," using text corresponding to the utterance, "a picture taken in Hawaii, Send a message ".

According to one embodiment, the intelligent server 200 may generate a sample utterance by changing parameters in the RFU to parameters generated from text corresponding to a user utterance.

For example, the RFU stored in the intelligent server 200 may be a "picture taken in Hawaii, a message to Emma", and a text corresponding to a user utterance "send a picture to Tommy through a message" taken in Tokyo. The intelligent server 200 can change the 'Hawaii' in the RFU to 'Tokyo' and the 'Emma' to 'Tommy' to generate a sample speech 'Send a message to Tommy, a picture taken in Tokyo'.

In one embodiment, if the user utterance as a result of operation 1202 is a root utterance, then at operation 1208, the intent classifier 912 may generate at least one of the at least one Lt; / RTI > of a given number of domains with a high confidence value among the domains of the plurality of domains. According to one embodiment, the intent classifier 912 may determine the path rule ID corresponding to the intent. According to one embodiment, the intent classifier 912 may further determine a confidence value corresponding to each of the at least one intent.

For example, the Intent Classifier 912 may determine at least one intention in the first domain with the highest confidence value among the at least one domain determined in operation 1201, and the confidence value among at least one domain determined in operation 1201 It is possible to determine at least one intention in the second largest third domain.

In another example, the Intent Classifier 912 may determine at least one intention in only the first domain with the highest confidence value among the at least one domain determined in operation 1201. [

In operation 1207, after an operation 1208, if the user utterance is root utterance as a result of operation 1202, the intelligent server 200 determines the reliability (or pass rule ID) of at least one intention A path rule corresponding to each intention can be generated using a specified number of intention values and a user utterance. In addition, the intelligent server 200 can generate a path rule corresponding to each intention using a predetermined number of intentions and user utterances having a higher confidence value among at least one intent in the third domain. According to one embodiment, the intelligent server 200 may further generate a sample speech corresponding to a pass rule.

For example, the intelligent server 200 generates two path rules corresponding to each intention using the intention corresponding to the upper two intention and the user utterance in the descending order of the confidence value among the five intentions in the first domain . In addition, the intelligent server 200 can generate two path rules corresponding to each intention using the intention corresponding to the top two and the user utterance in the descending order of the reliability values among the three intentions in the third domain . In other words, the intelligent server 200 can generate a total of four pass rules.

As another example, the intelligent server 200 can generate four path rules corresponding to each intention using intention and user utterance corresponding to the top four in order of increasing confidence value among the five intentions in the first domain have.

At operation 1209, the intelligent server 200 may send a sample speech corresponding to the generated pass rule and pass rule to the electronic device 100.

13 is a flowchart illustrating a method of transmitting a plurality of sample utterances by the intelligent server 200 according to one embodiment.

Referring to FIG. 13, in operation 1301, the domain classifier 911 may determine at least one domain (or an application) based on at least one of text or context information corresponding to a user utterance. According to one embodiment, the domain classifier 911 may further determine a confidence value corresponding to each of the at least one domain.

At operation 1302, the Intent Classifier 912 determines at least one intention in the first domain with the highest confidence value among at least one domain determined in operation 1201, based on at least one of text or context information corresponding to the user utterance You can decide. According to one embodiment, the intent classifier 912 may determine the path rule ID corresponding to the intent. According to one embodiment, the intent classifier 912 may further determine a confidence value corresponding to each of the at least one intent.

At operation 1303, the intelligent server 200 may determine if the confidence value of the intent with the greatest confidence value exceeds a specified value.

If the confidence value of the intent with the highest confidence value as a result of operation 1303 exceeds a specified value, then at operation 1304, the intelligent server 200 will intention to the user utterance and context information with the intention of having the highest confidence value You can decide.

If the confidence value of the intent with the highest confidence value as a result of operation 1303 does not exceed the specified value, then at operation 1305, the intelligent server 200 may use the context information to determine whether the user utterance is root utterance have. According to one embodiment, the context information may include app information that was executed before the intelligent service was executed, and the intelligent server 200 may use this context information to determine whether the user utterance is root utterance.

If, as a result of operation 1303, the user utterance is not a root utterance, then at operation 1306, the domain classifier 911 determines whether the user utterance is a user utterance in the specified state, Can be determined. According to one embodiment, the domain classifier 911 may further determine a confidence value corresponding to each of the at least one domain.

In one embodiment, the designated state is a state in which the intelligent service is executed when the electronic device 100 displays the home screen, a state in which the intelligent service is executed when the screen immediately before the screen of the electronic device 100 is displayed, Or a state in which the intelligent service is executed when the designated representative screen of the application in which the electronic device 100 is being executed is displayed.

At operation 1307, the Intent Classifier 912 determines at least one intention in the second domain with the highest confidence value among at least one domain determined in operation 1306, based on at least one of text or context information corresponding to the user utterance You can decide. According to one embodiment, the intent classifier 912 may determine the path rule ID corresponding to the intent. According to one embodiment, the intent classifier 912 may further determine a confidence value corresponding to each of the at least one intent.

In one embodiment, if at least one domain determined in operation 1306 is the same as at least one domain determined in operation 1301, then the intent classifier 912 uses context information corresponding to the specified state to determine at least one Can be determined. For example, when the screen of the electronic device 100 is displayed before the screen of the electronic device 100 is displayed, assuming that the user's utterance is received in the state where the intelligent service is executed, at least one Can be determined.

In one embodiment, if the user utterance is not a root utterance as a result of operation 1303, the intelligent server 200 does not perform operation 1306, and in operation 1307, the intent classifier 912 receives context information May be used to determine at least one intention in the second domain.

In operation 1308, the intelligent server 200 generates a path rule corresponding to each intention using a predetermined number of intentions and user utterances having a high reliability value among at least one intention (or path rule ID) in the first domain can do. In addition, the intelligent server 200 may generate a path rule corresponding to each intention using a predetermined number of intentions and a user utterance having a high reliability value among at least one intent in the second domain. According to one embodiment, the intelligent server 200 may further generate a sample speech corresponding to a pass rule.

For example, the intelligent server 200 generates two path rules corresponding to each intention using the intention corresponding to the upper two intention and the user utterance in the descending order of the confidence value among the five intentions in the first domain . Also, the intelligent server 200 can generate two path rules corresponding to each intention using the intention corresponding to the top two and the user utterance in the descending order of the reliability value among the four intentions in the second domain . In other words, the intelligent server 200 can generate a total of four pass rules.

In one embodiment, if the user utterance is the root utterance as a result of operation 1303, then at operation 1309, the intent classifier 912 generates at least one of the at least one Lt; / RTI > of a given number of domains with a high confidence value among the domains of the plurality of domains. According to one embodiment, the intent classifier 912 may determine the path rule ID corresponding to the intent. According to one embodiment, the intent classifier 912 may further determine a confidence value corresponding to each of the at least one intent.

For example, since the intent classifier 912 has determined at least one intent in the first domain in operation 1302, the trust value of at least one domain determined in operation 1301 is at least one The intention can be further determined.

In operation 1308, after operation 1309, if the user utterance is the root utterance as a result of operation 1303, the intelligent server 200 determines the reliability (or pass rule ID) of at least one intention A path rule corresponding to each intention can be generated using a specified number of intention values and a user utterance. In addition, the intelligent server 200 can generate a path rule corresponding to each intention using a predetermined number of intentions and user utterances having a higher confidence value among at least one intent in the third domain. According to one embodiment, the intelligent server 200 may further generate a sample speech corresponding to a pass rule.

For example, the intelligent server 200 generates two path rules corresponding to each intention using the intention corresponding to the upper two intention and the user utterance in the descending order of the confidence value among the five intentions in the first domain . In addition, the intelligent server 200 can generate two path rules corresponding to each intention using the intention corresponding to the top two and the user utterance in the descending order of the reliability values among the three intentions in the third domain . In other words, the intelligent server 200 can generate a total of four pass rules.

At operation 1310, the intelligent server 200 may send a sample speech corresponding to the generated pass rule and pass rule to the electronic device 100. [

Operations 1201 to 1209 and operations 1301 to 1310 have been described on the assumption that the domain classifier 911 and the intent classifier 912 of the intelligent server 200 are separately configured. According to various embodiments, the intelligent server 200 may collectively configure the domain classifier 911 and the intent classifier 912 to determine the domain and intent using at least one of the textual or contextual information corresponding to the user utterance have. Even in accordance with various embodiments, the intelligent server 200 can determine a confidence value corresponding to a domain and an intention, and can generate a plurality of pass rules.

According to various embodiments, the intelligent server 200 may generate a slot based on the intent determined by the text and intent classifier 912 corresponding to the user utterance, and may determine a confidence value corresponding to the slot. The intelligent server 200 may transmit data to the electronic device 100 to modify the parameter corresponding to the slot having the reliability value of the specified value or less. The operation for the electronic device 100 to modify the parameters will be described below.

According to various embodiments, the intelligent server 200 may generate a plurality of pass rules corresponding to a plurality of domains when the intent determined from the user utterance may correspond to a plurality of domains and one pass rule can not be determined To the electronic device (100). The intelligent server 200 specifies a separate path rule ID (e.g., a conflict path rule ID) that causes a plurality of path rules, and determines the path rule ID as a separate path rule ID in the above case have.

For example, when the user utterance is " send to the mother late ", the operation of sending the statement " late " to the mother whose intention is determined may be performed using an e-mail app, a message app, and a messenger app. In the example, the intelligent server 200 may determine the path rule ID as a separate path rule ID, and may generate and pass multiple path rules using the email app, the message app, and the messenger app to the electronic device 100. When executing the pass rule, the electronic device 100 may receive a user input for selecting an app and execute a pass rule corresponding to the user input. According to one embodiment, the electronic device 100 may store the user's preference settings and execute the pass rules corresponding to the preference values.

As another example, when the user's utterance is " edit this photo ", the operation of editing the photograph intended to be determined can be performed using a gallery app, a 360 photo editing app, and a moving photo editing app. In the example, the intelligent server 200 determines the pass rule ID as a separate pass rule ID, and generates a plurality of pass rules using the gallery app, the 360 photo editing app, and the moving photo editing app, Lt; / RTI > The electronic device 100 can execute the pass rule corresponding to the type information of the photograph corresponding to the editing object. For example, if the type of picture is 360 pictures, the electronic device 100 may execute a pass rule corresponding to the 360 picture editing application.

Hereinafter, when the sample utterance corresponding to the plurality of path rules and the plurality of path rules transmitted by the intelligent server 200 is included in the response received by the electronic device 100 at the operation 803, the specific operations of the operations 804 to 806 .

In an operation 804 according to an embodiment, the processor 150 may display a plurality of sample utterances on the display 120. At operation 805, the processor 150 may receive a first user input for selecting one of a plurality of sample utterances via the microphone 111 or the touch screen display 120. [ At operation 806, the processor 150 may perform a task associated with the selected sample utterance.

In an operation 805 according to an embodiment, a plurality of sample utterances may include a first sample utterance representing a first intent of user utterance and a second sample utterance representing a second intent other than a first intent. According to one embodiment, a plurality of sample utterances may comprise a first sample utterance corresponding to a first pass rule and a second sample utterance corresponding to a second pass rule.

In one embodiment, the processor 150, in response to receiving the first user input, may display information associated with the sequence of states associated with the selected sample utterance. According to one embodiment, the processor 150 may receive a second user input that again selects the selected sample speech. Processor 150 may perform a task associated with the selected sample utterance in accordance with operation 806 in response to receiving the second user input by having the selected sample utterance state sequence.

According to one embodiment, the information associated with the sequence of states associated with the selected sample utterance may include a description of the operation of the sequence of states (or pass rules) or an utterance to perform the task.

In an operation 804 according to an embodiment, the processor 150 may further display on the display 120 a message for receiving a user input indicating that there is no sample speech corresponding to a user utterance. At operation 805, the processor 150 may receive a user input indicating that there is no sample speech corresponding to a user utterance. Upon receiving the user input, the processor 150 may display a user utterance sentence, and may receive a user input that selects and modifies portions that do not match the user's intent. The processor 150 may send the received user input to the intelligent server 200. According to various embodiments, the intelligent server 200 may proceed to learn intelligent services based on the received user input.

14A to 14D, an embodiment is described in which the electronic device 100 displays a sample speech and receives user input.

FIG. 14A is a diagram illustrating input of a user utterance in a state where a gallery app according to an embodiment is executed. FIG.

14B is a diagram illustrating a screen in which a plurality of sample utterances are displayed according to an embodiment.

14C is a diagram illustrating a screen in which information related to a selected sample speech according to an embodiment is displayed.

FIG. 14D is a diagram illustrating a screen to which an intention of a user according to an embodiment is input

Referring to FIG. 14A according to one embodiment, the electronic device 100 may receive a user utterance 1401 of " Increase screen size " by executing an intelligent service while displaying pictures using the gallery app. The electronic device 100 may transmit the voice signal corresponding to the user's utterance and the context information corresponding to the display of the photograph using the gallery app to the intelligent server 200. [ The intelligent server 200 can transmit a plurality of sample utterances corresponding to the plurality of pass rules and the plurality of pass rules to the electronic device 100 based on the voice signal and the context information.

The processor 150 may display on the display 120 a plurality of received sample fictions 1402, 1403, 1404, and 1405 and a message 1406 of 'Not applicable' as shown in FIG. 14B. The electronic device 100 may receive a first user input for selecting one of a plurality of sample fires via the microphone 111 or the touch screen display 120. [ Processor 150 may perform tasks associated with the selected sample utterance.

In one embodiment, the processor 150 may receive a first user input that selects a sample speech 1404, " Increasing resolution in settings. &Quot; According to one embodiment, the processor 150, in response to receiving the first user input, generates a representative speech 1407 to immediately execute the pass rule corresponding to 'magnify resolution in setting', as shown in FIG. 14C, Can be displayed.

The processor 150 may receive a second user input that selects 'magnify resolution in settings' or selects the object 1408 shown in FIG. 14C. In response to receiving the second user input, the processor 150 may perform the task by executing a pass rule corresponding to 'magnifying resolution in setting'.

In one embodiment, the user utterance is 'screen size depressed', but the intelligent server 200 may generate the text 'increase screen size' from the user utterance. The intelligent server 200 may transmit to the electronic device 100 a sample speech corresponding to a plurality of pass rules and a plurality of pass rules based on the generated text and context information. The electronic device 100 may display on the display 120 a plurality of sample speech 1402, 1403, 1404, 1405 and a message 1406 of 'N / A' as shown in FIG. 14B.

According to one embodiment, the processor 150 may receive a user input that selects the 'no match' message 1406 shown in FIG. 14B. When the user 150 receives the user input, the processor 150 may display 'increase screen size' as shown in FIG. 14D. Processor 150 may receive a user input that selects and modifies " Increase " 1409, which is a portion that is inconsistent with the user's intent. According to one embodiment, the processor 150 may receive a user input that displays another candidate list corresponding to 'increase' and selects one of the candidates, or may receive a textual user input or a speech input.

In an operation 804 according to one embodiment, the processor 150 may display on the display 120 one or more images associated with at least one sample of the plurality of sample fires. According to one embodiment, the displayed one or more images may represent states corresponding to a sample utterance.

15 is a diagram showing a screen in which an image related to a sample speech according to an embodiment is further displayed.

The processor 150 may display an image 1502 corresponding to the states corresponding to the path rule executed when each sample speech 1501 is selected, as shown in FIG. In one embodiment, the displayed state may correspond to each operation screen when the pass rule is executed.

In an operation 804 according to an embodiment, the processor 150 may determine an order to display at least one sample speech and display at least one sample speech according to the determined order.

According to one embodiment, the response received at operation 803 may further include a value corresponding to each of at least one sample speech. For example, a value corresponding to a sample utterance may be a confidence value.

According to one embodiment, the processor 150 may determine an order to display at least one sample speech in order of magnitude of the value corresponding to the sample speech. For example, the processor 150 may display the sample utterances in order of increasing confidence values.

According to one embodiment, the processor 150 may visually represent at least one sample speech, based on the determined order. For example, the processor 150 may visually represent a sample utterance corresponding to the first order of at least one sample utterance, with the remaining sample utterances. According to one embodiment, the processor 150 may display one or more images associated with a sample utterance corresponding to the first order or may display larger than the remaining sample utterances.

According to various embodiments, the processor 150 may determine how to display the sample speech based on the user's selection history data.

16A is a diagram showing that a user utterance is input in a state where a gallery app according to an embodiment is executed.

16B is a diagram showing a screen in which a sample speech is displayed according to a determined order according to an embodiment.

16C is a view showing a screen in which an image related to the sample speech corresponding to the priority according to the embodiment is further displayed.

16D is a diagram showing a screen in which the size of the sample speech is displayed differently according to the determined order according to an embodiment.

Referring to FIG. 16A according to one embodiment, the electronic device 100 may receive a user utterance 1601 of " Tell Hawaii Message " by executing an intelligent service while displaying a photo list using the gallery app. The electronic device 100 may transmit the voice signal corresponding to the user's utterance and the context information corresponding to the display of the photograph using the gallery app to the intelligent server 200. [ The intelligent server 200 can transmit a plurality of sample utterances corresponding to the plurality of pass rules and the plurality of pass rules to the electronic device 100 based on the voice signal and the context information. The intelligent server 200 may further transmit a reliability value corresponding to a plurality of sample utterances.

In one embodiment, a plurality of sample utterances may include "send photo messages taken in Hawaii," "show pictures taken in Hawaii," and "let me know how to get to Hawaii." In one embodiment, the confidence values corresponding to each sample speech may be as follows: "Send a picture message from Hawaii," "Show me pictures taken in Hawaii," and "Tell me how to get to Hawaii."

According to one embodiment, the processor 150 may display a plurality of sample utterances in the order of increasing confidence values as shown in Fig. 16B.

According to one embodiment, the processor 150 may display one or more images 1602 associated with 'send photographic messages taken in Hawaii' having the highest confidence value, as shown in FIG. 16C. The images may represent states corresponding to a sample utterance.

According to one embodiment, the processor 150 may display the sample speech as large as the confidence value is large, as shown in Fig. 16D.

According to various embodiments, the processor 150 may display the sample speech in a manner that combines at least some of the embodiments that represent the sample speech discussed above.

In various embodiments, when receiving a plurality of pass rules, the electronic device 100 does not receive a user input that selects a pass rule, as described above, and sends one pass rule that satisfies a specified condition among a plurality of pass rules You can do it right away. According to various embodiments, the executed path rule may not be consistent with the intent of the user, so the electronic device 100 may perform the operations described below. The operation of the embodiment described below can be performed when the reliability value of the path rule determined by the intelligent server is greater than a specified value.

FIG. 17 is a flowchart illustrating a method for an electronic device according to an embodiment to perform a pass rule and then receive a feedback input to perform another pass rule.

Referring to FIG. 17, at operation 1701, the processor 150 may receive a user utterance via the microphone 111.

At operation 1702, the processor 150 may transmit data associated with the user utterance to the intelligent server via the wireless communication circuitry.

According to one embodiment, the data associated with a user utterance may include a voice signal corresponding to a user utterance.

According to one embodiment, the processor 150 may transmit context information indicating the current state of the electronic device to the intelligent server via the wireless communication circuitry. The context information may include, for example, running application information or state information of an executing application. In one embodiment, the state information of an app may include a state ID, specific screen information of an app, or specific status information in a specific screen

At operation 1703, the processor 150 may receive a response from the intelligent server via the wireless communication circuitry.

According to one embodiment, the received response may include a plurality of pass rules and a sample speech corresponding to each of the plurality of pass rules. The plurality of pass rules and sample utterances may be generated or selected by the intelligent server 200 based on user utterances. According to one embodiment, the response may further include a confidence value corresponding to each of the plurality of pass rules.

At operation 1704, the processor 150 may select one of the plurality of pass rules and perform an operation in accordance with the selected pass rule.

In one embodiment, the processor 150 may select a path rule with the highest confidence value among the plurality of pass rules.

At operation 1705, the processor 150 may receive a feedback input via the microphone 111 or the touch screen display 120.

In one embodiment, the feedback input may correspond to feedback on the result of performing the action in accordance with the selected path rule.

18 is a diagram showing a screen on which an electronic device according to an embodiment receives a feedback input.

The processor 150 may display a screen as shown in FIG. 18 on the display 120 and receive a feedback input. For example, the processor 150 may receive positive feedback, such as " yes " or " yes " through the microphone 111 and may receive negative feedback, such as " .

Alternatively, the processor 150 receives a touch input for selecting a first object 1701 corresponding to a positive feedback input through the touch screen display 120, or a second object 1702 corresponding to a negative feedback input It is possible to receive the touch input to be selected.

Referring again to FIG. 17, if the feedback input is a negative input, at operation 1706, the processor 150 displays on the display 120 a sample speech corresponding to the remaining pass rules not selected in the action 1704 included in the response Can

19 is a diagram showing a screen on which a sample speech corresponding to the remaining pass rule according to the embodiment is displayed.

The processor 150 may display on the display 120 a plurality of sample utterances 1801 corresponding to the remaining pass rules as shown in Fig.

17, at operation 1707, the processor 150 may receive a user input for selecting one of a plurality of sample utterances via the microphone 111 or the touch screen display 120. [

At operation 1708, the processor 150 may perform an operation according to a path rule corresponding to the selected sample speech.

20 is a flowchart illustrating a method for an electronic device according to an embodiment to perform a pass rule and then receive a feedback input to perform another pass rule.

Referring to FIG. 20, in operation 2001, the processor 150 may receive a user utterance via the microphone 111.

In operation 2002, the processor 150 may transmit data associated with a user utterance to the intelligent server 200 via a wireless communication circuit.

In operation 2003, the processor 150 may receive a response from the intelligent server via a wireless communication circuit.

According to one embodiment, the received response may include one pass rule and a sample speech corresponding to the pass rule. The pass rules and sample utterances may be generated or selected by the intelligent server 200 based on user utterances.

In operation 2004, the processor 150 may perform an operation in accordance with the received path rule.

In operation 2005, the processor 150 may receive a feedback input via the microphone 111 or the touch screen display 120.

If the feedback input is a negative input, at operation 2008, the processor 150 sends data (e.g., user utterance and context information) related to user utterances to the intelligent server 200, It is possible to receive the path rule.

In one embodiment, the processor 150 may request alternative path rule transmission when transmitting data associated with user utterances to the intelligent server 200.

At operation 2007, the processor 150 may display on the display 120 a sample speech corresponding to a plurality of alternative pass rules received.

In operation 2008, the processor 150 may receive user input to select one of a plurality of sample fires via the microphone 111 or the touch screen display 120. [

In operation 2009, the processor 150 may perform an operation in accordance with a path rule corresponding to the selected sample speech.

In various embodiments, the electronic device 100 may send data related to user input to the intelligent server 200 to select a sample utterance. The intelligent server 200 can use the received data to determine the intention of the user.

21 is a diagram showing a screen on which an electronic device according to an embodiment receives a feedback input.

According to one embodiment, after operation 1704 or after operation 2004, processor 150 may display a screen as shown in FIG. 21 on display 120 and receive a feedback input. For example, the processor 150 receives a touch input for selecting a first object 2111 corresponding to a positive feedback input through the touch screen display 120, or receives a second object 2112 corresponding to a negative feedback input, A touch input for selecting a touch input.

FIGS. 22A and 22B show a screen displaying a UI associated with a function provided by the electronic device 100 when negative feedback is received according to an embodiment.

According to one embodiment, when a negative feedback input is received, the processor 150 may display a screen as in FIG. 22A or FIG. 22B.

According to one embodiment, when the task that is the object of the negative feedback is a task using an app screen other than the intelligent app (or when the task is executed according to a path rule), the processor 150 can display a screen have. For example, if a negative feedback task is a task of retrieving a photograph using a gallery application, and a negative feedback input is received after the task is performed, the processor 150 may display a screen as shown in FIG. 22A .

According to one embodiment, when the task that is the target of the negative feedback is a task using only the intelligent app screen (or the task is not performed according to the path rule), the processor 150 can display the screen as shown in FIG. have. For example, a task that is a target of negative feedback may be a task that provides weather reports or a task that provides answers according to a query. If a negative feedback input is received after the task is performed, the processor 150 displays a screen as shown in FIG. Can be displayed.

According to one embodiment, as shown in FIG. 22A, the processor 150 includes a UI 2211 for modifying a text corresponding to a misidentified user utterance, a UI 2212 for selecting a pass rule, a pass rule A UI 2213 for searching and selecting, a UI 2214 for searching in the electronic device 100, and a UI 2215 for searching on the Internet can be displayed on the display 120. [ According to one embodiment, the processor 150 may receive user input selecting one of the displayed UIs 2211, 2212, 2213, 2214, and 2215.

According to one embodiment, as shown in FIG. 22B, the processor 150 may include a UI 2221 for modifying text corresponding to a misidentified user utterance, a feedback field indicating that a domain that does not match the user's intention is selected A UI 2222 for searching and selecting a path rule, a UI 2224 for searching in the electronic device 100, and a UI 2225 for searching on the Internet are displayed on the display 120 Can be displayed. The processor 150 may receive user input selecting one of the displayed UIs 2221, 2222, 2223, 2224, and 2225.

According to one embodiment, the UIs 2211 and 2221 for correcting the text corresponding to the misrecognized user utterance are texts 2211a and 2221a recognized from the user utterance by the intelligent server 200 received from the intelligent server 200 ). According to one embodiment, the UI 2212 for selecting pass rules may include a plurality of sample utterances representing each pass rule.

Hereinafter, with reference to FIGS. 23A to 27, an operation performed by the processor 150 when each UI is selected will be described.

23A shows a screen on which a text to be modified is displayed according to an embodiment. 23B shows a screen in which the text to be modified is modified according to an embodiment. 23C shows a result screen in which a text corresponding to a user utterance is modified according to an embodiment. FIG. 23D shows a screen including a result of modifying a text corresponding to a user utterance according to an embodiment and a UI for updating a speech model corresponding to the corrected text. FIG. 23E shows a screen including a pop-up window in response to a network connection failure according to an exemplary embodiment.

According to one embodiment, the processor 150 is a user who selects a portion (2211b, 2221b) of text that the intelligent server 200 received from the intelligent server 200 shown in Figures 22A and 22B, Input can be received. According to one embodiment, in response to receiving a user input, the processor 150 may designate selected texts 2211b and 2221b as text to be modified. For example, the processor 150 may receive a user input selecting " photto " in " Show my recent photto " in FIG. 22A or FIG. In an embodiment, the processor 150 may specify " photto " as the text to be modified.

According to one embodiment, the processor 150 may display the modified text 2311 and the modified complete object 2312, as shown in FIG. 23A. According to one embodiment, the text to be modified 2311 can be modified by input via a virtual keyboard or the like, such as the text 2321 after modification shown in Fig. 23B. For example, " photto " (2311) shown in Fig. 23A can be modified to " photos " 2321 shown in Fig. 23B.

According to one embodiment, after the modification target text 2311 is modified, the processor 150 may receive a user input that selects the modification completed object 2322. [ After the user input for selecting the modified object 2322 is received according to one embodiment, the processor 150 may display the screen shown in FIG. 23C or 23D on the display 120. FIG.

According to one embodiment, after the user input for selecting the modified object 2322 is received, the language settings of the intelligent app and the language settings of the electronic device 100 are different or the recognized text (e.g., "Show my recent photto Quot;) is received via the microphone 111, the processor 150 displays a screen showing the result 2331 of the text corresponding to the user's utterance as shown in FIG. 23C Can be displayed.

According to one embodiment, the processor 150 may transmit data (e.g., modified text data) related to the modified results to the intelligent server 200 via the wireless communication circuitry. According to one embodiment, data associated with the modified results sent to the intelligent server 200 may be stored in the automated speech recognition database 211 of the intelligent server 200. [

According to one embodiment, data associated with the modified results stored in the automatic speech recognition database 211 may be used when the automatic speech recognition module 210 converts user speech into text data. For example, the automatic speech recognition module 210 of the intelligent server 200 may apply a weight value to the modified text to convert the user speech into text data. According to one embodiment, as the number of times of correction of the text increases, the weight value applied to the text may increase, and the accuracy of automatic speech recognition may be increased.

According to one embodiment, after the user input for selecting the modified object 2322 is received, the user's utterance corresponding to the recognized text is the same as the language setting of the intelligent app and the language setting of the electronic device 100, , The processor 150 determines whether the text corresponding to the user's utterance has been modified 2341 and the UI for updating the speech model corresponding to the modified text as shown in FIG. 2342) can be displayed.

According to one embodiment, when a user input is received that selects the UI 2342 for updating the speech model corresponding to the modified text, the processor 150 transmits additional user utterances corresponding to the recognized text to the microphone 111, Lt; / RTI >

According to one embodiment, the processor 150 may transmit data (e.g., voice signals) associated with additional user utterances received via the microphone 111 to the intelligent server 200 via the wireless communication circuitry. According to one embodiment, data associated with additional user utterances sent to the intelligent server 200 may be stored in the automated voice recognition database 211 of the intelligent server 200. [ According to one embodiment, the data associated with additional user utterances stored in the automatic speech recognition database 211 may be used when the automatic speech recognition module 210 converts user utterances to text data.

According to one embodiment, the electronic device 100 must be connected to the network in order for the processor 150 to transfer data associated with additional user utterances to the intelligent server 200. In one embodiment, if a user input is received that selects UI 2342 to update the speech model corresponding to the modified text and the electronic device 100 is not connected to the network, A message 2351 may be displayed on the display 120 indicating that a network connection is required.

According to one embodiment, the processor 150 may receive a user input that selects one of a plurality of sample utterances included in the UI 2212 for selecting the pass rule shown in FIG. 22A. In response to receiving the user input, the processor 150 may execute the pass rules associated with the selected sample utterance. The details of this are the same as those described with reference to Fig. 8 and Figs. 14A to 14D.

24A shows a screen for searching a path rule according to an embodiment. 24B shows a path rule search result screen according to a search word input according to an embodiment. FIG. 24C shows a screen after a pass rule is selected according to an embodiment.

According to one embodiment, the processor 150 may receive user input selecting the UIs 2213, 2223 for searching and selecting the path rules shown in Figures 22A and 22B. According to one embodiment, in response to user input reception, the processor 150 may display on the display 120 a screen for searching for a path rule as shown in FIG. 24A. 24A, a screen for searching for a path rule includes a message 2411 for guiding the user to search for and select a path rule, a search window 2412 for displaying a user input, and a command sentence list 2413 ).

According to one embodiment, the processor 150 may receive input via a virtual keyboard or the like, display the input text 2421 in a search window, and display an instruction sentence list 2422 containing the entered text have. According to one embodiment, the processor 150 may receive a user input that selects one of the command characters included in the displayed command statement list. For example, the processor 150 may receive a text entry of " Remin ", display the entered " Remin " 2421 in the search window as shown in FIG. 24B, A sentence list 2422 can be displayed. Processor 150 may receive a user input that selects a command statement " Reminder for 10min ".

Upon receiving a user input that selects an instruction statement in accordance with one embodiment, the processor 150 may display a result screen on the display 120 that represents the selected instruction statement. According to one embodiment, for example, the result screen may include a selected command statement " Reminder for 10min " 2431 and a guide message 2432, as shown in FIG. 24C.

According to one embodiment, the processor 150 may send data associated with the selected command statement (e.g., path rule information corresponding to the selected command statement) to the intelligent server 200. According to one embodiment, the intelligent server 200 may store data related to the command statement. According to one embodiment, the intelligent server 200 may generate path rules using data associated with stored command statements. For example, if the intelligent server 200 receives data (e.g., a voice signal) related to a user utterance recognized as " Show my recent photto " from the electronic device 100, Quot; Reminder for 10min " using the data related to " Reminder for 10min ".

FIG. 25 shows a screen after a UI for feedback that a domain that does not match a user's intention according to an embodiment is selected.

According to one embodiment, the domain (or application) selected by the intelligent server 200 may not match the intent of the user. For example, the application selected by the intelligent server 200 is an 'Internet application', but an application matching the user's intention may be a 'gallery application'.

According to one embodiment, the processor 150 may receive a user input that selects a UI 2222 to feedback that a domain that does not match the intent of the user shown in FIG. 22B is selected. According to one embodiment, in response to receiving a user input, the processor 150 displays a message 2511 indicating that a domain input that does not match the user's intent has been selected, as shown in FIG. 25, . According to one embodiment, the processor 150 may further display a message (e.g., " It was not a request to use this app ") indicating that the application was selected incorrectly.

According to one embodiment, the processor 150 may send data related to user input to the intelligent server 200 indicating that a domain that does not match the user's intent is selected. According to one embodiment, the intelligent server 200 can increase the accuracy of domain determination by using the received data.

26 shows a screen for performing a search in an electronic device according to an embodiment.

According to one embodiment, the processor 150 may receive user input to select UIs 2214 and 2224 for searching in the electronic device 100 shown in Figures 22A and 22B. According to one embodiment, in response to receiving a user input, the processor 150 may display on the display 120 a screen for performing a search within the electronic device 100, as shown in FIG. 26, a screen for performing a search in the electronic device 100 may include a search window 2611 in which user input is displayed, a search history list 2612, and a proposal application list 2613. According to one embodiment, the search window 2611 is an area in which an input received through a virtual keyboard or the like is displayed, the search history list 2612 includes search histories, the proposal application list 2613 includes user information, And < RTI ID = 0.0 > input. ≪ / RTI >

According to one embodiment, the processor 150 receives input via a virtual keyboard or the like, displays the input text on a search window, and uses the input text as a search term to perform operations that the electronic device 100 can perform Or an installed application or the like.

According to one embodiment, the processor 150 may use a part of text corresponding to a user utterance as a search term. For example, processor 150 may display " recent photto " of " Show my recent photto " in search window 2611 and processor 150 may use the search word itself displayed in search window 2611 A search may be performed or a search may be performed using a search term modified by user input.

FIG. 27 shows a screen for performing a search on the Internet according to an embodiment.

According to one embodiment, the processor 150 may receive user input selecting the UIs 2215 and 2225 for searching on the Internet, as shown in Figures 22A and 22B. According to one embodiment, in response to receiving a user input, the processor 150 may display on the display 120 a screen for performing a search within the electronic device 100 as shown in FIG. Referring to FIG. 27, a screen for performing a search in the electronic device 100 may include a search window 2711 in which user input is displayed.

According to one embodiment, the processor 150 receives an input via a virtual keyboard or the like, displays the input text on the search window 2711, and searches the Internet using the input text as a search term.

According to one embodiment, the processor 150 may use a part of text corresponding to a user utterance as a search term. For example, the processor 150 may display " recent photto " of the " Show my recent photto " in the search window, and the processor 150 may perform a search using the search word itself displayed in the search window, The search can be performed using the search term modified by the search term.

28 is a block diagram of an electronic device 2801 in a network environment 2800 in accordance with various embodiments. An electronic device according to various embodiments disclosed herein can be various types of devices. A personal computer, such as a personal digital assistant (PDA), a tablet PC, a laptop PC (e.g., a desktop PC, a workstation, or a server) , A portable multimedia device (e.g., an e-book reader or an MP3 player), a portable medical device (e.g., cardiac, blood glucose, blood pressure, or temperature sensor), a camera, or a wearable device. (Eg, watches, rings, bracelets, braces, necklaces, eyeglasses, contact lenses, or head-mounted devices (HMDs) Such as, for example, a television, a digital video disk (DVD) player, an audio device, an audio device, an audio device, Devices (eg, A headset, or a headset), a refrigerator, an air conditioner, a vacuum cleaner, an oven, a microwave oven, a washing machine, an air purifier, a settop box, a home automation control panel, a security control panel, a game console, Or the like.

In other embodiments, the electronic device may be a navigation device, a global navigation satellite system (GNSS), an event data recorder (EDR) (e.g., vehicle / ship / airplane black box), an automotive infotainment device (Eg, a head-up display for a vehicle), an industrial or household robot, a drone, an automated teller machine (ATM), a point of sale (POS) Or at least one of an object Internet device (e.g., a light bulb, a sprinkler device, a fire alarm, a thermostat, or a streetlight). The electronic device according to the embodiment of the present document is not limited to the above-described devices and may also be provided with a plurality of electronic devices such as a smart phone having a function of measuring biometric information (e.g., heartbeat or blood sugar) It is possible to combine the functions of the devices. In this document, the term user may refer to a person using an electronic device or a device using an electronic device (e.g., an artificial intelligence electronic device).

28, electronic device 2801 (e.g., electronic device 100) in network environment 2800 communicates with electronic device 2802 via short range wireless communication 2898 or via network 2899 To communicate with electronic device 2804 or server 2808 (e.g., intelligent server 200). According to one embodiment, the electronic device 2801 may communicate with the electronic device 2804 via the server 2808. [

According to one embodiment, the electronic device 2801 includes a bus 2810, a processor 2820 (e.g., a processor 150), a memory 2830, an input device 2850 (e.g., a microphone or a mouse) An audio module 2870, a sensor module 2876, an interface 2877, a haptic module 2879, a camera module 2880, a power management module 2888, and a battery 2889, a communication module 2890 ), And a subscriber identity module 2896. In some embodiments, electronic device 2801 may omit at least one of the components (e.g., display 2860 or camera module 2880) or additionally include other components.

Bus 2810 may include circuitry to connect components 2820-2890 to each other and to communicate signals (e.g., control messages or data) between the components.

Processor 2820 may be one of a central processing unit (CPU), an application processor (AP), a graphics processing unit (GPU), an image signal processor (ISP) Or more. According to one embodiment, the processor 2820 may be implemented as a system on chip (SoC) or a system in package (SiP). Processor 2820 may control an operating system or at least one other component (e.g., hardware or software component) of an electronic device 2801 coupled to processor 2820 by driving an application program , Various data processing and calculation can be performed. Processor 2820 may load and process instructions or data received from at least one of the other components (e.g., communication module 2890) into volatile memory 2832 and store the resulting data in nonvolatile memory 2834 .

Memory 2830 may include volatile memory 2832 or non-volatile memory 2834. [ Volatile memory 2832 may be comprised of, for example, random access memory (RAM) (e.g., DRAM, SRAM, or SDRAM). The non-volatile memory 2834 may be implemented as, for example, a programmable read-only memory (PROM), a one time PROM, an erasable programmable read-only memory (EPROM), an electrically erasable programmable ROM (EEPROM), a mask ROM, a hard disk drive, or a solid state drive (SSD). The nonvolatile memory 2834 may also include a built-in memory 2836 disposed therein or a stand-alone enclosure 2836 that can be used only when needed, depending on the form of connection with the electronic device 2801. [ And a memory 2838. The external memory 2838 may be a flash drive such as a compact flash (CF), a secure digital (SD), a micro-SD, a mini-SD, an extreme digital (xD) , Or a memory stick. The external memory 2838 may be functionally or physically connected to the electronic device 2801 via a wired (e.g., cable or universal serial bus) or wireless (e.g., Bluetooth).

 The memory 2830 may store, for example, instructions or data related to at least one other software component of the electronic device 2801, e.g., program 2840. [ The program 2840 may include, for example, a kernel 2841, a library 2843, an application framework 2845, or an application program (interchangeably "application") 2847.

Input device 2850 may include a microphone, a mouse, or a keyboard. According to one embodiment, the keyboard may be connected by a physical keyboard, or may be represented by a virtual keyboard via display 2860. [

The display device 2860 may include a display, a hologram device, or a control circuit for controlling the projector and the device. The display may include, for example, a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a microelectromechanical system (MEMS) display, or an electronic paper display . The display, according to one embodiment, can be implemented flexibly, transparently, or wearably. The display may include an interchangeably "force sensor " capable of sensing a user's touch, gesture, proximity, or hovering input, which can measure touch circuitry or pressure to the touch . The touch circuit or pressure sensor may be implemented integrally with the display, or may be implemented with one or more sensors separate from the display. The hologram device can display stereoscopic images in the air using the interference of light. The projector can display images by projecting light onto the screen. The screen may be located, for example, inside or outside the electronic device 2801.

Audio module 2870 can, for example, bidirectionally convert sound and electrical signals. According to one embodiment, the audio module 2870 is configured to acquire sound through an input device 2850 (e.g., a microphone), or to output devices (not shown) (E. G., Wireless speakers or wireless headphones) or electronic devices 2806 (e. G., Wired speakers or wired headphones) coupled to the electronic device 2801 Can be output.

The sensor module 2876 measures or senses the operating state (e.g., power or temperature) of the interior of the electronic device 2801 or an external environmental condition (e.g., altitude, humidity, or brightness) And generate an electrical signal or data value corresponding to the measured or sensed state information. The sensor module 2876 can be, for example, a gesture sensor, a gyro sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor (e.g., an RGB (red, green, An infrared sensor, a biological sensor such as an iris sensor, a fingerprint sensor or a heartbeat rate monitoring (HRM) sensor, an electronic nose sensor, an electromyography sensor, an electroencephalogram sensor, an electrocardiogram (ECG) Sensor), a temperature sensor, a humidity sensor, an illumination sensor, or an ultraviolet (UV) sensor. The sensor module 2876 may further include a control circuit for controlling at least one or more sensors belonging thereto. In some embodiments, the electronic device 2801 may control the sensor module 2876 using a processor (e.g., a sensor hub) separate from the processor 2820 or the processor 2820. When using a separate processor (e. G., A sensor hub), the electronic device 2801 does not wake up the processor 2820 while the processor 2820 is in a sleep state, Or at least some of the operation or state of the microprocessor 2876.

Interface 2877 may be implemented as a high definition multimedia interface (HDMI), a USB, an optical interface, an RS-232 (recommended standard 232), a D-sub (D-subminiature), an MHL high-definition link interface, an SD card / multi-media card (MMC) interface, or an audio interface. The connection terminal 2878 can physically connect the electronic device 2801 and the electronic device 2806. According to one embodiment, connection terminal 2878 may include, for example, a USB connector, an SD card / MMC connector, or an audio connector (e.g., a headphone connector).

The haptic module 2879 can convert an electrical signal into a mechanical stimulus (e.g., vibration or motion) or an electrical stimulus. For example, the haptic module 2879 may provide a stimulus associated with the tactile or kinesthetic sense to the user. The haptic module 2879 may include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

The camera module 2880 can take, for example, a still image and a moving image. Camera module 2880 may include, in accordance with one embodiment, one or more lenses (e.g., a wide angle lens and a telephoto lens, or a front lens and a rear lens), an image sensor, an image signal processor, a xenon lamp, etc.).

The power management module 2888 is a module for managing the power of the electronic device 2801, and may be configured as at least a part of, for example, a power management integrated circuit (PMIC).

The battery 2889 may be recharged by an external power source, for example, a primary battery, a secondary battery, or a fuel cell to supply power to at least one component of the electronic device 2801. [

Communication module 2890 may be used to establish communication channels between electronic device 2801 and external devices such as first external electronic device 2802, second external electronic device 2804, or server 2808, And support wired or wireless communication through the established communication channel. According to one embodiment, the communication module 2890 includes a wireless communication module 2892 or a wired communication module 2894 and is coupled to a first network 2898 (e.g., Bluetooth or IrDA (e. g., a short distance communication network such as an infrared data association) or a second network 2899 (e. g., a long distance communication network such as a cellular network).

The wireless communication module 2892 may support, for example, cellular communication, short-range wireless communication, or GNSS communication. Cellular communications may include, for example, long-term evolution (LTE), LTE Advance, code division multiple access (CDMA), wideband CDMA (WCDMA), universal mobile telecommunications system (UMTS) ), Or Global System for Mobile Communications (GSM). The short-range wireless communication may include, for example, wireless fidelity, Wi-Fi Direct, Li-Fi, Bluetooth, Bluetooth low energy, Zigbee, near field communication, magnetic secure transmission, radio frequency (RF), or body area network (BAN). The GNSS may include, for example, a Global Positioning System (GPS), a Global Navigation Satellite System (Glonass), a Beidou Navigation Satellite System (Beidou), or a Galileo (European Global Satellite-based Navigation System). In this document, "GPS" can be used interchangeably with "GNSS ".

According to one embodiment, the wireless communication module 2892 performs the identification and authentication of the electronic device 2801 within the communication network using, for example, the subscriber identity module 2896, when supporting cellular communication can do. According to one embodiment, the wireless communication module 2892 may include a CP that is separate from the processor 2820 (e.g., an AP). In such a case, the CP may be used in place of or in place of the processor 2820 while the processor 2820 is in an inactive (e.g., sleep) state, or while the processor 2820 is in the active state, Together, it may perform at least some of the functions associated with at least one of the components 2810-2896 of the electronic device 2801. According to one embodiment, the wireless communication module 2892 may be composed of a plurality of communication modules that support only the corresponding communication method of a cellular communication module, a short-range wireless communication module, or a GNSS communication module.

The wired communication module 2894 may include, for example, a local area network (LAN), a power line communication, or a plain old telephone service (POTS).

The first network 2898 may be a Wi-Fi direct or Bluetooth capable of transmitting or receiving commands or data via a direct wireless connection between the electronic device 2801 and the first external electronic device 2802, . The second network 2899 may be a telecommunication network (e.g., a local area network (LAN)) capable of transmitting or receiving commands or data between the electronic device 2801 and the second external electronic device 2804, A computer network such as a wide area network (WAN), the Internet, or a telephone network).

According to various embodiments, the command or the data may be transmitted or received between the electronic device 2801 and the second external electronic device 2804 via the server 2808 connected to the second network. Each of the first and second external electronic devices 2802 and 2804 may be the same or a different kind of device as the electronic device 2801. [ According to various embodiments, all or a portion of the operations performed in the electronic device 2801 may be performed in one or more other electronic devices (e.g., electronic devices 2802, 2804, or server 2808). According to the example, in the event that the electronic device 2801 has to perform a certain function or service automatically or on demand, the electronic device 2801 may, instead of or in addition to executing the function or service itself, (E. G., Electronic device 2802,2804, or server 2808) may request some functionality from other devices (e. G., Electronic device 2802,2804, or server 2808) Function or an additional function and transmit the result to the electronic device 2801. The electronic device 2801 can directly or additionally process the received result to provide the requested function or service. There is a server computing techniques can be used - for example, cloud computing, distributed computing, or client.

An electronic device according to an embodiment disclosed in this document includes a housing; A touch screen display disposed within the housing and exposed through a first portion of the housing; A microphone disposed within the housing and exposed through a second portion of the housing; At least one speaker disposed within the housing and exposed to a third portion of the housing; A wireless communication circuit disposed within the housing; A processor disposed within the housing and electrically coupled to the touch screen display, the microphone, the at least one speaker, and the wireless communication circuitry; And a memory disposed in the housing and electrically coupled to the processor, wherein the memory, in execution, causes the processor to receive, in a first example, a first user utterance via the microphone, Wherein said first request comprises a first request to perform said first task with said electronic device and transmits first data associated with said first user utterance to an external server via said wireless communication circuit, Receiving a first response from an external server, the first response including information about sequences of first states of the electronic device for performing the first task, and wherein the electronic device is operable to receive sequences of first states To perform the first task, and in a second example, to receive a second user utterance via the microphone Wherein the second user utterance includes a second request to perform the second task with the electronic device and transmits second data associated with the second user utterance to the external server via the wireless communication circuit, Receiving a second response from the external server via a wireless communication circuit, the second response including at least one sample speech indicating the intent of the user utterance, and the at least one sample speech being associated with the second user utterance For displaying the at least one sample speech on the touch screen display and for selecting one of the at least one sample speech through the microphone or the touch screen display, Wherein the electronic device is adapted to receive the selected one of the at least one sample speech To perform the second task by having a sequence of second states associated with the second task.

In one embodiment, the at least one sample user utterance may comprise a first sample utterance representing a first intent and a second sample utterance representing a second intent different from the first intent.

In one embodiment, the instructions may further cause the processor to display one or more images associated with the at least one sample speech on the touch screen display.

In one embodiment, the one or more images may represent the second states.

In one embodiment, the instructions may cause the processor to determine an order in which to display the at least one sample speech, and to display the at least one sample speech in the determined order.

In an embodiment, the second response further comprises a value corresponding to each of the at least one sample utterances, and the instructions cause the processor to perform the steps of: ordering the at least one sample utterance in order of magnitude of the value Can be determined.

In one embodiment, the instructions may cause the processor to visually represent a sample utterance corresponding to a first order of the at least one sample utterance, with the remaining sample utterances.

In one embodiment, the instructions cause the processor to: in response to receiving the first user input, display information related to a sequence of second states associated with the selected sample utterance, 2 user input, and in response to receiving the second user input, perform the second task.

In one embodiment, the displayed information may comprise at least one of an operation of the sequence of second states or an utterance to cause the second task to perform.

Further, an electronic device according to an embodiment disclosed in this document includes: a housing; A touch screen display disposed within the housing and exposed through a first portion of the housing; A microphone disposed within the housing and exposed through a second portion of the housing; At least one speaker disposed within the housing and exposed to a third portion of the housing; A wireless communication circuit disposed within the housing; A processor disposed within the housing and electrically coupled to the touch screen display, the microphone, the at least one speaker, and the wireless communication circuitry; And a memory disposed in the housing and electrically coupled to the processor, wherein the memory, in execution, causes the processor to receive a user utterance through the microphone, The method comprising: sending a request for performing a user utterance to the external server via the wireless communication circuit; receiving a response from the external server via the wireless communication circuit; Wherein the plurality of sample utterances comprises a first sample utterance corresponding to a first pass rule and a second sample utterance corresponding to a second pass rule, Wherein the touch screen display is generated by the external server based on data associated with an utterance, Receiving a first user input for selecting one of the at least one sample speech via the microphone or the touch screen display and the electronic device displaying a selected one of the at least one sample speech And to have the task have a sequence of states associated with it.

In one embodiment, the instructions may further cause the processor to display one or more images associated with the at least one sample speech on the touch screen display.

In one embodiment, the one or more images may represent the states.

In one embodiment, the instructions may cause the processor to determine an order in which to display the at least one sample speech, and to display the at least one sample speech in the determined order.

In one embodiment, the response further comprises a value corresponding to each of the at least one sample utterances, and the instructions cause the processor to perform the steps of displaying the at least one sample utterance in order of magnitude of the value Can be determined.

In one embodiment, the instructions may cause the processor to visually represent a sample utterance corresponding to a first order of the at least one sample utterance, with the remaining sample utterances.

In one embodiment, the instructions cause the processor to: display, in response to receiving the first user input, information relating to a sequence of states associated with the selected sample speech, and to select a second And to perform the task in response to receiving the user input and receiving the second user input.

In one embodiment, the displayed information may comprise at least one of an operation of the sequence of states or an utterance to cause the task to be performed.

Also, a storage medium according to one embodiment disclosed herein is a storage medium storing computer-readable instructions, which when executed by a processor of an electronic device causes the electronic device to receive a user utterance Wherein the user utterance comprises a request to perform a task with the electronic device, the method comprising: sending data associated with the user utterance to an external server; receiving a response from the external server; Wherein the at least one sample utterance is selected by the external server based on the user utterance and displaying the at least one sample utterance, the at least one sample utterance indicating the at least one sample A first user input for selecting one of the utterances; And performing the task by causing the electronic device to have a sequence of states associated with a selected one of the at least one sample utterances.

In one embodiment, the at least one sample user utterance may comprise a first sample utterance representing a first intent and a second sample utterance representing a second intent different from the first intent.

In one embodiment, the instructions, when executed by a processor of the electronic device, cause the electronic device to further display one or more images associated with the at least one sample speech.

It should be understood that the various embodiments of the present document and the terminology used are not intended to limit thetechniques described in this document to specific embodiments, but rather to include various modifications, equivalents, and / or alternatives of the embodiments. In connection with the description of the drawings, like reference numerals may be used for similar components. The singular expressions may include plural expressions unless the context clearly dictates otherwise. In this document, the expressions "A or B," "at least one of A and / or B," "A, B or C," or "at least one of A, B, and / Possible combinations. Expressions such as " first, "" second," " first, "or" second, " But is not limited to those components. When it is mentioned that some (e.g., first) component is "(functionally or communicatively) connected" or "connected" to another (second) component, May be connected directly to the component, or may be connected through another component (e.g., a third component).

In this document, the term " adapted to or configured to "as used herein is intended to encompass all types of information, including, but not limited to, "Quot;, " made to do ", " designed to ", or "designed to" In some situations, the expression "a device configured to" may mean that the device can "do " with other devices or components. For example, a processor configured (or configured) to perform the phrases "A, B, and C" may be a processor dedicated to performing the operations (e.g., an embedded processor) By executing the above programs, it can mean a general-purpose processor (e.g., CPU or AP) that can perform the corresponding operations.

As used herein, the term "module" includes a unit of hardware, software or firmware and may be used interchangeably with terms such as, for example, logic, logic blocks, components, . A "module" may be an integrally constructed component or a minimum unit or part thereof that performs one or more functions. "Module" may be implemented either mechanically or electronically, for example, by application-specific integrated circuit (ASIC) chips, field-programmable gate arrays (FPGAs) And may include programmable logic devices.

At least some of the devices (e.g., modules or functions thereof) or methods (e.g., operations) according to various embodiments may be stored in a computer readable storage medium (e.g., memory 2830) . ≪ / RTI > When the instruction is executed by a processor (e.g., processor 2820), the processor may perform a function corresponding to the instruction. The computer-readable recording medium may be a hard disk, a floppy disk, a magnetic medium such as a magnetic tape, an optical recording medium such as a CD-ROM, a DVD, a magnetic-optical medium such as a floppy disk, The instructions may include code generated by the compiler or code that may be executed by the interpreter.

Each of the components (e.g., modules or program modules) according to various embodiments may be comprised of a single entity or a plurality of entities, and some subcomponents of the previously mentioned subcomponents may be omitted, or other subcomponents . Alternatively or additionally, some components (e.g., modules or program modules) may be integrated into one entity to perform the same or similar functions performed by each respective component prior to integration. Operations performed by modules, program modules, or other components in accordance with various embodiments may be performed sequentially, in a parallel, repetitive, or heuristic manner, or at least some operations may be performed in a different order, Can be added.

Claims (20)

In an electronic device,
housing;
A touch screen display disposed within the housing and exposed through a first portion of the housing;
A microphone disposed within the housing and exposed through a second portion of the housing;
At least one speaker disposed within the housing and exposed to a third portion of the housing;
A wireless communication circuit disposed within the housing;
A processor disposed within the housing and electrically coupled to the touch screen display, the microphone, the at least one speaker, and the wireless communication circuitry; And
A memory disposed within the housing and electrically coupled to the processor, the memory having, in execution,
In the first example,
Wherein the first user utterance comprises a first request for performing a first task with the electronic device,
Transmitting first data associated with the first user utterance to an external server via the wireless communication circuit,
Receive a first response from the external server via the wireless communication circuit and the first response includes information about sequences of first states of the electronic device for performing the first task, To perform the first task by having the sequences of first states,
In the second example,
Wherein the second user utterance comprises a second request for performing a second task with the electronic device,
Transmitting second data related to the second user utterance to the external server through the wireless communication circuit,
Wherein the second response comprises at least one sample speech indicating an intention of the user utterance, and wherein the at least one sample utterance comprises a second user utterance Is selected by the external server,
Displaying the at least one sample speech on the touch screen display,
Receive a first user input for selecting one of the at least one sample speech via the microphone or the touch screen display,
Wherein the electronic device is configured to perform the second task by having the sequence of second states associated with the selected one of the at least one sample utterance. The method according to claim 1,
Wherein the at least one sample user utterance comprises:
A first sample speech representing a first intent and a second sample speech representing a second intent different from the first intent. The method according to claim 1,
Wherein the instructions cause the processor to:
And causes the touch screen display to display one or more images associated with the at least one sample speech. The method of claim 3,
Wherein the at least one image represents the second states. The method according to claim 1,
Wherein the instructions cause the processor to:
Determine an order of displaying the at least one sample speech,
And to display the at least one sample speech according to the determined order. The method of claim 5,
Wherein the second response further comprises a value corresponding to each of the at least one sample speech,
Wherein the instructions cause the processor to:
And determine an order of displaying the at least one sample speech in order of magnitude of the value. The method of claim 5,
Wherein the instructions cause the processor to:
To cause the sample utterance corresponding to the first order of the at least one sample utterance to be visually displayed differently from the remaining sample utterances. The method according to claim 1,
Wherein the instructions cause the processor to:
Responsive to receiving the first user input, displaying information related to a sequence of second states associated with the selected sample speech,
Receiving a second user input that again selects the selected sample speech,
And in response to receiving the second user input, perform the second task. The method of claim 8,
Wherein the displayed information comprises at least one of an operation of the sequence of second states or an ignition to cause the second task to perform. In an electronic device,
housing;
A touch screen display disposed within the housing and exposed through a first portion of the housing;
A microphone disposed within the housing and exposed through a second portion of the housing;
At least one speaker disposed within the housing and exposed to a third portion of the housing;
A wireless communication circuit disposed within the housing;
A processor disposed within the housing and electrically coupled to the touch screen display, the microphone, the at least one speaker, and the wireless communication circuitry; And
A memory disposed within the housing and electrically coupled to the processor, the memory having, in execution,
Receiving a user utterance through the microphone, the user utterance comprising a request to perform a task with the electronic device,
Transmitting data related to the user's utterance to an external server via the wireless communication circuit,
The response being received from the external server via the wireless communication circuit, the response including a plurality of sample utterances, wherein the plurality of sample utterances correspond to a first sample utterance and a second pass rule corresponding to a first pass rule Wherein the first pass rule and the second pass rule are generated by the external server based on data associated with the user utterance,
Displaying the at least one sample speech on the touch screen display,
Receive a first user input for selecting one of the at least one sample speech via the microphone or the touch screen display,
Wherein the electronic device is configured to perform the task by having the sequence of states associated with the selected one of the at least one sample utterance. The method of claim 10,
Wherein the instructions cause the processor to:
And causes the touch screen display to display one or more images associated with the at least one sample speech. The method of claim 11,
Wherein the at least one image represents the states. The method of claim 10,
Wherein the instructions cause the processor to:
Determine an order of displaying the at least one sample speech,
And to display the at least one sample speech according to the determined order. 14. The method of claim 13,
Wherein the response further comprises a value corresponding to each of the at least one sample speech,
The instructions cause the processor to:
And determine an order of displaying the at least one sample speech in order of magnitude of the value. 14. The method of claim 13,
The instructions cause the processor to:
To cause the sample utterance corresponding to the first order of the at least one sample utterance to be visually displayed differently from the remaining sample utterances. The method of claim 10,
The instructions cause the processor to:
Responsive to receiving the first user input, displaying information related to a sequence of states associated with the selected sample speech,
Receiving a second user input that again selects the selected sample speech,
And in response to receiving the second user input, perform the task. 18. The method of claim 16,
Wherein the displayed information comprises at least one of an operation of the sequence of states or an utterance to cause the task to perform. 22. A storage medium storing computer-readable instructions, the instructions being executable by a processor of the electronic device to cause the electronic device to:
Receiving a user utterance, the user utterance comprising a request to perform a task with the electronic device,
Transmitting data associated with the user utterance to an external server,
Receiving a response from the external server, the response including at least one sample utterance indicative of intent of the user utterance, the at least one sample utterance being selected by the external server based on the user utterance,
Displaying the at least one sample speech,
Receiving a first user input for selecting one of the at least one sample speech; and
To perform the task by causing the electronic device to have a sequence of states associated with a selected one of the at least one sample utterance. 19. The method of claim 18,
Wherein the at least one sample user utterance comprises a first sample utterance representing a first intent and a second sample utterance representing a second intent different from the first intent. 19. The method of claim 18,
Wherein the instructions, when executed by a processor of the electronic device, cause the electronic device to:
And to display one or more images associated with the at least one sample speech.

Images