KR102402224B1 - Device for performing a task corresponding to user utterance - Google Patents

Abstract

An electronic device is disclosed. An electronic device according to an embodiment includes a housing; a touch screen display disposed within the housing and exposed through a first portion of the housing; a microphone disposed within the housing and exposed through a second portion of the housing; at least one speaker disposed within the housing and exposed through a third portion of the housing; wireless communication circuitry disposed within the housing; a processor disposed within the housing and electrically coupled to the touchscreen display, the microphone, the at least one speaker, and the wireless communication circuitry; and a memory disposed within the housing and electrically coupled to the processor, wherein the memory, when executed, causes the processor to, in a first example, receive a first user utterance through the microphone, the first user The utterance includes a first request for performing a first task with the electronic device, and transmits first data related to the first user utterance to an external server through the wireless communication circuit, and through the wireless communication circuit Receive a first response from an external server, wherein the first response includes information about sequences of first states of the electronic device for performing the first task, wherein the electronic device performs the sequences of first states perform the first task by having to have, in a second example, receive a second user utterance through the microphone, the second user utterance comprising a second request to perform a second task with the electronic device and transmit second data related to the second user utterance to the external server through the wireless communication circuit, and receive a second response from the external server through the wireless communication circuit, wherein the second response is transmitted to the user at least one sample utterance indicating an intention of the utterance, wherein the at least one sample utterance is selected by the external server based on the second user utterance, and displaying the at least one sample utterance on the touchscreen display and receiving a first user input for selecting one of the at least one sample utterances through the microphone or the touchscreen display, and causing the electronic device to display a second state related to the selected one of the at least one sample utterances. The instructions for performing the second task may be stored by having a sequence of . In addition to this, various embodiments identified through the specification are possible.

Description

Electronic device for performing a task corresponding to user utterance

Embodiments disclosed in this document relate to a technique for performing a task corresponding to a user's utterance.

In addition to the traditional input method using a keyboard or mouse, recent electronic devices may support various input methods such as voice input. For example, electronic devices such as smart phones or tablets may recognize a user's voice input while the voice recognition service is running, execute an operation corresponding to the voice input, or provide a search result.

Recently, speech recognition services are being developed based on a technology for processing natural language. The technology for processing natural language is a technology that recognizes the intention of a user's utterance and provides a result that matches the intention to the user.

The voice recognition service may suggest only one task by using the user's utterance. Accordingly, when an ambiguous user's utterance is used, the existing service proposes a task inconsistent with the user's intention, thereby causing inconvenience to the user.

Various embodiments of the present disclosure are intended to propose an electronic device that performs a task corresponding to a user's intention when an ambiguous user's utterance is received.

An electronic device according to an embodiment disclosed in this document includes: a housing; a touch screen display disposed within the housing and exposed through a first portion of the housing; a microphone disposed within the housing and exposed through a second portion of the housing; at least one speaker disposed within the housing and exposed through a third portion of the housing; wireless communication circuitry disposed within the housing; a processor disposed within the housing and electrically coupled to the touchscreen display, the microphone, the at least one speaker, and the wireless communication circuitry; and a memory disposed within the housing and electrically coupled to the processor, wherein the memory, when executed, causes the processor to, in a first example, receive a first user utterance through the microphone, the first user The utterance includes a first request for performing a first task with the electronic device, and transmits first data related to the first user utterance to an external server through the wireless communication circuit, and through the wireless communication circuit Receive a first response from an external server, wherein the first response includes information about sequences of first states of the electronic device for performing the first task, wherein the electronic device performs the sequences of first states perform the first task by having to have, in a second example, receive a second user utterance through the microphone, the second user utterance comprising a second request to perform a second task with the electronic device and transmit second data related to the second user utterance to the external server through the wireless communication circuit, and receive a second response from the external server through the wireless communication circuit, wherein the second response is transmitted to the user at least one sample utterance indicating an intention of the utterance, wherein the at least one sample utterance is selected by the external server based on the second user utterance, and displaying the at least one sample utterance on the touchscreen display and receiving a first user input for selecting one of the at least one sample utterances through the microphone or the touchscreen display, and causing the electronic device to display a second state related to the selected one of the at least one sample utterances. The instructions for performing the second task may be stored by having a sequence of .

In addition, an electronic device according to an embodiment disclosed in this document may include: a housing; a touch screen display disposed within the housing and exposed through a first portion of the housing; a microphone disposed within the housing and exposed through a second portion of the housing; at least one speaker disposed within the housing and exposed through a third portion of the housing; wireless communication circuitry disposed within the housing; a processor disposed within the housing and electrically coupled to the touchscreen display, the microphone, the at least one speaker, and the wireless communication circuitry; and a memory disposed within the housing and electrically coupled to the processor, wherein the memory, when executed, causes the processor to receive a user utterance through the microphone, wherein the user utterance performs a task with the electronic device. a request to perform, transmitting data related to the user utterance to an external server via the wireless communication circuitry, receiving a response from the external server via the wireless communication circuitry, wherein the response comprises a plurality of sample utterances; wherein the plurality of sample utterances include a first sample utterance corresponding to a first pass rule and a second sample utterance corresponding to a second pass rule, wherein the first pass rule and the second pass rule are determined by the user. generating by the external server based on data related to a utterance, displaying the at least one sample utterance on the touchscreen display, and selecting one of the at least one sample utterance through the microphone or the touchscreen display and receive a first user input for utterance, and cause the electronic device to have a sequence of states related to a selected one of the at least one sample utterance to perform the task.

In addition, the storage medium according to an embodiment disclosed in this document is a storage medium for storing computer-readable instructions, wherein when the instructions are executed by a processor of an electronic device, the electronic device causes the electronic device to receive a user's utterance operation, the user utterance includes a request for performing a task with the electronic device, transmitting data related to the user utterance to an external server, receiving a response from the external server, wherein the response is the user utterance at least one sample utterance indicating the intention of The operation of performing the task may be performed by receiving a first user input for selecting one of the utterances, and causing the electronic device to have a sequence of states related to the selected one of the at least one sample utterances .

According to various embodiments of the present disclosure, it is possible to enable the user to execute a desired task by displaying a group of candidates corresponding to a task expected to match the user's intention from ambiguous user utterance and receiving an input for selecting the candidate.

According to various embodiments of the present disclosure, reliability of an intelligent service may be improved by executing a task corresponding to a user's intention.

1 is a view showing an integrated intelligent system according to various embodiments of the present invention.
2 is a block diagram illustrating a user terminal of an integrated intelligent system according to an embodiment of the present invention.
3 is a diagram illustrating running an intelligent app of a user terminal according to an embodiment of the present invention.
4 is a block diagram illustrating an intelligent server of an integrated intelligent system according to an embodiment of the present invention.
5 is a diagram illustrating a method for generating a path rule by a path natural language understanding module (NLU) according to an embodiment of the present invention.
6 is a diagram illustrating that a context module of a processor collects a current state according to an embodiment of the present invention.
7 is a diagram illustrating that a persona module manages user information according to an embodiment of the present invention.
8 is a flowchart illustrating a method of performing a task based on a user utterance request according to an embodiment.
9 is a block diagram illustrating a natural language understanding module of an intelligent server according to an embodiment.
10 illustrates an operation included in a pass rule according to an embodiment.
11A is a diagram illustrating that a user's utterance is input while a gallery app is running, according to an exemplary embodiment.
11B is a diagram illustrating a screen on which a plurality of sample utterances are displayed, according to an exemplary embodiment.
12 is a flowchart illustrating a method for an intelligent server to transmit a plurality of sample utterances according to an embodiment.
13 is a flowchart illustrating a method for an intelligent server to transmit a plurality of sample utterances according to an embodiment.
14A is a diagram illustrating that a user's utterance is input while a gallery app is running, according to an exemplary embodiment.
14B is a diagram illustrating a screen on which a plurality of sample utterances are displayed, according to an exemplary embodiment.
14C is a diagram illustrating a screen on which information related to a selected sample utterance is displayed, according to an exemplary embodiment.
14D is a diagram illustrating a screen on which a user's intention is input, according to an exemplary embodiment;
15 is a diagram illustrating a screen on which an image related to a sample utterance is further displayed, according to an exemplary embodiment.
16A is a diagram illustrating that a user's utterance is input while a gallery app is running, according to an exemplary embodiment.
16B is a diagram illustrating a screen on which sample utterances are displayed according to a determined order, according to an exemplary embodiment.
16C is a diagram illustrating a screen on which an image related to a sample utterance corresponding to a priority is further displayed, according to an exemplary embodiment.
16D is a diagram illustrating a screen in which sizes of sample utterances are displayed differently according to a determined order, according to an exemplary embodiment.
17 is a flowchart illustrating a method of an electronic device performing another pass rule by receiving a feedback input after performing a pass rule, according to an exemplary embodiment.
18 is a diagram illustrating a screen on which an electronic device receives a feedback input, according to an embodiment.
19 is a diagram illustrating a screen on which sample utterances corresponding to remaining pass rules are displayed, according to an exemplary embodiment.
20 is a flowchart illustrating a method of performing another pass rule by receiving a feedback input after an electronic device performs a pass rule, according to an exemplary embodiment.
21 is a diagram illustrating a screen on which an electronic device receives a feedback input, according to an embodiment.
22A and 22B illustrate screens on which a UI related to a function provided by the electronic device is displayed when negative feedback is received, according to an exemplary embodiment.
23A illustrates a screen on which a text to be edited is displayed, according to an exemplary embodiment.
23B illustrates a screen in which a text to be edited is corrected, according to an exemplary embodiment.
23C illustrates a result screen in which text corresponding to a user's utterance is corrected, according to an exemplary embodiment.
23D illustrates a screen including a result of a text corresponding to a user's utterance corrected and a UI for updating a voice model corresponding to the corrected text, according to an exemplary embodiment.
23E illustrates a screen including a pop-up window according to a network connection failure according to an embodiment.
24A illustrates a screen for searching for a pass rule according to an embodiment.
24B illustrates a pass rule search result screen according to a search word input according to an exemplary embodiment.
24C illustrates a screen after a pass rule is selected according to an exemplary embodiment.
25 illustrates a screen after a UI for feedback that a domain that does not match a user's intention is selected is selected, according to an exemplary embodiment.
26 illustrates a screen for performing a search in an electronic device according to an exemplary embodiment.
27 illustrates a screen for performing a search on the Internet according to an embodiment.
28 is a block diagram of an electronic device in a network environment according to various embodiments of the present disclosure;
In connection with the description of the drawings, the same or similar reference numerals may be used for the same or similar components.

Hereinafter, various embodiments of the present invention will be described with reference to the accompanying drawings. However, this is not intended to limit the present invention to specific embodiments, and it should be understood that various modifications, equivalents, and/or alternatives of the embodiments of the present invention are included.

Before describing an embodiment of the present invention, an integrated intelligent system to which an embodiment of the present invention can be applied will be described.

1 is a view showing an integrated intelligent system according to various embodiments of the present invention.

Referring to FIG. 1 , the integrated intelligent system 10 may include a user terminal 100 , an intelligent server 200 , a personalized information server 300 , or a proposal server 400 .

The user terminal 100 is a service required by the user through an app (or an application program) (eg, an alarm app, a message app, a photo (gallery) app, etc.) stored inside the user terminal 100 can provide For example, the user terminal 100 may execute and operate another app through an intelligent app (or a voice recognition app) stored in the user terminal 100 . Through the intelligent app of the user terminal 100, the other app may be executed and a user input for executing an operation may be received. The user input may be received through, for example, a physical button, a touch pad, a voice input, a remote input, or the like. According to an embodiment, the user terminal 100 may correspond to various terminal devices (or electronic devices) connectable to the Internet, such as a mobile phone, a smart phone, a personal digital assistant (PDA), or a notebook computer.

According to an embodiment, the user terminal 100 may receive the user's utterance as a user input. The user terminal 100 may receive the user's utterance and generate a command for operating the app based on the user's utterance. Accordingly, the user terminal 100 may operate the app by using the command.

The intelligent server 200 may receive a user voice input from the user terminal 100 through a communication network and change it into text data. In another embodiment, the intelligent server 200 may generate (or select) a path rule based on the text data. The pass rule may include information about an action (or an operation) for performing the function of the app or information about a parameter required to execute the action. Also, the pass rule may include an order of the operation of the app. The user terminal 100 may receive the pass rule, select an app according to the pass rule, and execute an operation included in the pass rule in the selected app.

For example, the user terminal 100 may execute the operation and display a screen corresponding to the state of the user terminal 100 that has executed the operation on the display. For another example, the user terminal 100 may execute the operation and not display the result of performing the operation on the display. The user terminal 100 may, for example, execute a plurality of operations and display only some results of the plurality of operations on a display. The user terminal 100 may, for example, display only the result of executing the last-order operation on the display. As another example, the user terminal 100 may receive a user input and display a result of executing the operation on the display.

The personalized information server 300 may include a database in which user information is stored. For example, the personalization information server 300 may receive user information (eg, context information, app execution, etc.) from the user terminal 100 and store it in the database. The intelligent server 200 may be used when receiving the user information from the personalization information server 300 through a communication network and generating a pass rule for user input. According to an embodiment, the user terminal 100 may receive user information from the personalization information server 300 through a communication network and use it as information for managing a database.

The proposal server 400 may include a database in which information about a function or an application to be introduced or provided in the terminal is stored. For example, the proposal server 400 may include a database for functions that the user can use by receiving user information of the user terminal 100 from the personalization information server 300 . The user terminal 100 may receive information on the to-be provided function from the proposal server 400 through a communication network and provide the information to the user.

2 is a block diagram illustrating a user terminal of an integrated intelligent system according to an embodiment of the present invention.

Referring to FIG. 2 , the user terminal 100 may include an input module 110 , a display 120 , a speaker 130 , a memory 140 , or a processor 150 . The user terminal 100 may further include a housing, and components of the user terminal 100 may be seated inside the housing or located on the housing.

According to an embodiment, the input module 110 may receive a user input from a user. For example, the input module 110 may receive a user input from a connected external device (eg, a keyboard or a headset). As another example, the input module 110 may include a touch screen (eg, a touch screen display) coupled to the display 120 . As another example, the input module 110 may include a hardware key (or a physical key) located in the user terminal 100 (or the housing of the user terminal 100 ).

According to an embodiment, the input module 110 may include a microphone (eg, the microphone 111 of FIG. 3 ) capable of receiving the user's utterance as a voice signal. For example, the input module 110 may include a speech input system, and may receive a user's speech as a voice signal through the speech input system.

According to an embodiment, the display 120 may display an image or video, and/or an execution screen of an application. For example, the display 120 may display a graphic user interface (GUI) of the app.

According to an embodiment, the speaker 130 may output a voice signal. For example, the speaker 130 may output a voice signal generated inside the user terminal 100 to the outside.

According to an embodiment, the user terminal 100 may further include a wireless communication circuit (not shown). The wireless communication circuit may support various communication, and may support establishment of a communication channel between the user terminal 100 and the intelligent server 200 and performing wireless communication through the established communication channel.

According to an embodiment, the memory 140 may store a plurality of apps 141 and 143 . The plurality of apps 141 and 143 stored in the memory 140 may be selected, executed, and operated according to a user input.

According to an embodiment, the memory 140 may include a database capable of storing information necessary for recognizing a user input. For example, the memory 140 may include a log database capable of storing log information. As another example, the memory 140 may include a persona database that may store user information.

According to an embodiment, the memory 140 may store a plurality of apps 141 and 143 , and the plurality of apps 141 and 143 may be loaded and operated. For example, the plurality of apps 141 and 143 stored in the memory 140 may be loaded and operated by the execution manager module 153 of the processor 150 . The plurality of apps 141 and 143 may include execution services 141a and 143a that perform a function or a plurality of operations (or unit operations) 141b and 143b. The execution services 141a and 143a are generated by the execution manager module 153 of the processor 150 and may execute a plurality of operations 141b and 143b.

According to an embodiment, when the operations 141b and 143b of the apps 141 and 143 are executed, an execution status screen according to the execution of the operations 141b and 143b may be displayed on the display 120 . The execution status screen may be, for example, a screen in which operations 141b and 143b are completed. The execution status screen may be, for example, a screen of a partial landing in which the execution of the operations 141b and 143b is stopped (eg, when parameters required for the operations 141b and 143b are not input). .

According to an embodiment, the execution services 141a and 143a may execute the operations 141b and 143b according to a pass rule. For example, the execution services 141a and 143a are activated by the execution manager module 153, receive an execution request from the execution manager module 153 according to the pass rule, and perform an operation 141b, 143b) may execute the operations of the apps 141 and 143 . The execution services 141a and 143a may transmit completion information to the execution manager module 153 upon completion of the operations 141b and 143b.

According to an embodiment, when a plurality of operations 141b and 143b are executed in the apps 141 and 143, the plurality of operations 141b and 143b may be sequentially executed. When the execution of one operation (operation 1) is completed, the execution services 141a and 143a may open the next operation (operation 2) and transmit completion information to the execution manager module 153 . Here, opening an arbitrary operation may be understood as transitioning the arbitrary operation to an executable state or preparing to execute the arbitrary operation. In other words, if an operation is not opened, the operation cannot be executed. When the completion information is received, the execution manager module 153 may transmit an execution request for the next operation 141b and 143b to the execution service (eg, operation 2). According to an embodiment, when a plurality of apps 141 and 143 are executed, the plurality of apps 141 and 143 may be sequentially executed. For example, when the execution of the last operation of the first app 141 is completed and completion information is received, the execution manager module 153 transmits the execution request of the first operation of the second app 143 to the execution service 143a can send

According to an embodiment, when a plurality of operations 141b and 143b are executed in the apps 141 and 143 , a result screen according to each execution of the plurality of operations 141b and 143b is displayed on the display 120 . can be displayed. According to an embodiment, only a part of a plurality of result screens according to the execution of the plurality of operations 141b and 143b may be displayed on the display 120 .

According to an embodiment, the memory 140 may store an intelligent app (eg, a voice recognition app) interlocked with the intelligent agent 151 . The app linked with the intelligent agent 151 may receive and process the user's utterance as a voice signal. According to an embodiment, the app linked with the intelligent agent 151 may be operated by a specific input input through the input module 110 (eg, input through a hardware key, input through a touch screen, specific voice input). can

According to an embodiment, the processor 150 may control the overall operation of the user terminal 100 . For example, the processor 150 may receive a user input by controlling the input module 110 . The processor 150 may control the display 120 to display an image. The processor 150 may control the speaker 130 to output a voice signal. The processor 150 may control the memory 140 to retrieve or store necessary information.

According to an embodiment, the processor 150 may include an intelligent agent 151 , an execution manager module 153 , or an intelligent service module 155 . In an embodiment, the processor 150 may execute the instructions stored in the memory 140 to drive the intelligent agent 151 , the execution manager module 153 , or the intelligent service module 155 . Several modules mentioned in various embodiments of the present invention may be implemented as hardware or software. In various embodiments of the present disclosure, an operation performed by the intelligent agent 151 , the execution manager module 153 , or the intelligent service module 155 may be understood as an operation performed by the processor 150 .

According to an embodiment, the intelligent agent 151 may generate a command for operating an app based on a voice signal received as a user input. According to an embodiment, the execution manager module 153 may receive the generated command from the intelligent agent 151 , select, execute, and operate the apps 141 and 143 stored in the memory 140 . According to an embodiment, the intelligent service module 155 may manage user information and use it to process a user input.

The intelligent agent 151 may transmit the user input received through the input module 110 to the intelligent server 200 for processing.

According to an embodiment, the intelligent agent 151 may pre-process the user input before transmitting the user input to the intelligent server 200 . According to an embodiment, the intelligent agent 151 may pre-process the user input, an adaptive echo canceller (AEC) module, a noise suppression (NS) module, and an end-point detection (end-point) module. detection) (EPD) module or automatic gain control (AGC) module. The adaptive echo canceller may cancel an echo included in the user input. The noise suppression module may suppress background noise included in the user input. The endpoint detection module may detect an endpoint of the user's voice included in the user input to find a portion where the user's voice is present. The automatic gain control module may adjust the volume of the user input to be suitable for recognizing and processing the user input. According to one embodiment, the intelligent agent 151 may include all of the pre-processing components for performance, but in another embodiment, the intelligent agent 151 may include some of the pre-processing components to operate with low power. .

According to an embodiment, the intelligent agent 151 may include a wake-up recognition module for recognizing a user's call. The wake-up recognition module may recognize the user's wake-up command through the voice recognition module, and upon receiving the wake-up command, may activate the intelligent agent 151 to receive a user input. According to an embodiment, the wake-up recognition module of the intelligent agent 151 may be implemented in a low-power processor (eg, a processor included in an audio codec). According to an embodiment, the intelligent agent 151 may be activated according to a user input through a hardware key. When the intelligent agent 151 is activated, an intelligent app (eg, a voice recognition app) interlocked with the intelligent agent 151 may be executed.

According to an embodiment, the intelligent agent 151 may include a voice recognition module for executing a user input. The voice recognition module may recognize a user input for executing an action in the app. For example, the voice recognition module includes a limited user (voice) input that executes an action such as the wake-up command in the apps 141 and 143 (eg, a “click” that executes a shooting action when the camera app is running) the same utterance) can be recognized. The voice recognition module that assists the intelligent server 200 to recognize a user input, for example, can recognize and quickly process a user command that can be processed in the user terminal 100 . According to an embodiment, the voice recognition module for executing the user input of the intelligent agent 151 may be implemented in the app processor.

According to an embodiment, the voice recognition module (including the voice recognition module of the wake-up module) of the intelligent agent 151 may recognize a user input using an algorithm for recognizing a voice. The algorithm used to recognize the voice may be, for example, at least one of a hidden markov model (HMM) algorithm, an artificial neural network (ANN) algorithm, or a dynamic time warping (DTW) algorithm.

According to an embodiment, the intelligent agent 151 may convert the user's voice input into text data. According to an embodiment, the intelligent agent 151 may transmit the user's voice to the intelligent server 200 to receive the converted text data. Accordingly, the intelligent agent 151 may display the text data on the display 120 .

According to an embodiment, the intelligent agent 151 may receive the pass rule transmitted from the intelligent server 200 . According to an embodiment, the intelligent agent 151 may transmit the pass rule to the execution manager module 153 .

According to an embodiment, the intelligent agent 151 transmits the execution result log according to the pass rule received from the intelligent server 200 to the intelligent service module 155, and the transmitted execution result The log may be accumulated and managed in the user's preference information of the persona manager 155b.

According to an embodiment, the execution manager module 153 receives the pass rule from the intelligent agent 151 and executes the apps 141 and 143 , and the apps 141 and 143 include an operation 141b in the pass rule. , 143b) can be executed. For example, the execution manager module 153 may transmit command information for executing the actions 141b and 143b to the apps 141 and 143 , and the actions 141b and 143b from the apps 141 and 143 . of completion information can be received.

According to an embodiment, the execution manager module 153 transmits and receives command information for executing the operations 141b and 143b of the apps 141 and 143 between the intelligent agent 151 and the apps 141 and 143. can The execution manager module 153 binds the apps 141 and 143 to be executed according to the pass rule, and transmits command information of the operations 141b and 143b included in the pass rule to the apps 141 and 143 . can do. For example, the execution manager module 153 sequentially transmits the actions 141b and 143b included in the pass rule to the apps 141 and 143, and performs the actions 141b and 143b of the apps 141 and 143. They may be sequentially executed according to the pass rule.

According to an embodiment, the execution manager module 153 may manage the execution state of the operations 141b and 143b of the apps 141 and 143 . For example, the execution manager module 153 may receive information on the execution state of the operations 141b and 143b from the apps 141 and 143 . When the execution state of the operations 141b and 143b is, for example, a partial landing (eg, a parameter required for the operations 141b and 143b is not input), the execution manager module 153 may transmit information about the stopped state to the intelligent agent 151 . The intelligent agent 151 may request the user to input necessary information (eg, parameter information) by using the received information. When the execution state of the operations 141b and 143b is another, for example, an operation state, a utterance may be received from the user, and the execution manager module 153 may include the running apps 141 and 143 and the apps. Information on the execution state of (141, 143) may be transmitted to the intelligent agent (151). The intelligent agent 151 may receive the parameter information of the user's utterance through the intelligent server 200 , and may transmit the received parameter information to the execution manager module 153 . The execution manager module 153 may change the parameters of the operations 141b and 143b into new parameters by using the received parameter information.

According to an embodiment, the execution manager module 153 may transmit parameter information included in the pass rule to the apps 141 and 143 . When the plurality of apps 141 and 143 are sequentially executed according to the pass rule, the execution manager module 153 may transfer parameter information included in the pass rule from one app to another app.

According to an embodiment, the execution manager module 153 may receive a plurality of pass rules. The execution manager module 153 may select a plurality of pass rules based on the user's utterance. For example, when the user's utterance specifies some apps 141 to execute the partial operation 141a, but does not specify another app 143 to execute the remaining operation 143b, the execution manager module 153 may The same app 141 (eg, a gallery app) to execute the action 141a is executed, and a different app 143 (eg, the Messages app, the Telegram app) that can execute the remaining actions 143b is executed respectively. A plurality of different pass rules may be received. The execution manager module 153 may, for example, execute identical operations 141b and 143b (eg, consecutive identical operations 141b and 143b) of the plurality of pass rules. When the same operation is executed, the execution manager module 153 may display a status screen on the display 120 for selecting different apps 141 and 143 included in the plurality of pass rules, respectively.

According to an embodiment, the intelligent service module 155 may include a context module 155a, a persona module 155b, or a proposal module 155c.

The context module 155a may collect current states of the apps 141 and 143 from the apps 141 and 143 . For example, the context module 155a may collect the current state of the apps 141 and 143 by receiving context information indicating the current state of the apps 141 and 143 .

The persona module 155b may manage personal information of a user who uses the user terminal 100 . For example, the persona module 155b may collect usage information and performance results of the user terminal 100 to manage the user's personal information.

The suggestion module 155c may predict the user's intention and recommend a command to the user. For example, the suggestion module 155c may recommend a command to the user in consideration of the user's current state (eg, time, place, situation, and app).

3 is a diagram illustrating running an intelligent app of a user terminal according to an embodiment of the present invention.

Referring to FIG. 3 , the user terminal 100 receives a user input and executes an intelligent app (eg, a voice recognition app) interlocked with the intelligent agent 151 .

According to an embodiment, the user terminal 100 may run an intelligent app for recognizing a voice through the hardware key 112 . For example, when receiving a user input through the hardware key 112 , the user terminal 100 may display a user interface (UI) 121 of the intelligent app on the display 120 . The user may, for example, touch the voice recognition button 121a on the UI 121 of the intelligent app to input a voice 113b while the UI 121 of the intelligent app is displayed on the display 120. . As another example, the user may input the voice 113b by continuously pressing the hardware key 112 to input the voice 113b.

According to an embodiment, the user terminal 100 may execute an intelligent app for recognizing a voice through the microphone 111 . For example, the user terminal 100 displays the UI 121 of the intelligent app on the display 120 when a specified voice (eg, wake up!) is input 113a through the microphone 111 . can do.

4 is a block diagram illustrating an intelligent server of an integrated intelligent system according to an embodiment of the present invention.

Referring to FIG. 4 , the intelligent server 200 includes an automatic speech recognition (ASR) module 210 , a natural language understanding (NLU) module 220 , and a path planner module. 230 , a dialogue manager (DM) module 240 , a natural language generator (NLG) module 250 or a text to speech (TTS) module 260 . can do.

The natural language understanding module 220 or the path planner module 230 of the intelligent server 200 may generate a path rule.

According to an embodiment, the automatic speech recognition (ASR) module 210 may convert a user input received from the user terminal 100 into text data.

According to an embodiment, the automatic voice recognition module 210 may convert a user input received from the user terminal 100 into text data. For example, the automatic speech recognition module 210 may include a speech recognition module. The speech recognition module may include an acoustic model and a language model. For example, the acoustic model may include information related to vocalization, and the language model may include information about unit phoneme information and a combination of unit phoneme information. The speech recognition module may convert the user's speech into text data by using speech related information and information on unit phoneme information. Information on the acoustic model and the language model may be stored in, for example, an automatic speech recognition database (ASR DB) 211 .

According to an embodiment, the natural language understanding module 220 may determine the user's intention by performing syntactic analysis or semantic analysis. The grammatical analysis may divide the user input into grammatical units (eg, words, phrases, morphemes, etc.), and determine which grammatical elements the divided units have. The semantic analysis may be performed using semantic matching, rule matching, formula matching, and the like. Accordingly, the natural language understanding module 220 may obtain a domain, an intent, or a parameter (or a slot) necessary for expressing the intention in which the user input is intended.

According to an embodiment, the natural language understanding module 220 uses a matching rule divided into a domain, an intent, and a parameter (or a slot) necessary to identify the intent. can determine the intent and parameters of For example, the one domain (eg, alarm) may include a plurality of intents (eg, set an alarm, clear an alarm, etc.), and one intent may include a plurality of parameters (eg, time, number of repetitions, alarm tone, etc.). etc.) may be included. The plurality of rules may include, for example, one or more essential element parameters. The matching rule may be stored in a natural language understanding database (NLU DB) 221 .

According to an embodiment, the natural language understanding module 220 determines the meaning of a word extracted from a user input using linguistic features (eg, grammatical elements) such as morphemes and phrases, and converts the meaning of the identified word into a domain. and matching the intention to determine the user's intention. For example, the natural language understanding module 220 may determine the user intent by calculating how many words extracted from the user input are included in each domain and intent. According to an embodiment, the natural language understanding module 220 may determine a parameter of a user input by using a word that is a basis for recognizing the intention. According to an embodiment, the natural language understanding module 220 may determine the user's intention by using the natural language recognition database 221 in which linguistic characteristics for recognizing the intention of the user input are stored. According to another embodiment, the natural language understanding module 220 may determine the user's intention by using a personal language model (PLM). For example, the natural language understanding module 220 may determine the user's intention by using personalized information (eg, a contact list, a music list). The personalized language model may be stored in, for example, the natural language recognition database 221 . According to an embodiment, not only the natural language understanding module 220 but also the automatic speech recognition module 210 may recognize the user's voice by referring to the personalized language model stored in the natural language recognition database 221 .

According to an embodiment, the natural language understanding module 220 may generate a pass rule based on an intention and parameters of a user input. For example, the natural language understanding module 220 may select an app to be executed based on the intention of the user input and determine an operation to be performed in the selected app. The box salmon understanding module 220 may generate a pass rule by determining a parameter corresponding to the determined operation. According to an embodiment, the pass rule generated by the natural language understanding module 220 may include information about an app to be executed, an action to be executed in the app, and parameters necessary for executing the action.

According to an embodiment, the natural language understanding module 220 may generate one pass rule or a plurality of pass rules based on the intention and parameters of the user input. For example, the natural language understanding module 220 receives a pass rule set corresponding to the user terminal 100 from the pass planner module 230, maps intent and parameters of a user input to the received pass rule set, and passes You can decide the rules.

According to another embodiment, the natural language understanding module 220 determines one pass rule or a plurality of passes by determining an app to be executed, an action to be executed in the app, and parameters required to execute the action based on the intention and parameters of the user input. You can create rules. For example, the natural language understanding module 220 uses the information of the user terminal 100 to convert the to-be-executed app and the operation to be executed in the app in the form of an ontology or a graph model according to the intention of the user input. You can create pass rules by arranging them. The generated path rule may be stored in, for example, a path rule database (PR DB) 231 through the path planner module 230 . The generated pass rule may be added to a pass rule set of the database 231 .

According to an embodiment, the natural language understanding module 220 may select at least one pass rule from among a plurality of generated pass rules. For example, the natural language understanding module 220 may select an optimal pass rule for the plurality of pass rules. As another example, the natural language understanding module 220 may select a plurality of pass rules when only some actions are specified based on the user's utterance. The natural language understanding module 220 may determine one pass rule from among the plurality of pass rules according to an additional input from the user.

According to an embodiment, the natural language understanding module 220 may transmit a pass rule to the user terminal 100 as a request for a user input. For example, the natural language understanding module 220 may transmit one pass rule corresponding to the user input to the user terminal 100 . As another example, the natural language understanding module 220 may transmit a plurality of pass rules corresponding to the user input to the user terminal 100 . The plurality of pass rules may be generated by the natural language understanding module 220 when, for example, only some actions are specified based on a user's utterance.

According to an embodiment, the path planner module 230 may select at least one pass rule from among a plurality of pass rules.

According to an embodiment, the path planner module 230 may transmit a path rule set including a plurality of path rules to the natural language understanding module 220 . The plurality of pass rules of the pass rule set may be stored in the form of a table in the pass rule database 231 connected to the pass planner module 230 . For example, the pass planner module 230 transmits a pass rule set corresponding to information (eg, OS information, app information) of the user terminal 100 received from the intelligent agent 151 to the natural language understanding module 220 . can The table stored in the pass rule database 231 may be stored for each domain or domain version, for example.

According to an embodiment, the path planner module 230 may select one pass rule or a plurality of pass rules from the pass rule set and transmit it to the natural language understanding module 220 . For example, the path planner module 230 selects one pass rule or a plurality of pass rules by matching the user's intention and parameters with a pass rule set corresponding to the user terminal 100 to perform the natural language understanding module 220 can be transmitted as

According to an embodiment, the path planner module 230 may generate one pass rule or a plurality of pass rules by using user intention and parameters. For example, the path planner module 230 may generate one pass rule or a plurality of pass rules by determining an app to be executed and an action to be executed in the app based on user intention and parameters. According to an embodiment, the pass planner module 230 may store the generated pass rule in the pass rule database 231 .

According to an embodiment, the pass planner module 230 may store the pass rule generated by the natural language understanding module 220 in the pass rule database 231 . The generated pass rule may be added to a pass rule set stored in the pass rule database 231 .

According to an embodiment, the table stored in the pass rule database 231 may include a plurality of pass rules or a plurality of pass rule sets. The plurality of pass rules or the plurality of pass rule sets may reflect the type, version, type, or characteristic of a device performing each pass rule.

According to an embodiment, the conversation manager module 240 may determine whether the intention of the user identified by the natural language understanding module 220 is clear. For example, the conversation manager module 240 may determine whether the user's intention is clear based on whether the parameter information is insufficient. The conversation manager module 240 may determine whether the parameter identified by the natural language understanding module 220 is sufficient to perform the task. According to an embodiment, when the user's intention is not clear, the conversation manager module 240 may perform a feedback requesting information necessary from the user. For example, the conversation manager module 240 may perform a feedback requesting information about a parameter for identifying the user's intention.

According to an embodiment, the conversation manager module 240 may include a content provider module. When the content providing module can perform an operation based on the intention and parameters identified by the natural language understanding module 220, the content providing module may generate a result of performing the task corresponding to the user input. According to an embodiment, the conversation manager module 240 may transmit the result generated by the content providing module to the user terminal 100 in response to a user input.

According to an embodiment, the natural language generating module (NLG) 250 may change the specified information into a text form. The information changed to the text form may be in the form of natural language utterance. The specified information may be, for example, information on additional input, information guiding completion of an operation corresponding to the user input, or information guiding the user's additional input (eg, feedback information on user input). The information changed in the text format may be transmitted to the user terminal 100 and displayed on the display 120 , or transmitted to the text-to-speech conversion module 260 and changed into a voice format.

According to an embodiment, the text-to-speech conversion module 260 may change information in a text format into information in a voice format. The text-to-speech conversion module 260 may receive text-type information from the natural language generating module 250 , convert the text-type information into speech-type information, and transmit it to the user terminal 100 . The user terminal 100 may output the information in the form of voice to the speaker 130 .

According to an embodiment, the natural language understanding module 220 , the path planner module 230 , and the conversation manager module 240 may be implemented as one module. For example, the natural language understanding module 220 , the path planner module 230 , and the dialog manager module 240 are implemented as one module to determine the user's intention and parameters, and respond to the determined user's intentions and parameters. It can generate responses (eg pass rules). Accordingly, the generated response may be transmitted to the user terminal 100 .

5 is a diagram illustrating a method of generating a path rule of a path planner module according to an embodiment of the present invention.

Referring to FIG. 5 , according to an embodiment, the natural language understanding module 220 may classify the function of the app into unit operations A to F and store it in the pass rule database 231 . For example, the natural language understanding module 220 performs a plurality of pass rules (A-B1-C1, A-B1-C2, A-B1-C3-D-F, A-B1-C3-D-E-F) divided into unit operations. The included pass rule set may be stored in the pass rule database 231 .

According to an embodiment, the pass rule database 231 of the pass planner module 230 may store a pass rule set for performing an app function. The pass rule set may include a plurality of pass rules including a plurality of operations. In the plurality of pass rules, operations executed according to parameters input to each of the plurality of operations may be sequentially arranged. According to an embodiment, the plurality of pass rules may be configured in the form of an ontology or a graph model and stored in the pass rule database 231 .

According to an embodiment, the natural language understanding module 220 may configure the plurality of pass rules (A-B1-C1, A-B1-C2, A-B1-C3-D-F, A-) corresponding to the intention and parameter of the user input. B1-C3-D-E-F), the optimal pass rule (A-B1-C3-D-F) can be selected.

According to an embodiment, when there is no pass rule that perfectly matches the user input, the natural language understanding module 220 may transmit a plurality of rules to the user terminal 100 . For example, the natural language understanding module 220 may select a pass rule (eg, A-B1) partially corresponding to the user input. The natural language understanding module 220 may include one or more pass rules (eg, A-B1-C1, A-B1-C2, A-B1-C3) including a pass rule (eg, A-B1) partially corresponding to the user input. -D-F, A-B1-C3-D-E-F) may be selected and transmitted to the user terminal 100 .

According to an embodiment, the natural language understanding module 220 may select one of a plurality of pass rules based on the additional input of the user terminal 100 and transmit the selected one pass rule to the user terminal 100 . have. For example, the natural language understanding module 220 may configure a plurality of pass rules (eg, A-B1-C1, A-B1) according to a user input additionally input from the user terminal 100 (eg, an input for selecting C3). -C2, A-B1-C3-D-F, A-B1-C3-D-E-F) may be selected and transmitted to the user terminal 100 by selecting one pass rule (eg, A-B1-C3-D-F).

According to another embodiment, the natural language understanding module 220 may determine a user's intention and parameters corresponding to a user input additionally input to the user terminal 100 (eg, an input for selecting C3), and the determined The user's intention or parameters may be transmitted to the user terminal 100 . Based on the transmitted intention or the parameter, the user terminal 100 configures a plurality of pass rules (eg, A-B1-C1, A-B1-C2, A-B1-C3-D-F, A-B1-C3- D-E-F), one pass rule (eg A-B1-C3-D-F) can be selected.

Accordingly, the user terminal 100 may complete the operation of the apps 141 and 143 according to the selected one pass rule.

According to an embodiment, when a user input lacking information is received by the intelligent server 200 , the natural language understanding module 220 may generate a pass rule partially corresponding to the received user input. For example, the natural language understanding module 220 may transmit the partially corresponding pass rule to the intelligent agent 151 (①). The intelligent agent 151 may transmit the partially corresponding pass rule to the execution manager module 153 (②), and the execution manager module 153 may execute the first app 141 according to the pass rule. . The execution manager module 153 may transmit (③) information on insufficient parameters to the intelligent agent 151 while executing the first app 141 . The intelligent agent 151 may request additional input from the user by using the information on the insufficient parameters. The intelligent agent 151 may transmit to the intelligent server 200 for processing when an additional input is received by the user (④). The natural language understanding module 220 may generate an added pass rule based on the intention and parameter information of the additionally input user input and transmit it to the intelligent agent 151 (⑤). The intelligent agent 151 may execute the second app 143 by transmitting (⑥) the pass rule to the execution manager module 153 .

According to an embodiment, the natural language understanding module 220 may transmit a user information request to the personalization information server 300 when a user input in which some information is omitted is received by the intelligent server 200 . The personalized information server 300 may transmit information of a user who has input a user input stored in the persona database to the natural language understanding module 220 . The natural language understanding module 220 may select a pass rule corresponding to a user input in which a partial operation is omitted by using the user information. Accordingly, the natural language understanding module 220 receives an additional input by requesting the missing information even if a user input in which some information is missing is received by the intelligent server 200, or a pass rule corresponding to the user input by using the user information. can be decided

6 is a diagram illustrating that a context module of a processor collects a current state according to an embodiment of the present invention.

Referring to FIG. 6 , the context module 155a requests context information indicating the current state of the apps 141 and 143 to the apps 141 and 143 when receiving (①) a context request from the intelligent agent 151 ( ②) can be done. According to an embodiment, the context module 155a may receive (③) the context information from the apps 141 and 143 and transmit it to the intelligent agent 151 (④).

According to an embodiment, the context module 155a may receive a plurality of context information through the apps 141 and 143 . For example, the context information may be information on the most recently executed apps 141 and 143 . As another example, the context information may be information about a current state within the apps 141 and 143 (eg, information about a corresponding photo when viewing a photo in a gallery).

According to an embodiment, the context module 155a may receive context information indicating the current state of the user terminal 100 from not only the apps 141 and 143 but also the device platform. The context information may include general context information, user context information, or device context information.

The general context information may include general information of the user terminal 100 . The general context information may be confirmed through an internal algorithm by receiving data through a sensor hub of a device platform or the like. For example, the general context information may include information about a current space-time. The information on the current space-time may include, for example, information on the current time or the current location of the user terminal 100 . The current time may be confirmed through time on the user terminal 100 , and information on the current location may be identified through a global positioning system (GPS). As another example, the general context information may include information on physical movement. The information on the physical movement may include, for example, information about walking, running, driving, and the like. The physical motion information may be identified through a motion sensor. The driving information may not only confirm driving through the motion sensor, but also check boarding and parking by detecting a Bluetooth connection in the vehicle. As another example, the general context information may include user activity information. The user activity information may include, for example, information on commuting, shopping, travel, and the like. The user activity information may be confirmed using information about a place registered in a database by a user or an app.

The user context information may include information about a user. For example, the user context information may include information about the user's emotional state. The information on the emotional state may include, for example, information about the user's happiness, sadness, anger, and the like. As another example, the user context information may include information about the user's current state. The information on the current state may include, for example, information on interests, intentions, etc. (eg, shopping).

The device context information may include information on the state of the user terminal 100 . For example, the device context information may include information on a pass rule executed by the execution manager module 153 . As another example, the device information may include information about a battery. The information on the battery may be confirmed through, for example, charging and discharging states of the battery. As another example, the device information may include information on connected devices and networks. Information on the connected device may be confirmed through, for example, a communication interface to which the device is connected.

7 is a diagram illustrating that a persona module manages user information according to an embodiment of the present invention.

Referring to FIG. 7 , the persona module 155b may receive information of the user terminal 100 from the apps 141 and 143 , the execution manager module 153 , or the context module 155a. The apps 141 and 143 and the execution manager module 153 may store result information of executing the actions 141b and 143b of the app in the action log database. The context module 155a may store information on the current state of the user terminal 100 in the context database. The persona module 155b may receive the stored information from the action log database or the context database. Data stored in the operation log database and the context database may be analyzed by, for example, an analysis engine and transmitted to the persona module 155b.

According to an embodiment, the persona module 155b may transmit information received from the apps 141 and 143 , the execution manager module 153 or the context module 155a to the suggestion module 155c. For example, the persona module 155b may transmit data stored in the operation log database or the context database to the proposal module 155c.

According to an embodiment, the persona module 155b may transmit information received from the apps 141 and 143 , the execution manager module 153 , or the context module 155a to the personalization information server 300 . For example, the persona module 155b may periodically transmit data accumulated and stored in the action log database or the context database to the personalized information server 300 .

According to an embodiment, the persona module 155b may transmit data stored in the action log database or the context database to the proposal module 155c. User information generated by the persona module 155b may be stored in a persona database. The persona module 155b may periodically transmit user information stored in the persona database to the personalization information server 300 . According to an embodiment, information transmitted to the personalized information server 300 by the persona module 155b may be stored in a persona database. The personalized information server 300 may infer user information necessary for generating a pass rule of the intelligent server 200 by using the information stored in the persona database.

According to an embodiment, the user information inferred using the information transmitted by the persona module 155b may include profile information or preference information. The profile information or preference information may be inferred through the user's account and accumulated information.

The profile information may include personal information of the user. For example, the profile information may include demographic information of the user. The demographic information may include, for example, a user's gender and age. As another example, the profile information may include life event information. The life event information may be inferred by comparing log information with a life event model, for example, and reinforced by analyzing a behavior pattern. As another example, the profile information may include interest information. The interest information may include, for example, a shopping item of interest, a field of interest (eg, sports, politics, etc.). As another example, the profile information may include activity area information. The activity area information may include, for example, information on a home, a place of work, and the like. The information on the activity area may include information on the area in which priority is recorded based on the accumulated staying time and the number of visits as well as information on the location of the place. As another example, the profile information may include active time information. The activity time information may include, for example, information on waking up time, commuting time, sleep time, and the like. The information on the commuting time may be inferred using the activity area information (eg, information about home and work place). The information on the sleep time may be inferred through the unused time of the user terminal 100 .

The preference information may include user preference information. For example, the preference information may include information about app preference. The app preference may be inferred through, for example, a usage record of the app (eg, usage record by time and location). The preference of the app may be used to determine an app to be executed according to the user's current state (eg, time, place). As another example, the preference information may include information about contact preference. The contact preference may be inferred, for example, by analyzing contact frequency (eg, contact frequency by time and place) information of the contact. The contact preference may be used to determine a contact to contact according to the user's current status (eg, contact for a duplicate name). As another example, the preference information may include setting information. The setting information may be inferred, for example, by analyzing information on a setting frequency of a specific setting value (eg, a frequency of setting a specific setting value for each time and place). The setting information may be used to set a specific setting value according to the user's current state (eg, time, place, situation). As another example, the preference information may include place preference. The place preference may be inferred, for example, through a visit record of a specific place (eg, a visit record by time). The place preference may be used to determine a place being visited according to the user's current state (eg, time). As another example, the preference information may include a command preference. The command preference may be inferred through, for example, command usage frequency (eg, usage frequency by time and location). The command preference may be used to determine a command pattern to be used according to the user's current status (eg, time, place). In particular, the command preference may include information on the menu most selected by the user in the current state of the running app by analyzing log information.

8 is a flowchart illustrating a method of performing a task based on a user utterance request according to an embodiment.

Hereinafter, it is assumed that the user terminal (or electronic device) 100 of FIG. 2 performs the process of FIG. 8 . Also, in the description of FIG. 8 , an operation described as being performed by the electronic device may be understood as being controlled by the processor 150 of the electronic device 100 . It may be implemented as instructions (instructions) that may be executed (or executed) by the processor 150 of the electronic device 100 . The instructions may be stored in, for example, a computer recording medium or the memory 140 of the electronic device 100 illustrated in FIG. 2 .

Referring to FIG. 8 , in operation 801 , the processor 150 may receive a user's utterance through the microphone 111 .

According to an embodiment, the user's utterance may include a request for performing a task with the electronic device 100 . For example, the user utterance may be 'Send a picture of Hawaii to the mother using the message app', and the user utterance includes a request to perform a task of sending a picture taken in Hawaii to the mother using the message app.

In operation 802 , the processor 150 may transmit data related to the user's utterance to an external server through a wireless communication circuit.

According to an embodiment, the data related to the user's utterance may include a voice signal corresponding to the user's utterance.

According to an embodiment, the processor 150 may transmit context information indicating the current state of the electronic device to an external server through a wireless communication circuit. The context information may include, for example, information about the running app or state information of the running app. In an embodiment, the state information of the app may include a state ID, specific screen information of the app, or specific state information on a specific screen.

In operation 803 , the processor 150 may receive a response from an external server through a wireless communication circuit.

According to an embodiment, the received response includes at least one sample utterance indicating an intention of the user utterance, and the at least one sample utterance is selected by an external server (eg, the intelligent server 200) based on the user utterance. it could be

According to an embodiment, the at least one sample utterance may include a first sample utterance indicating a first intention of the user utterance and a second sample utterance indicating a second intention different from the first intention. For example, the received response may include a plurality of sample utterances, and each sample utterance may indicate an intention of a different user utterance.

In an embodiment, the external server may generate at least one pass rule based on data related to a user's utterance. According to an embodiment, the at least one sample utterance may include a first sample utterance corresponding to a first pass rule and a second sample utterance corresponding to a second pass rule. For example, the received response may include a plurality of sample utterances, and each sample utterance may correspond to a different pass rule.

In operation 804 , the processor 150 may display at least one sample utterance on the touch screen display 120 . A method for the processor 150 to display a sample utterance will be described in detail below.

In operation 805 , the processor 150 may receive a first user input for selecting one of at least one sample utterance through the microphone 111 or the touch screen display 120 . A method for the processor 150 to receive the first user input will be described in detail below.

In operation 806, the processor 150 may perform a task related to the selected sample utterance.

According to an embodiment, the processor 150 may perform the task by causing the electronic device to have a sequence of states related to a selected one of at least one sample utterance.

In an embodiment, after operation 802, when the external server that may correspond to the intelligent server 200 clearly determines the intention of the user's utterance, the electronic device may perform a task according to the determined intention of the user's utterance. However, since the type of the user's utterance is not limited, the external server may not be able to clearly determine the intention of the user's utterance.

Hereinafter, when the external server cannot clearly determine the intention of the user's utterance, a method for the external server to generate a pass rule based on data related to the user's utterance will be described.

9 is a block diagram illustrating a natural language understanding module of the intelligent server 200 according to an embodiment.

Referring to FIG. 9 , the natural language understanding module 910 may include a domain classifier 911 , an intent classifier 912 , a slot tagger 913 , and a format handler 914 . . According to an embodiment, the natural language understanding module of FIG. 9 may correspond to the natural language understanding module 220 of FIG. 2 .

The domain classifier 911 may determine the domain to which the pass rule belongs based on at least one of text corresponding to the user's utterance and context information. According to an embodiment, the domain may correspond to an app used when a pass rule is executed.

According to an embodiment, the domain classifier 911 may use text corresponding to the user's utterance received from the automatic speech recognition module 210 . According to an embodiment, the domain classifier 911 may use text directly input by the user.

According to an embodiment, the context information may include information about an app that was executed before a user's utterance was input (or information about an app running in the background) or state information of the app. In an embodiment, the state information of the app may include a state ID, specific screen information of the app, or specific state information on a specific screen.

According to an embodiment, the domain classifier 911 may determine a plurality of domains. The domain classifier 911 may further determine a confidence level corresponding to each domain. According to an embodiment, the sum of the plurality of reliability values may be 1.

The intention classifier 912 may determine an intention in the domain determined by the domain classifier 911 based on at least one of text and context information corresponding to the user's utterance. According to an embodiment, the intention may correspond to an operation performed using an app corresponding to a domain, and the intention classifier 912 may determine a pass rule ID corresponding to the intention.

According to an embodiment, the context information may include information about an app that was executed before a user's utterance was input (or information about an app running in the background) or state information of the app.

According to an embodiment, when the domain classifier 911 determines a plurality of domains, the intention classifier 912 may determine intentions in each of the plurality of domains.

According to an embodiment, the intention classifier 912 may determine a plurality of intentions. The intent classifier 912 may further determine a confidence value corresponding to each intent. According to an embodiment, the sum of the plurality of reliability values may be 1.

The slot tagger 913 may generate a slot necessary for executing a pass rule based on the text corresponding to the user's utterance and the intention (or pass rule ID) determined by the intention classifier 912 . According to an embodiment, when the intention classifier 912 determines a plurality of intentions, the slot tagger 913 may generate slots for the plurality of intentions.

According to an embodiment, the slot tagger 913 may generate a plurality of slots for each intention. The slot tagger 913 may further determine a reliability value corresponding to each slot.

The format converter 914 may change the parameter generated by the slot tagger 913 into a format usable by the path planner module. For example, the format converter 914 may convert a slot called 'yesterday' into a data type having '-1' in a format called date.

According to an embodiment, the natural language understanding module may be composed of modules such as the domain classifier 911 , the intent classifier 912 , the slot tagger 913 , and the format converter 914 , and the modules are integrated. may consist of

The natural language understanding module may pass the pass rule ID and slot to the pass planner module. The path planner module may change a slot to a parameter of a pass rule, generate a pass rule using context information of the electronic device, and the like, and transmit the generated pass rule to the electronic device.

10 illustrates an operation included in a pass rule according to an embodiment.

In an embodiment, the pass rule generated by the intelligent server 200 using the user's utterance may include operations 1 to 5 as shown in FIG. 10 .

For example, operations 1 to 5 may be operations according to a pass rule generated based on a user utterance “find a picture of Hawaii”, and operations 1 to 5 may be operations for finding pictures taken in Hawaii using the gallery app. It may correspond to a task.

According to an embodiment, when the electronic device 100 displays a home screen, the intelligent server 200 performs operation 1 by the electronic device 100 by an input such as a touch, regardless of whether the user's utterance is received. And whether the user's utterance is received when operation 2 has already been performed, a pass rule for searching for a photo taken in Hawaii by using the gallery app may be generated.

In an embodiment, the natural language understanding module of the intelligent server 200 may further use context information as well as text corresponding to the user's utterance. According to an embodiment, the intelligent server 200 has already performed operations 1 and 2 when a user utterance is received when the electronic device 100 displays a home screen and when the electronic device 100 receives an input such as a touch. When performing, a pass rule in the case where the user utterance is received may be generated differently.

For example, when the electronic device 100 displays the home screen and a user utterance “Find Hawaii” is received, the intelligent server 200 may generate a pass rule for searching Hawaii in the Internet app.

As another example, when a user utterance “Find Hawaii” is received when the electronic device 100 displays the gallery app screen, the intelligent server 200 may generate a pass rule for finding Hawaii photos in the gallery app. At this time, the user's intention was to search for Hawaii on the Internet, but the user's utterance is received while the gallery app is running, and a pass rule for finding Hawaii pictures may be created.

The intelligent server 200 may generate a plurality of pass rules and sample utterances corresponding to the pass rules by a method to be described below in order to generate a pass rule conforming to the user's intention, and transmit the generated pass rule to the electronic device 100 . Before describing a specific method, a screen on which a plurality of sample utterances generated by the intelligent server 200 are displayed on the electronic device 100 will be described with reference to FIGS. 11A and 11B .

11A is a diagram illustrating that a user's utterance is input while a gallery app is running, according to an exemplary embodiment.

11B is a diagram illustrating a screen on which a plurality of sample utterances are displayed, according to an exemplary embodiment.

In an embodiment, the electronic device 100 may receive the user utterance 1101 “Find Hawaii” on the screen on which the gallery app is executed, as shown in FIG. 11A , and transmit it to the intelligent server 200 .

The intelligent server 200 uses the text corresponding to the user's utterance of "Find Hawaii" and context information that the executed app is the gallery app, and uses the pass rule and Internet app to find Hawaii photos using the gallery app. You can create a pass rule that searches for . The intelligent server 200 may generate a sample utterance corresponding to each pass rule and transmit it to the electronic device 100 .

In an embodiment, the electronic device 100 displays the sample utterance 1102 corresponding to the received pass rule as shown in FIG. 11B , and receives an input for selecting the sample utterance to execute the pass rule corresponding to the selected sample utterance. can

Hereinafter, a detailed method for transmitting a plurality of sample utterances by the intelligent server 200 will be described with reference to FIGS. 12 and 13 .

12 is a flowchart illustrating a method for the intelligent server 200 to transmit a plurality of sample utterances according to an embodiment.

Referring to FIG. 12 , in operation 1201 , the domain classifier 911 may determine at least one domain (or an app) based on at least one of text and context information corresponding to a user utterance. According to an embodiment, the domain classifier 911 may further determine a reliability value corresponding to each of at least one domain.

In operation 1202 , the intelligent server 200 may determine whether the reliability value of the domain having the largest reliability value exceeds a specified value.

If, as a result of operation 1202, the confidence value of the domain having the largest confidence value exceeds the specified value, in operation 1203, the intelligent server 200 converts the domain according to the user's utterance and context information to the domain having the largest confidence value. can decide

If, as a result of operation 1202, the confidence value of the domain having the largest confidence value does not exceed the specified value, in operation 1204, the intelligent server 200 determines whether the user utterance is a root utterance using context information. can judge According to an embodiment, the root utterance may mean a user utterance received by executing an intelligent service while the electronic device 100 is displaying the home screen. According to an embodiment, the context information may include information about an app that was executed before the intelligent service was executed, and the intelligent server 200 may determine whether the user's utterance is a root utterance using this context information.

If the user utterance is not a root utterance as a result of operation 1202, in operation 1205, the domain classifier 911 may determine at least one domain (or an app) based on the user utterance on the assumption that the user utterance is a root utterance. have. According to an embodiment, the domain classifier 911 may further determine a reliability value corresponding to each of at least one domain.

In operation 1206, the intention classifier 912 performs at least one intention in the first domain having the largest confidence value among the at least one domain determined in operation 1201 based on at least one of text and context information corresponding to the user's utterance; At least one intention in the second domain having the largest reliability value among the at least one domain determined in operation 1205 may be determined. According to an embodiment, the intention classifier 912 may determine a pass rule ID corresponding to the intention. According to an embodiment, the intention classifier 912 may further determine a confidence value corresponding to each of the at least one intention.

In operation 1207 , the intelligent server 200 generates a pass rule corresponding to each intention by using a specified number of intentions having a large reliability value among at least one intention (or pass rule ID) in the first domain and user utterances. can do. Also, the intelligent server 200 may generate a pass rule corresponding to each intention by using a specified number of intentions having a large reliability value among at least one intention in the second domain and user utterances. According to an embodiment, the intelligent server 200 may further generate a sample utterance corresponding to a pass rule.

For example, the intelligent server 200 may generate two pass rules corresponding to each intention by using the user utterance and the intention corresponding to the top two in the order of the reliability value among the five intentions in the first domain. can In addition, the intelligent server 200 may generate two pass rules corresponding to each intention by using the user utterance and the intention corresponding to the top two in the order of the reliability value among the four intentions in the second domain. . In other words, the intelligent server 200 may generate a total of four pass rules.

In an embodiment, the intelligent server 200 may store a representative utterance (RU) expressing the intent of each pass rule. According to an embodiment, the intelligent server 200 may further store representative sentences (representative friendly utterances (RFU)) in which the representative sentences are easily understood by the user. The intelligent server 200 may generate a sample utterance corresponding to the pass rule by using the RFU.

For example, the intelligent server 200 uses the RFU to use the text corresponding to the user's utterance, "What is the picture taken in Hawaii, will you send it to Emma through a message?" You can create a sample utterance that reads “Send me a message.”

According to an embodiment, the intelligent server 200 may generate a sample utterance by changing a parameter in the RFU into a parameter generated from text corresponding to the user utterance.

For example, the RFU stored in the intelligent server 200 may be "a photo taken in Hawaii, send a message to Emma", and the text corresponding to the user's utterance may be "a photo taken in Tokyo sent to Tommy through a message". The intelligent server 200 changes 'Hawaii' in the RFU to 'Tokyo', and changes 'Emma' to 'Tommy' to generate a sample utterance such as "Photo taken in Tokyo, send a message to Tommy."

In an embodiment, if the user utterance is a root utterance as a result of operation 1202, in operation 1208, the intention classifier 912 determines at least one in operation 1201 based on at least one of text and context information corresponding to the user utterance. It is possible to determine at least one intention in each of the specified number of domains having a large reliability value among the domains of . According to an embodiment, the intention classifier 912 may determine a pass rule ID corresponding to the intention. According to an embodiment, the intention classifier 912 may further determine a confidence value corresponding to each of the at least one intention.

For example, the intention classifier 912 may determine at least one intention in the first domain having the largest reliability value among the at least one domain determined in operation 1201, and the reliability value of the at least one domain determined in operation 1201 is At least one intent in the second largest third domain may be determined.

As another example, the intention classifier 912 may determine at least one intention in only the first domain having the largest reliability value among the at least one domain determined in operation 1201 .

In an embodiment, if the user utterance is a root utterance as a result of operation 1202 , after operation 1208 , in operation 1207 , the intelligent server 200 determines the reliability of at least one intention (or pass rule ID) in the first domain. A pass rule corresponding to each intention may be generated using a specified number of intentions having a large value and user utterances. Also, the intelligent server 200 may generate a pass rule corresponding to each intention by using a specified number of intentions having a large reliability value among at least one intention in the third domain and user utterances. According to an embodiment, the intelligent server 200 may further generate a sample utterance corresponding to a pass rule.

For example, the intelligent server 200 may generate two pass rules corresponding to each intention by using the user utterance and the intention corresponding to the top two in the order of the reliability value among the five intentions in the first domain. can In addition, the intelligent server 200 may generate two pass rules corresponding to each intention by using the user's utterance and the intention corresponding to the top two in the order of the reliability value among the three intentions in the third domain. . In other words, the intelligent server 200 may generate a total of four pass rules.

As another example, the intelligent server 200 may generate four pass rules corresponding to each intention by using the user utterance and the intention corresponding to the top four in the order of the reliability value among the five intentions in the first domain. have.

In operation 1209 , the intelligent server 200 may transmit the generated pass rule and a sample utterance corresponding to the pass rule to the electronic device 100 .

13 is a flowchart illustrating a method for the intelligent server 200 to transmit a plurality of sample utterances according to an embodiment.

Referring to FIG. 13 , in operation 1301 , the domain classifier 911 may determine at least one domain (or an app) based on at least one of text and context information corresponding to a user utterance. According to an embodiment, the domain classifier 911 may further determine a reliability value corresponding to each of at least one domain.

In operation 1302, the intention classifier 912 determines at least one intention in the first domain having the largest confidence value among the at least one domain determined in operation 1201 based on at least one of text and context information corresponding to the user's utterance. can decide According to an embodiment, the intention classifier 912 may determine a pass rule ID corresponding to the intention. According to an embodiment, the intention classifier 912 may further determine a confidence value corresponding to each of the at least one intention.

In operation 1303 , the intelligent server 200 may determine whether the confidence value of the intention having the largest confidence value exceeds a specified value.

If, as a result of operation 1303, the confidence value of the intention having the largest confidence value exceeds the specified value, in operation 1304, the intelligent server 200 converts the intention according to the user's speech and context information to the intention having the largest confidence value. can decide

If, as a result of operation 1303, the confidence value of the intention having the largest confidence value does not exceed the specified value, in operation 1305, the intelligent server 200 may determine whether the user utterance is a root utterance using context information. have. According to an embodiment, the context information may include information about an app that was executed before the intelligent service was executed, and the intelligent server 200 may determine whether the user's utterance is a root utterance using this context information.

If the user utterance is not a root utterance as a result of operation 1303, in operation 1306, the domain classifier 911 performs at least one domain (or an app) based on the user utterance on the premise that the user utterance is a user utterance in a specified state. ) can be determined. According to an embodiment, the domain classifier 911 may further determine a reliability value corresponding to each of at least one domain.

In an embodiment, the specified state is a state in which an intelligent service is executed when the electronic device 100 is displaying the home screen, and a state in which the intelligent service is executed when a screen is displayed immediately before the screen of the electronic device 100 is displayed. It may include a state or a state in which an intelligent service is executed when a designated representative screen of an app that the electronic device 100 is running is displayed.

In operation 1307, the intention classifier 912 determines at least one intention in the second domain having the largest confidence value among the at least one domain determined in operation 1306, based on at least one of text and context information corresponding to the user's utterance. can decide According to an embodiment, the intention classifier 912 may determine a pass rule ID corresponding to the intention. According to an embodiment, the intention classifier 912 may further determine a confidence value corresponding to each of the at least one intention.

In an embodiment, when the at least one domain determined in operation 1306 is the same as the at least one domain determined in operation 1301 , the intention classifier 912 uses context information corresponding to the specified state at least one domain in the second domain. can determine the intention of For example, the intention classifier 912 may set at least one in the second domain on the premise that a user's utterance is received while the intelligent service is executed when the screen is displayed immediately before the screen of the electronic device 100 is displayed. can determine the intention of

In an embodiment, if the user utterance is not a root utterance as a result of operation 1303, the intelligent server 200 does not perform operation 1306, and in operation 1307, the intention classifier 912 provides context information corresponding to the specified state. can be used to determine at least one intention in the second domain.

In operation 1308 , the intelligent server 200 generates a pass rule corresponding to each intention by using a specified number of intentions having a large reliability value among at least one intention (or pass rule ID) in the first domain and user utterances. can do. Also, the intelligent server 200 may generate a pass rule corresponding to each intention by using a specified number of intentions having a large reliability value among at least one intention in the second domain and user utterances. According to an embodiment, the intelligent server 200 may further generate a sample utterance corresponding to a pass rule.

For example, the intelligent server 200 may generate two pass rules corresponding to each intention by using the user utterance and the intention corresponding to the top two in the order of the reliability value among the five intentions in the first domain. can In addition, the intelligent server 200 may generate two pass rules corresponding to each intention by using the user utterance and the intention corresponding to the top two in the order of the reliability value among the four intentions in the second domain. . In other words, the intelligent server 200 may generate a total of four pass rules.

In an embodiment, if the user utterance is a root utterance as a result of operation 1303, in operation 1309, the intention classifier 912 determines at least one determined in operation 1301 based on at least one of text or context information corresponding to the user utterance. It is possible to determine at least one intention in each of the specified number of domains having a large reliability value among the domains of . According to an embodiment, the intention classifier 912 may determine a pass rule ID corresponding to the intention. According to an embodiment, the intention classifier 912 may further determine a confidence value corresponding to each of the at least one intention.

For example, since the intent classifier 912 determines at least one intent in the first domain in operation 1302, at least one of the at least one intent in the third domain having the second largest confidence value among the at least one domain determined in operation 1301 You can decide your intentions further.

In an embodiment, when the user utterance is a root utterance as a result of operation 1303 , after operation 1309 , in operation 1308 , the intelligent server 200 determines the reliability of at least one intention (or pass rule ID) in the first domain. A pass rule corresponding to each intention may be generated using a specified number of intentions having a large value and user utterances. Also, the intelligent server 200 may generate a pass rule corresponding to each intention by using a specified number of intentions having a large reliability value among at least one intention in the third domain and user utterances. According to an embodiment, the intelligent server 200 may further generate a sample utterance corresponding to a pass rule.

For example, the intelligent server 200 may generate two pass rules corresponding to each intention by using the user utterance and the intention corresponding to the top two in the order of the reliability value among the five intentions in the first domain. can In addition, the intelligent server 200 may generate two pass rules corresponding to each intention by using the user's utterance and the intention corresponding to the top two in the order of the reliability value among the three intentions in the third domain. . In other words, the intelligent server 200 may generate a total of four pass rules.

In operation 1310 , the intelligent server 200 may transmit the generated pass rule and a sample utterance corresponding to the pass rule to the electronic device 100 .

The above-described operations 1201 to 1209 and operations 1301 to 1310 have been described on the premise that the domain classifier 911 and the intention classifier 912 of the intelligent server 200 are separately configured. According to various embodiments, the intelligent server 200 integrally configures the domain classifier 911 and the intent classifier 912 to determine the domain and intent using at least one of text or context information corresponding to a user's utterance. have. Even according to various embodiments, the intelligent server 200 may determine a reliability value corresponding to a domain and an intention, and may generate a plurality of pass rules.

According to various embodiments, the intelligent server 200 may generate a slot based on the text corresponding to the user's utterance and the intention determined by the intention classifier 912 , and may determine a reliability value corresponding to the slot. The intelligent server 200 may transmit data for correcting a parameter corresponding to a slot having a reliability value less than or equal to a specified value to the electronic device 100 . An operation for the electronic device 100 to modify a parameter will be described below.

According to various embodiments, the intelligent server 200 generates a plurality of pass rules corresponding to the plurality of domains when one pass rule cannot be determined because the intention determined from the user's utterance may correspond to a plurality of domains. It can be transmitted to the electronic device 100 . The intelligent server 200 designates a separate pass rule ID (eg, a conflict pass rule ID) causing a plurality of pass rules, and in such a case, the pass rule ID may be determined as a separate pass rule ID. have.

For example, when the user's utterance is "Send mom that you are late", the operation of sending the sentence 'I'm late' to the mother, which is the determined intention, may be performed using an email app, a message app, a messenger app, or the like. In an example, the intelligent server 200 may determine a pass rule ID as a separate pass rule ID, generate a plurality of pass rules using an email app, a message app, and a messenger app, and transmit the generated pass rule ID to the electronic device 100 . When executing the pass rule, the electronic device 100 may receive a user input for selecting an app and execute the pass rule corresponding to the user input. According to an embodiment, the electronic device 100 may store the user's preferred app setting value and execute a pass rule corresponding to the setting value.

As another example, when the user's utterance is "Edit this photo", the operation of editing the photo, which is the determined intention, may be performed using a gallery app, a 360 photo editing app, a moving photo editing app, and the like. In an example, the intelligent server 200 determines the pass rule ID as a separate pass rule ID, and generates a plurality of pass rules using the gallery app, the 360 photo editing app, and the moving photo editing app to the electronic device 100 . can be transmitted The electronic device 100 may execute a pass rule corresponding to the type information of the photo corresponding to the editing target. For example, when the type of photo is a 360 photo, the electronic device 100 may execute a pass rule corresponding to a 360 photo editing app.

Hereinafter, when the plurality of pass rules transmitted by the intelligent server 200 and sample utterances corresponding to the plurality of pass rules are included in the response received by the electronic device 100 in operation 803, specific operations of operations 804 to 806 will be described below. This is explained.

In operation 804 according to an embodiment, the processor 150 may display a plurality of sample utterances on the display 120 . In operation 805 , the processor 150 may receive a first user input for selecting one of a plurality of sample utterances through the microphone 111 or the touch screen display 120 . In operation 806, the processor 150 may perform a task related to the selected sample utterance.

In operation 805 according to an embodiment, the plurality of sample utterances may include a first sample utterance indicating a first intention of the user utterance and a second sample utterance indicating a second intention different from the first intention. According to an embodiment, the plurality of sample utterances may include a first sample utterance corresponding to a first pass rule and a second sample utterance corresponding to a second pass rule.

In an embodiment, in response to receiving the first user input, the processor 150 may display information related to a sequence of states related to the selected sample utterance. According to an embodiment, the processor 150 may receive a second user input for re-selecting the selected sample utterance. The processor 150 may perform a task related to the selected sample utterance in operation 806 in response to receiving the second user input by having the sequence of selected sample utterance states.

According to an embodiment, the information related to the sequence of states related to the selected sample utterance may include an explanation of an operation of the sequence of states (or a pass rule) or an utterance for performing the task.

In operation 804 according to an embodiment, the processor 150 may further display a message for receiving a user input indicating that there is no sample utterance corresponding to the user utterance on the display 120 . In operation 805 , the processor 150 may receive a user input indicating that there is no sample utterance corresponding to the user utterance. When the user input is received, the processor 150 may receive a user input for displaying a user uttered sentence and selecting and correcting a portion that does not match the user's intention. The processor 150 may transmit the received user input to the intelligent server 200 . According to various embodiments, the intelligent server 200 may perform learning of an intelligent service based on the received user input.

Hereinafter, an embodiment in which the electronic device 100 displays a sample utterance and receives a user input will be described with reference to FIGS. 14A to 14D .

14A is a diagram illustrating that a user's utterance is input while a gallery app is running, according to an exemplary embodiment.

14B is a diagram illustrating a screen on which a plurality of sample utterances are displayed, according to an exemplary embodiment.

14C is a diagram illustrating a screen on which information related to a selected sample utterance is displayed, according to an exemplary embodiment.

14D is a diagram illustrating a screen on which a user's intention is input, according to an exemplary embodiment;

Referring to FIG. 14A according to an embodiment, the electronic device 100 may receive a user utterance 1401 saying “increase the screen size” by executing an intelligent service while displaying a photo using the gallery app. The electronic device 100 may transmit context information corresponding to a picture being displayed to the intelligent server 200 using a voice signal corresponding to the user's utterance and a gallery app. The intelligent server 200 may transmit a plurality of pass rules and a plurality of sample utterances corresponding to the plurality of pass rules to the electronic device 100 based on the voice signal and context information.

The processor 150 may display the received plurality of sample utterances 1402 , 1403 , 1404 , 1405 and a message 1406 'not applicable' on the display 120 as shown in FIG. 14B . The electronic device 100 may receive a first user input for selecting one of a plurality of sample utterances through the microphone 111 or the touch screen display 120 . The processor 150 may perform a task related to the selected sample utterance.

In an embodiment, the processor 150 may receive a first user input for selecting the sample utterance 1404 'Increase the resolution in settings'. According to an embodiment, the processor 150 responds to the reception of the first user input, and as shown in FIG. 14C , a representative utterance 1407 for directly executing a pass rule corresponding to 'increasing resolution in settings'. can be displayed.

The processor 150 may receive a second user input for selecting 'increase resolution in settings' or selecting the object 1408 illustrated in FIG. 14C . The processor 150 may perform a task by executing a pass rule corresponding to 'increasing resolution in settings' in response to receiving the second user input.

In an embodiment, although the user's utterance is 'press the screen size', the intelligent server 200 may generate the text 'increase the screen size' from the user's utterance. The intelligent server 200 may transmit a plurality of pass rules based on the generated text and context information and a sample utterance corresponding to the plurality of pass rules to the electronic device 100 . As shown in FIG. 14B , the electronic device 100 may display a plurality of sample utterances 1402 , 1403 , 1404 , and 1405 and a message 1406 'not applicable' on the display 120 .

According to an embodiment, the processor 150 may receive a user input for selecting the message 1406 'not applicable' shown in FIG. 14B . When receiving the user input, the processor 150 may display 'increase the screen size' as shown in FIG. 14D . The processor 150 may receive a user input for selecting and correcting 'increase' 1409, which is a part that does not match the user's intention. According to an embodiment, the processor 150 may display another candidate list corresponding to 'increase' and receive a user input for selecting one of the candidates, or receive a text user input or an utterance input.

In operation 804 according to an embodiment, the processor 150 may display one or more images related to at least one sample utterance among a plurality of sample utterances on the display 120 . According to an embodiment, the displayed one or more images may indicate states corresponding to the sample utterance.

15 is a diagram illustrating a screen on which an image related to a sample utterance is further displayed, according to an exemplary embodiment.

As shown in FIG. 15 , the processor 150 may display an image 1502 corresponding to states corresponding to a pass rule that is executed when each sample utterance 1501 is selected. In an embodiment, the displayed state may correspond to each operation screen when the pass rule is executed.

In operation 804 according to an embodiment, the processor 150 may determine an order of displaying at least one sample utterance, and display the at least one sample utterance according to the determined order.

According to an embodiment, the response received in operation 803 may further include a value corresponding to each of at least one sample utterance. For example, the value corresponding to the sample utterance may be a confidence value.

According to an embodiment, the processor 150 may determine the order of displaying at least one sample utterance in the order of magnitudes of values corresponding to the sample utterances. For example, the processor 150 may display the sample utterances in an order of increasing reliability values.

According to an embodiment, the processor 150 may visually display at least one sample utterance differently based on the determined order. For example, the processor 150 may visually display a sample utterance corresponding to a first order among at least one sample utterance differently from other sample utterances. According to an embodiment, the processor 150 may display one or more images related to the sample utterance corresponding to the first order or display it larger than the remaining sample utterances.

According to various embodiments, the processor 150 may determine a method of displaying a sample utterance based on the user's selection history data.

16A is a diagram illustrating that a user's utterance is input while a gallery app is running, according to an exemplary embodiment.

16B is a diagram illustrating a screen on which sample utterances are displayed according to a determined order, according to an exemplary embodiment.

16C is a diagram illustrating a screen on which an image related to a sample utterance corresponding to a priority is further displayed, according to an exemplary embodiment.

16D is a diagram illustrating a screen in which sizes of sample utterances are displayed differently according to a determined order, according to an exemplary embodiment.

Referring to FIG. 16A according to an embodiment, the electronic device 100 may receive a user utterance 1601 saying “Tell me about the Hawaiian message” by executing an intelligent service while displaying a photo list using the gallery app. The electronic device 100 may transmit context information corresponding to a picture being displayed to the intelligent server 200 using a voice signal corresponding to the user's utterance and a gallery app. The intelligent server 200 may transmit a plurality of pass rules and a plurality of sample utterances corresponding to the plurality of pass rules to the electronic device 100 based on the voice signal and context information. The intelligent server 200 may further transmit a confidence value corresponding to a plurality of sample utterances.

In an embodiment, the plurality of sample utterances may include 'send a message to a photo taken in Hawaii', 'show me a photo taken in Hawaii', and 'tell me a way to go to Hawaii'. In an embodiment, the reliability value corresponding to each sample utterance may be increased in the order of 'send a message to a photo taken in Hawaii', 'show me a photo taken in Hawaii', and 'tell me how to go to Hawaii'.

According to an embodiment, the processor 150 may display a plurality of sample utterances in an order of increasing reliability values as shown in FIG. 16B .

According to an embodiment, the processor 150 may display one or more images 1602 related to 'send a message for pictures taken in Hawaii' having the highest reliability value as shown in FIG. 16C . The images may represent states corresponding to sample firing.

According to an embodiment, as shown in FIG. 16D , the processor 150 may display a larger sample utterance as the reliability value increases.

According to various embodiments, the processor 150 may display a sample utterance by combining at least some of the above-described embodiments for displaying a sample utterance.

According to various embodiments of the present disclosure, when receiving the plurality of pass rules, the electronic device 100 selects one pass rule that satisfies a specified condition among the plurality of pass rules without receiving a user input for selecting the pass rule as described above. You can run it right away. According to various embodiments, since the executed pass rule may not conform to the user's intention, the electronic device 100 may perform an operation described below. The operations of the embodiments described below may be performed when the reliability value of the pass rule determined by the intelligent server is greater than the specified value.

17 is a flowchart illustrating a method of an electronic device performing another pass rule by receiving a feedback input after performing a pass rule, according to an exemplary embodiment.

Referring to FIG. 17 , in operation 1701 , the processor 150 may receive a user's utterance through the microphone 111 .

In operation 1702 , the processor 150 may transmit data related to the user's utterance to the intelligent server through the wireless communication circuit.

According to an embodiment, the data related to the user's utterance may include a voice signal corresponding to the user's utterance.

According to an embodiment, the processor 150 may transmit context information indicating the current state of the electronic device to the intelligent server through a wireless communication circuit. The context information may include, for example, information about the running app or state information of the running app. In an embodiment, the state information of the app may include a state ID, specific screen information of the app, or specific state information on a specific screen.

In operation 1703 , the processor 150 may receive a response from the intelligent server through a wireless communication circuit.

According to an embodiment, the received response may include a plurality of pass rules and a sample utterance corresponding to each of the plurality of pass rules. The plurality of pass rules and sample utterances may be generated or selected by the intelligent server 200 based on the user utterance. According to an embodiment, the response may further include a reliability value corresponding to each of the plurality of pass rules.

In operation 1704 , the processor 150 may select one pass rule from among a plurality of pass rules and perform an operation according to the selected pass rule.

In an embodiment, the processor 150 may select a pass rule having the highest reliability value among a plurality of pass rules.

In operation 1705 , the processor 150 may receive a feedback input through the microphone 111 or the touchscreen display 120 .

In an embodiment, the feedback input may correspond to feedback on a result of executing an operation according to a selected pass rule.

18 is a diagram illustrating a screen on which an electronic device receives a feedback input, according to an embodiment.

The processor 150 may display a screen as shown in FIG. 18 on the display 120 and receive a feedback input. For example, the processor 150 may receive positive feedback such as “like” or “yes” through the microphone 111 , and receive negative feedback such as “not this”, “no” or “something else”. can receive

As another example, the processor 150 receives a touch input for selecting a first object 1701 corresponding to a positive feedback input through the touch screen display 120 or receives a second object 1702 corresponding to a negative feedback input through the touch screen display 120 . A selection touch input may be received.

Referring back to FIG. 17 , if the feedback input is a negative input, in operation 1706 , the processor 150 displays, on the display 120 , sample utterances corresponding to the remaining pass rules not selected in operation 1704 included in the response. can

19 is a diagram illustrating a screen on which sample utterances corresponding to remaining pass rules are displayed, according to an exemplary embodiment.

As shown in FIG. 18 , the processor 150 may display a plurality of sample utterances 1801 corresponding to the remaining pass rules on the display 120 .

Referring back to FIG. 17 , in operation 1707 , the processor 150 may receive a user input for selecting one of a plurality of sample utterances through the microphone 111 or the touch screen display 120 .

In operation 1708 , the processor 150 may perform an operation according to a pass rule corresponding to the selected sample utterance.

20 is a flowchart illustrating a method of performing another pass rule by receiving a feedback input after an electronic device performs a pass rule, according to an exemplary embodiment.

Referring to FIG. 20 , in operation 2001 , the processor 150 may receive a user utterance through the microphone 111 .

In operation 2002, the processor 150 may transmit data related to the user's utterance to the intelligent server 200 through a wireless communication circuit.

In operation 2003, the processor 150 may receive a response from the intelligent server through the wireless communication circuit.

According to an embodiment, the received response may include one pass rule and a sample utterance corresponding to the pass rule. The pass rule and the sample utterance may be generated or selected by the intelligent server 200 based on the user utterance.

In operation 2004, the processor 150 may perform an operation according to the received pass rule.

In operation 2005 , the processor 150 may receive a feedback input through the microphone 111 or the touch screen display 120 .

When the feedback input is a negative input, in operation 2008 , the processor 150 transmits data related to the user's utterance (eg, user utterance and context information) to the intelligent server 200 , and receives a plurality of alternatives from the intelligent server 200 . You can receive pass rules.

In an embodiment, the processor 150 may request transmission of an alternative pass rule when transmitting data related to a user's utterance to the intelligent server 200 .

In operation 2007 , the processor 150 may display sample utterances corresponding to the plurality of received alternative pass rules on the display 120 .

In operation 2008 , the processor 150 may receive a user input for selecting one of a plurality of sample utterances through the microphone 111 or the touch screen display 120 .

In operation 2009, the processor 150 may perform an operation according to a pass rule corresponding to the selected sample utterance.

In various embodiments, the electronic device 100 may transmit data related to a user input for selecting a sample utterance to the intelligent server 200 . The intelligent server 200 may use the received data to determine the user's intention.

21 is a diagram illustrating a screen on which an electronic device receives a feedback input, according to an embodiment.

According to an embodiment, after operation 1704 or after operation 2004, the processor 150 may display a screen as shown in FIG. 21 on the display 120 and receive a feedback input. For example, the processor 150 receives a touch input for selecting the first object 2111 corresponding to the positive feedback input through the touch screen display 120 or the second object 2112 corresponding to the negative feedback input may receive a touch input for selecting .

22A and 22B illustrate screens on which a UI related to a function provided by the electronic device 100 is displayed when negative feedback is received, according to an exemplary embodiment.

According to an embodiment, when a negative feedback input is received, the processor 150 may display a screen as shown in FIG. 22A or 22B .

According to an embodiment, when the task that is the target of negative feedback is a task using an app screen other than an intelligent app (or when the task is performed according to a pass rule), the processor 150 may display a screen as shown in FIG. 22A . have. For example, if the task that is the target of negative feedback is a task of searching for photos using a gallery application, and a negative feedback input is received after the task is performed, the processor 150 may display a screen as shown in FIG. 22A .

According to an embodiment, when the task that is the target of negative feedback is a task that uses only the intelligent app screen (or the task is not performed according to a pass rule), the processor 150 may display a screen as shown in FIG. 22B . have. For example, if the task that is the target of negative feedback is a task of informing about weather or a task of providing an answer according to a query, and the negative feedback input is received after the task is performed, the processor 150 displays a screen as shown in FIG. 22B can be displayed

According to an embodiment, as shown in FIG. 22A , the processor 150 sets a UI 2211 for correcting text corresponding to an erroneously recognized user utterance, a UI 2212 for selecting a pass rule, and a pass rule. A UI 2213 for searching and selecting, a UI 2214 for searching within the electronic device 100, and a UI 2215 for searching on the Internet may be displayed on the display 120 . According to an embodiment, the processor 150 may receive a user input for selecting one of the displayed UIs 2211 , 2212 , 2213 , 2214 , and 2215 .

According to an embodiment, as shown in FIG. 22B , the processor 150 provides a UI 2221 for correcting text corresponding to an erroneously recognized user utterance, feedback that a domain that does not match the user's intention is selected. A UI 2222 for searching and selecting a pass rule, a UI 2223 for searching within the electronic device 100, and a UI 2225 for searching on the Internet are displayed on the display 120. can be displayed The processor 150 may receive a user input for selecting one of the displayed UIs 2221 , 2222 , 2223 , 2224 , and 2225 .

According to an embodiment, the UIs 2211 and 2221 for correcting text corresponding to the erroneously recognized user's utterance are texts 2211a and 2221a received from the intelligent server 200 and recognized from the user's utterance by the intelligent server 200 . ) may be included. According to an embodiment, the UI 2212 for selecting a pass rule may include a plurality of sample utterances indicating each pass rule.

Hereinafter, an operation performed by the processor 150 when each UI is selected will be described with reference to FIGS. 23A to 27 .

23A illustrates a screen on which a text to be edited is displayed, according to an exemplary embodiment. 23B illustrates a screen in which a text to be edited is corrected, according to an exemplary embodiment. 23C illustrates a result screen in which text corresponding to a user's utterance is corrected, according to an exemplary embodiment. 23D illustrates a screen including a result of a text corresponding to a user's utterance corrected and a UI for updating a voice model corresponding to the corrected text, according to an exemplary embodiment. 23E illustrates a screen including a pop-up window according to a network connection failure according to an embodiment.

According to an embodiment, the processor 150 is a user who selects some of texts 2211b and 2221b recognized from the user's utterance by the intelligent server 200 received from the intelligent server 200 shown in FIGS. 22A and 22B . input can be received. According to an embodiment, in response to receiving the user input, the processor 150 may designate the selected partial texts 2211b and 2221b as the text to be edited. For example, the processor 150 may receive a user input for selecting “photto” from among “Show my recent photto” of FIG. 22A or 22B . In an embodiment, the processor 150 may designate “photto” as the text to be edited.

According to an embodiment, the processor 150 may display the editing target text 2311 and the modified object 2312 as shown in FIG. 23A . According to an embodiment, the correction target text 2311 may be modified like the corrected text 2321 shown in FIG. 23B by input through a virtual keyboard or the like. For example, “photto” 2311 shown in FIG. 23A may be modified to “photos” 2321 shown in FIG. 23B .

According to an embodiment, after the text to be edited 2311 is modified, the processor 150 may receive a user input for selecting the modified object 2322 . According to an embodiment, after a user input for selecting the modified object 2322 is received, the processor 150 may display the screen illustrated in FIG. 23C or 23D on the display 120 .

According to an embodiment, after the user input for selecting the modified object 2322 is received, the language setting of the intelligent app and the language setting of the electronic device 100 are different or recognized text (eg, “Show my recent photto”) ”), if there is a history of receiving the user's utterance through the microphone 111, the processor 150 displays a screen showing a result 2331 in which the text corresponding to the user's utterance is corrected, as shown in FIG. 23C . can be displayed

According to an embodiment, the processor 150 may transmit data related to the corrected result (eg, corrected text data) to the intelligent server 200 through a wireless communication circuit. According to an embodiment, data related to the corrected result transmitted to the intelligent server 200 may be stored in the automatic voice recognition database 211 of the intelligent server 200 .

According to an embodiment, data related to the corrected result stored in the automatic speech recognition database 211 may be used when the automatic speech recognition module 210 converts a user's utterance into text data. For example, the automatic speech recognition module 210 of the intelligent server 200 may convert the user's utterance into text data by applying a weight value to the modified text. According to an embodiment, as the number of corrections of the text increases, a weight value applied to the text may increase, and the accuracy of automatic voice recognition may increase.

According to an embodiment, after a user input for selecting the modified object 2322 is received, the language setting of the intelligent app and the language setting of the electronic device 100 are the same, and the user's utterance corresponding to the recognized text is output by the microphone 111 ), when there is no history received through the processor 150, as shown in FIG. 23D , the text corresponding to the user's utterance is corrected result 2341 and the UI for updating the voice model corresponding to the corrected text ( 2342) may be displayed.

According to an embodiment, when a user input for selecting a UI 2342 for updating a voice model corresponding to the corrected text is received, the processor 150 generates an additional user utterance corresponding to the recognized text into the microphone 111 . can be received through

According to an embodiment, the processor 150 may transmit data (eg, a voice signal) related to additional user utterance received through the microphone 111 to the intelligent server 200 through a wireless communication circuit. According to an embodiment, data related to additional user utterance transmitted to the intelligent server 200 may be stored in the automatic voice recognition database 211 of the intelligent server 200 . According to an embodiment, data related to additional user utterance stored in the automatic speech recognition database 211 may be used when the automatic speech recognition module 210 converts the user's utterance into text data.

According to an embodiment, in order for the processor 150 to transmit data related to an additional user's utterance to the intelligent server 200 , the electronic device 100 must be connected to a network. In an embodiment, when a user input for selecting a UI 2342 for updating a voice model corresponding to the modified text is received and the electronic device 100 is not connected to a network, the processor 150 is shown in 23e As described above, a message 2351 indicating that a network connection is required may be displayed on the display 120 .

According to an embodiment, the processor 150 may receive a user input for selecting one of a plurality of sample utterances included in the UI 2212 for selecting a pass rule shown in FIG. 22A . In response to receiving the user input, the processor 150 may execute a pass rule related to the selected sample utterance. Specific details thereof are the same as those described with reference to FIGS. 8 and 14A to 14D .

24A illustrates a screen for searching for a pass rule according to an embodiment. 24B illustrates a pass rule search result screen according to a search word input according to an exemplary embodiment. 24C illustrates a screen after a pass rule is selected according to an exemplary embodiment.

According to an embodiment, the processor 150 may receive a user input for selecting the UIs 2213 and 2223 for searching and selecting the pass rules shown in FIGS. 22A and 22B . According to an embodiment, in response to receiving a user input, the processor 150 may display a screen for searching for a pass rule on the display 120 as shown in FIG. 24A . Referring to FIG. 24A , the screen for searching a pass rule includes a message 2411 for guiding the user to search for and select a pass rule, a search window 2412 for displaying a user input, and a command sentence list 2413 indicating the pass rule. ) may be included.

According to an embodiment, the processor 150 may receive an input through a virtual keyboard, etc., display the input text 2421 in a search window, and display a command sentence list 2422 including the input text. have. According to an embodiment, the processor 150 may receive a user input for selecting one of the command characters included in the displayed command sentence list. For example, the processor 150 may receive a text input of “Remin”, display the input “Remin” 2421 in the search window as shown in FIG. 24B , and a command including “Remin” A sentence list 2422 may be displayed. The processor 150 may receive a user input for selecting a command sentence “Reminder for 10min”.

According to an embodiment, when a user input for selecting a command sentence is received, the processor 150 may display a result screen indicating the selected command sentence on the display 120 . According to an embodiment, for example, the result screen may include a selected command sentence “Reminder for 10min” 2431 and a guide message 2432 as shown in FIG. 24C .

According to an embodiment, the processor 150 may transmit data related to the selected command sentence (eg, pass rule information corresponding to the selected command sentence) to the intelligent server 200 . According to an embodiment, the intelligent server 200 may store data related to a command sentence. According to an embodiment, the intelligent server 200 may generate a pass rule using data related to a stored command sentence. For example, when the intelligent server 200 receives data (eg, a voice signal) related to a user utterance recognized as “Show my recent photto” from the electronic device 100 , the intelligent server 200 stores the command sentence A pass rule corresponding to “Reminder for 10min” can be created using related data.

25 illustrates a screen after a UI for feedback that a domain that does not match a user's intention is selected is selected, according to an exemplary embodiment.

According to an embodiment, the domain (or application) selected by the intelligent server 200 may not conform to the user's intention. For example, the application selected by the intelligent server 200 may be an 'internet application', but an application that meets the user's intention may be a 'gallery application'.

According to an embodiment, the processor 150 may receive a user input for selecting the UI 2222 for feedback that a domain that does not match the user's intention shown in FIG. 22B is selected. According to an embodiment, in response to receiving the user input, the processor 150 may display a message 2511 indicating that a feedback input indicating that a domain not meeting the user's intention is selected is received as shown in FIG. 25 . can According to an embodiment, the processor 150 may further display a message indicating that an application is incorrectly selected (eg, “It is not a request to use this app”).

According to an embodiment, the processor 150 may transmit data related to a user input indicating that a domain that does not match the user's intention is selected to the intelligent server 200 . According to an embodiment, the intelligent server 200 may increase the accuracy of domain determination by using the received data.

26 illustrates a screen for performing a search in an electronic device according to an exemplary embodiment.

According to an embodiment, the processor 150 may receive a user input for selecting the UIs 2214 and 2224 for searching in the electronic device 100 illustrated in FIGS. 22A and 22B . According to an embodiment, in response to receiving a user input, the processor 150 may display a screen for performing a search in the electronic device 100 on the display 120 as shown in FIG. 26 . Referring to FIG. 26 , a screen for performing a search within the electronic device 100 may include a search window 2611 displaying a user input, a search history list 2612 , and a suggested application list 2613 . According to an embodiment, the search window 2611 is an area in which an input received through a virtual keyboard is displayed, the search history list 2612 includes search histories, and the suggested application list 2613 includes user information or user information. Applications selected based on at least one of the inputs may be included.

According to an embodiment, the processor 150 receives an input through a virtual keyboard, etc., displays the input text in a search window, and uses the input text as a search word to perform an operation that the electronic device 100 may perform. Alternatively, you can search for installed applications and the like.

According to an embodiment, the processor 150 may use a part of text corresponding to the user's utterance as a search term. For example, the processor 150 may display “recent photto” among “Show my recent photto” in the search window 2611 , and the processor 150 uses the search word itself displayed in the search box 2611 . A search may be performed or a search may be performed using a search word modified by a user input.

27 illustrates a screen for performing a search on the Internet according to an embodiment.

According to an embodiment, the processor 150 may receive a user input for selecting the UIs 2215 and 2225 for searching on the Internet shown in FIGS. 22A and 22B . According to an embodiment, in response to receiving a user input, the processor 150 may display a screen for performing a search in the electronic device 100 on the display 120 as shown in FIG. 27 . Referring to FIG. 27 , a screen for performing a search within the electronic device 100 may include a search window 2711 in which a user input is displayed.

According to an embodiment, the processor 150 may receive an input through a virtual keyboard, etc., display the input text in the search window 2711, and search the Internet using the input text as a search word.

According to an embodiment, the processor 150 may use a part of text corresponding to the user's utterance as a search term. For example, the processor 150 may display “recent photto” among “Show my recent photto” in the search box, and the processor 150 performs a search using the search word itself displayed in the search box or a user input A search can be performed using the search term modified by .

28 is a block diagram of an electronic device 2801 in a network environment 2800 according to various embodiments of the present disclosure. The electronic device according to various embodiments disclosed in this document may be a device of various types. An electronic device may be, for example, a portable communication device (eg, a smartphone), a computer device (eg, a personal digital assistant (PDA), a tablet PC (tablet PC), a laptop PC (eg, a desktop PC, a workstation, or a server)). , a portable multimedia device (eg, an e-book reader or an MP3 player), a portable medical device (eg, a heart rate, blood sugar, blood pressure, or body temperature monitor), a camera, or a wearable device. Types (e.g. watches, rings, bracelets, anklets, necklaces, eyeglasses, contact lenses, or head-mounted-devices (HMDs)); For example, a skin pad or a tattoo), or a bioimplantable circuit.In some embodiments, the electronic device may include, for example, a television, a digital video disk (DVD) player, an audio device, an audio accessory. Devices (such as speakers, headphones, or headsets), refrigerators, air conditioners, vacuum cleaners, ovens, microwaves, washing machines, air purifiers, set-top boxes, home automation control panels, security control panels, game consoles, electronic dictionaries, electronic keys, It may include at least one of a camcorder and an electronic picture frame.

In another embodiment, the electronic device may include a navigation device, a global navigation satellite system (GNSS), an event data recorder (EDR) (eg, a black box for a vehicle/vessel/airplane), and an automotive infotainment device. (e.g. head-up displays for vehicles), industrial or home robots, drones, automated teller machines (ATMs), point of sales (POS) instruments, metering instruments (e.g. water, electricity, or gas metering instruments); Alternatively, it may include at least one of an IoT device (eg, a light bulb, a sprinkler device, a fire alarm, a thermostat, or a street lamp). The electronic device according to the embodiment of this document is not limited to the above-described devices, and, for example, as in the case of a smartphone equipped with a function of measuring personal biometric information (eg, heart rate or blood sugar), a plurality of The functions of the devices may be provided in a complex manner. In this document, the term user may refer to a person who uses an electronic device or a device (eg, an artificial intelligence electronic device) using the electronic device.

Referring to FIG. 28 , in a network environment 2800 , an electronic device 2801 (eg, the electronic device 100 ) communicates with the electronic device 2802 through short-range wireless communication 2898 or a network 2899 . Through this, it is possible to communicate with the electronic device 2804 or the server 2808 (eg, the intelligent server 200). According to an embodiment, the electronic device 2801 may communicate with the electronic device 2804 through the server 2808 .

According to an embodiment, the electronic device 2801 includes a bus 2810 , a processor 2820 (eg, the processor 150 ), a memory 2830 , an input device 2850 (eg, a microphone or a mouse), and a display device. 2860 , audio module 2870 , sensor module 2876 , interface 2877 , haptic module 2879 , camera module 2880 , power management module 2888 , and battery 2889 , communication module 2890 ), and a subscriber identification module 2896 . In some embodiments, the electronic device 2801 may omit at least one of the components (eg, the display device 2860 or the camera module 2880) or additionally include other components.

The bus 2810 may include a circuit that connects the components 2820 - 2890 to each other and transmits a signal (eg, a control message or data) between the components.

The processor 2820 is one of a central processing unit (CPU), an application processor (AP), a graphics processing unit (GPU), an image signal processor (ISP) of a camera, or a communication processor (CP). or more. According to an embodiment, the processor 2820 may be implemented as a system on chip (SoC) or a system in package (SiP). The processor 2820 may control at least one other component (eg, a hardware or software component) of the electronic device 2801 connected to the processor 2820 by, for example, driving an operating system or an application program, and , various data processing and operations can be performed. The processor 2820 loads and processes commands or data received from at least one of other components (eg, the communication module 2890 ) into the volatile memory 2832 , and stores the result data in the non-volatile memory 2834 . can

The memory 2830 may include a volatile memory 2832 or a non-volatile memory 2834 . The volatile memory 2832 may include, for example, random access memory (RAM) (eg, DRAM, SRAM, or SDRAM). The non-volatile memory 2834 may include, for example, programmable read-only memory (PROM), one time PROM (OTPROM), erasable PROM (EPROM), electrically EPROM (EEPROM), mask ROM, flash ROM, flash memory, HDD. (hard disk drive), or may be configured as a solid state drive (SSD). In addition, the non-volatile memory 2834 includes an internal memory 2836 disposed therein, or an external stand-alone type that can be connected and used only when necessary, depending on a connection type with the electronic device 2801 . memory 2838 . The external memory 2838 is a flash drive, for example, a compact flash (CF), a secure digital (SD), a Micro-SD, a Mini-SD, an extreme digital (xD), a multi-media card (MMC). , or a memory stick. The external memory 2838 may be functionally or physically connected to the electronic device 2801 through a wired (eg, a cable or universal serial bus (USB)) or wireless (eg, Bluetooth).

The memory 2830 may store, for example, at least one other software component of the electronic device 2801 , for example, a command or data related to the program 2840 . The program 2840 may include, for example, a kernel 2841 , a library 2843 , an application framework 2845 , or an application program (interchangeably “application”) 2847 .

The input device 2850 may include a microphone, a mouse, or a keyboard. According to an embodiment, the keyboard may be connected as a physical keyboard or may be displayed as a virtual keyboard through the display device 2860 .

The display device 2860 may include a display, a hologram device, or a projector and a control circuit for controlling the corresponding device. The display may include, for example, a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a microelectromechanical system (MEMS) display, or an electronic paper display. . The display, according to an embodiment, may be implemented to be flexible, transparent, or wearable. The display may include touch circuitry capable of sensing a user's touch, gesture, proximity, or hovering input, or a pressure sensor capable of measuring the intensity of pressure on the touch (interchangeably "force sensor"). can The touch circuit or pressure sensor may be implemented integrally with the display, or may be implemented with one or more sensors separate from the display. The holographic device may display a three-dimensional image in the air using interference of light. A projector can display an image by projecting light onto a screen. The screen may be located inside or outside the electronic device 2801 , for example.

The audio module 2870 may interactively convert a sound and an electrical signal, for example. According to an embodiment, the audio module 2870 acquires sound through the input device 2850 (eg, a microphone) or an output device (not shown) included in the electronic device 2801 (eg, a speaker or receiver), or through an external electronic device connected to the electronic device 2801 (eg, the electronic device 2802 (eg, wireless speakers or wireless headphones) or the electronic device 2806 (eg, wired speakers or wired headphones)). can be printed out.

The sensor module 2876 measures or senses, for example, an internal operating state (eg, power or temperature) of the electronic device 2801, or an external environmental state (eg, altitude, humidity, or brightness), An electrical signal or data value corresponding to the measured or sensed state information may be generated. The sensor module 2876 may include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, and a color sensor (eg, an RGB (red, green, blue) sensor). , IR (infrared) sensor, biometric sensor (such as iris sensor, fingerprint sensor, or heartbeat rate monitoring (HRM) sensor, olfactory (electronic nose) sensor, electromyography (EMG) sensor, electroencephalogram (EEG) sensor, electrocardiogram (ECG) sensor sensor), a temperature sensor, a humidity sensor, an illuminance sensor, or an ultra violet (UV) sensor. The sensor module 2876 may further include a control circuit for controlling at least one or more sensors included therein. In some embodiments, the electronic device 2801 may control the sensor module 2876 using the processor 2820 or a processor (eg, a sensor hub) separate from the processor 2820 . When a separate processor (eg, a sensor hub) is used, the electronic device 2801 does not wake the processor 2820 while the processor 2820 is in a sleep state, and the sensor module is operated by the separate processor. At least a portion of the operation or state of 2876 may be controlled.

The interface 2877, according to an embodiment, is a high definition multimedia interface (HDMI), USB, optical interface, RS-232 (recommended standard 232), D-sub (D-subminiature), MHL (mobile). It may include a high-definition link) interface, an SD card/multi-media card (MMC) interface, or an audio interface. The connection terminal 2878 may physically connect the electronic device 2801 and the electronic device 2806 . According to an embodiment, the connection terminal 2878 may include, for example, a USB connector, an SD card/MMC connector, or an audio connector (eg, a headphone connector).

The haptic module 2879 may convert an electrical signal into a mechanical stimulus (eg, vibration or movement) or an electrical stimulus. For example, the haptic module 2879 may provide a stimulus related to a tactile or kinesthetic sense to the user. The haptic module 2879 may include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

The camera module 2880 may capture still images and moving images, for example. The camera module 2880, according to an embodiment, includes one or more lenses (eg, a wide-angle lens and a telephoto lens, or a front lens and a rear lens), an image sensor, an image signal processor, or a flash (eg, a light emitting diode or a xenon lamp). (xenon lamp), etc.) may be included.

The power management module 2888 is a module for managing power of the electronic device 2801 and may be configured, for example, as at least a part of a power management integrated circuit (PMIC).

The battery 2889 may be recharged by an external power source, including, for example, a primary cell, a secondary cell, or a fuel cell to supply power to at least one component of the electronic device 2801 .

The communication module 2890 establishes, for example, a communication channel between the electronic device 2801 and an external device (eg, the first external electronic device 2802 , the second external electronic device 2804 , or the server 2808 ). And it may support the performance of wired or wireless communication through the established communication channel. According to an embodiment, the communication module 2890 includes a wireless communication module 2892 or a wired communication module 2894, and the first network 2898 (eg, Bluetooth or IrDA) using a corresponding communication module among them. (a short-range communication network such as infrared data association) or the second network 2899 (eg, a long-distance communication network such as a cellular network) may communicate with the external device.

The wireless communication module 2892 may support cellular communication, short-range wireless communication, or GNSS communication, for example. Cellular communication is, for example, long-term evolution (LTE), LTE Advance (LTE-A), code division multiple access (CDMA), wideband CDMA (WCDMA), universal mobile telecommunications system (UMTS), Wireless Broadband (WiBro) ), or GSM (Global System for Mobile Communications). Short-range wireless communication is, for example, Wi-Fi (wireless fidelity), Wi-Fi Direct, Li-Fi (light fidelity), Bluetooth, BLE (Bluetooth low energy), Zigbee, NFC (near field communication), MST ( magnetic secure transmission), radio frequency (RF), or body area network (BAN). The GNSS may include, for example, Global Positioning System (GPS), Global Navigation Satellite System (Glonass), Beidou Navigation Satellite System (hereinafter, “Beidou”) or Galileo (the European global satellite-based navigation system). In this document, "GPS" may be used interchangeably with "GNSS".

According to an embodiment, the wireless communication module 2892 performs identification and authentication of the electronic device 2801 in a communication network using, for example, the subscriber identification module 2896 when cellular communication is supported. can do. According to an embodiment, the wireless communication module 2892 may include a CP separate from the processor 2820 (eg, AP). In this case, the CP may be, for example, on behalf of the processor 2820 while the processor 2820 is in an inactive (eg, sleep) state, or with the processor 2820 while the processor 2820 is in an active state. Together, at least some functions of functions related to at least one component among components 2810 - 2896 of the electronic device 2801 may be performed. According to an embodiment, the wireless communication module 2892 may include a plurality of communication modules supporting only a corresponding communication method among a cellular communication module, a short-range wireless communication module, and a GNSS communication module.

Wired communication module 2894 may include, for example, a local area network (LAN), power line communication, or plain old telephone service (POTS).

The first network 2898, for example, uses Wi-Fi Direct or Bluetooth capable of transmitting or receiving commands or data through a wireless direct connection between the electronic device 2801 and the first external electronic device 2802 . may include The second network 2899 is, for example, a telecommunication network (eg, a local area network (LAN) or a communication network capable of transmitting or receiving commands or data between the electronic device 2801 and the second external electronic device 2804 ) a computer network, such as a wide area network (WAN), the Internet, or a telephone network).

According to various embodiments, the command or the data may be transmitted or received between the electronic device 2801 and the second external electronic device 2804 through the server 2808 connected to the second network. Each of the first and second external electronic devices 2802 and 2804 may be the same as or different from the electronic device 2801 . According to various embodiments, all or a part of operations executed by the electronic device 2801 may be executed by one or a plurality of other electronic devices (eg, the electronic devices 2802 and 2804, or the server 2808). According to an example, when the electronic device 2801 is to perform a function or service automatically or upon request, the electronic device 2801 performs at least an associated at least one function or service instead of or in addition to executing the function or service itself. Some functions may be requested from another device (eg, the electronic devices 2802, 2804, or the server 2808) The other electronic device (eg, the electronic devices 2802, 2804, or the server 2808) may request the requested function. The function or additional function may be executed and the result may be transmitted to the electronic device 2801. The electronic device 2801 may process the received result as it is or additionally to provide the requested function or service. For example, cloud computing, distributed computing, or client-server computing technology may be used.

An electronic device according to an embodiment disclosed in this document includes: a housing; a touch screen display disposed within the housing and exposed through a first portion of the housing; a microphone disposed within the housing and exposed through a second portion of the housing; at least one speaker disposed within the housing and exposed through a third portion of the housing; wireless communication circuitry disposed within the housing; a processor disposed within the housing and electrically coupled to the touchscreen display, the microphone, the at least one speaker, and the wireless communication circuitry; and a memory disposed within the housing and electrically coupled to the processor, wherein the memory, when executed, causes the processor to, in a first example, receive a first user utterance through the microphone, the first user The utterance includes a first request for performing a first task with the electronic device, and transmits first data related to the first user utterance to an external server through the wireless communication circuit, and through the wireless communication circuit Receive a first response from an external server, wherein the first response includes information about sequences of first states of the electronic device for performing the first task, wherein the electronic device performs the sequences of first states perform the first task by having to have, in a second example, receive a second user utterance through the microphone, the second user utterance comprising a second request to perform a second task with the electronic device and transmit second data related to the second user utterance to the external server through the wireless communication circuit, and receive a second response from the external server through the wireless communication circuit, wherein the second response is transmitted to the user at least one sample utterance indicating an intention of the utterance, wherein the at least one sample utterance is selected by the external server based on the second user utterance, and displaying the at least one sample utterance on the touchscreen display and receiving a first user input for selecting one of the at least one sample utterances through the microphone or the touchscreen display, and causing the electronic device to display a second state related to the selected one of the at least one sample utterances. The instructions for performing the second task may be stored by having a sequence of .

In an embodiment, the at least one sample user utterance may include a first sample utterance indicating a first intention and a second sample utterance indicating a second intention different from the first intention.

In an embodiment, the instructions may further cause the processor to display one or more images related to the at least one sample utterance on the touchscreen display.

In an embodiment, the one or more images may represent the second states.

In an embodiment, the instructions may cause the processor to determine an order of displaying the at least one sample utterance and display the at least one sample utterance according to the determined order.

In an embodiment, the second response further includes a value corresponding to each of the at least one sample utterance, and the instructions include an order in which the processor displays the at least one sample utterance in an order of magnitude of the value. can be made to determine

In an embodiment, the instructions may cause the processor to visually display a sample utterance corresponding to a first order among the at least one sample utterance differently from other sample utterances.

In an embodiment, the instructions are configured to cause the processor to, in response to receiving the first user input, display information related to a sequence of second states related to the selected sample utterance, and select the selected sample utterance again. Receive two user inputs, and perform the second task in response to receiving the second user input.

In an embodiment, the displayed information may include at least one of an operation of the sequence of the second states or an utterance for performing the second task.

In addition, an electronic device according to an embodiment disclosed in this document may include: a housing; a touch screen display disposed within the housing and exposed through a first portion of the housing; a microphone disposed within the housing and exposed through a second portion of the housing; at least one speaker disposed within the housing and exposed through a third portion of the housing; wireless communication circuitry disposed within the housing; a processor disposed within the housing and electrically coupled to the touchscreen display, the microphone, the at least one speaker, and the wireless communication circuitry; and a memory disposed within the housing and electrically coupled to the processor, wherein the memory, when executed, causes the processor to receive a user utterance through the microphone, wherein the user utterance performs a task with the electronic device. a request to perform, transmitting data related to the user utterance to an external server via the wireless communication circuitry, receiving a response from the external server via the wireless communication circuitry, wherein the response comprises a plurality of sample utterances; wherein the plurality of sample utterances include a first sample utterance corresponding to a first pass rule and a second sample utterance corresponding to a second pass rule, wherein the first pass rule and the second pass rule are determined by the user. generating by the external server based on data related to a utterance, displaying the at least one sample utterance on the touchscreen display, and selecting one of the at least one sample utterance through the microphone or the touchscreen display and receive a first user input for utterance, and cause the electronic device to have a sequence of states related to a selected one of the at least one sample utterance to perform the task.

In an embodiment, the instructions may further cause the processor to display one or more images related to the at least one sample utterance on the touchscreen display.

In an embodiment, the one or more images may represent the states.

In an embodiment, the instructions may cause the processor to determine an order of displaying the at least one sample utterance and display the at least one sample utterance according to the determined order.

In an embodiment, the response further includes a value corresponding to each of the at least one sample utterance, and the instructions cause the processor to determine an order of displaying the at least one sample utterance in an order of magnitude of the value. can be made to decide

In an embodiment, the instructions may cause the processor to visually display a sample utterance corresponding to a first order among the at least one sample utterance differently from other sample utterances.

In an embodiment, the instructions cause the processor to, in response to receiving the first user input, display information related to a sequence of states related to the selected sample utterance, and reselect the selected sample utterance. Receive a user input, and perform the task in response to receiving the second user input.

In an embodiment, the displayed information may include at least one of an operation of the sequence of states or an utterance for performing the task.

In addition, the storage medium according to an embodiment disclosed in this document is a storage medium for storing computer-readable instructions, wherein when the instructions are executed by a processor of an electronic device, the electronic device causes the electronic device to receive a user's utterance operation, the user utterance includes a request for performing a task with the electronic device, transmitting data related to the user utterance to an external server, receiving a response from the external server, wherein the response is the user utterance at least one sample utterance indicating the intention of The operation of performing the task may be performed by receiving a first user input for selecting one of the utterances, and causing the electronic device to have a sequence of states related to the selected one of the at least one sample utterances .

In an embodiment, the at least one sample user utterance may include a first sample utterance indicating a first intention and a second sample utterance indicating a second intention different from the first intention.

In an embodiment, the instructions, when executed by a processor of the electronic device, may cause the electronic device to further display one or more images related to the at least one sample utterance.

Various embodiments of this document and terms used therein are not intended to limit the technology described in this document to a specific embodiment, but it should be understood to include various modifications, equivalents, and/or substitutions of the embodiments. In connection with the description of the drawings, like reference numerals may be used for like components. The singular expression may include the plural expression unless the context clearly dictates otherwise. In this document, expressions such as “A or B”, “at least one of A and/or B”, “A, B or C” or “at least one of A, B and/or C” refer to all of the items listed together. Possible combinations may be included. Expressions such as "first," "second," "first," or "second," can modify the corresponding elements regardless of order or importance, and to distinguish one element from another element. It is used only and does not limit the corresponding components. When an (eg, first) component is referred to as being “connected (functionally or communicatively)” or “connected” to another (eg, second) component, that component is It may be directly connected to the component, or may be connected through another component (eg, a third component).

In this document, "adapted to or configured to", depending on the context, for example, hardware or software "suitable for," "having the ability to," "modified to, Can be used interchangeably with ""made to," "capable of," or "designed to." In some circumstances, the expression “a device configured to” may mean that the device is “capable of” with other devices or parts. For example, the phrase “a processor configured (or configured to perform) A, B, and C” may refer to a dedicated processor (eg, an embedded processor), or one stored in a memory device (eg, memory 2830), for performing the corresponding operations. By executing the above programs, it may mean a general-purpose processor (eg, CPU or AP) capable of performing corresponding operations.

As used herein, the term “module” includes a unit composed of hardware, software, or firmware, and may be used interchangeably with terms such as, for example, logic, logic block, component, or circuit. can A “module” may be an integrally formed component or a minimum unit or a part that performs one or more functions. A “module” may be implemented mechanically or electronically, for example, known or to be developed, application-specific integrated circuit (ASIC) chips, field-programmable gate arrays (FPGAs), or It may include a programmable logic device.

At least a portion of an apparatus (eg, modules or functions thereof) or a method (eg, operations) according to various embodiments of the present disclosure includes instructions stored in a computer-readable storage medium (eg, memory 2830) in the form of a program module. can be implemented as When the instruction is executed by a processor (eg, the processor 2820), the processor may perform a function corresponding to the instruction. Computer-readable recording media include hard disks, floppy disks, magnetic media (eg, magnetic tape), optical recording media (eg, CD-ROM, DVD, magneto-optical media (eg, floppy disks), built-in memory, etc.) An instruction may include code generated by a compiler or code that can be executed by an interpreter.

Each of the components (eg, a module or a program module) according to various embodiments may be composed of a single or a plurality of entities, and some sub-components of the aforementioned sub-components may be omitted, or other sub-components may be included. may include more. Alternatively or additionally, some components (eg, a module or a program module) may be integrated into one entity to perform the same or similar functions performed by each corresponding component before being integrated. Operations performed by modules, program modules, or other components according to various embodiments are sequentially, parallelly, repetitively or heuristically executed, or at least some operations are executed in a different order, omitted, or other operations This can be added.

Claims (20)

In an electronic device,
housing;
a touch screen display disposed within the housing and exposed through a first portion of the housing;
a microphone disposed within the housing and exposed through a second portion of the housing;
at least one speaker disposed within the housing and exposed through a third portion of the housing;
wireless communication circuitry disposed within the housing;
a processor disposed within the housing and electrically coupled to the touchscreen display, the microphone, the at least one speaker, and the wireless communication circuitry; and
a memory disposed within the housing and electrically coupled to the processor, wherein the memory, when executed, causes the processor to:
Receive a user utterance through the microphone, wherein the user utterance includes a request for performing a task with the electronic device,
Transmitting data related to the user's utterance to an external server through the wireless communication circuit,
receive a response from the external server through the wireless communication circuit, wherein the response includes at least one path rule and a sample utterance corresponding to each of the at least one path rule, the at least one path A rule and a sample utterance corresponding to the at least one pass rule are selected or formed by the external server based on the user utterance,
performing an action on the received at least one pass rule,
Receiving a feedback input corresponding to feedback on the result of the performed action through the microphone or the touch screen display,
according to the feedback input being a negative input, displaying at least one sample utterance corresponding to at least one alternative pass rule received from the external server,
Storing instructions for receiving a user input for selecting one of the displayed at least one sample utterance through the microphone or the touch screen display, and performing an action corresponding to a pass rule corresponding to the selected sample utterance electronic device. The method according to claim 1,
and the displayed at least one sample utterance corresponding to the at least one alternative pass rule is included in the response received from the external server. The method according to claim 1,
The instructions allow the processor to
and, according to the feedback input being the negative input, transmit additional data associated with the user utterance to the external server to receive the at least one alternative pass rule from the external server. The method according to claim 1,
wherein the response comprises a plurality of sample utterances comprising a first sample utterance indicating a first intent and a second sample utterance indicating a second intent different from the first intent. The method according to claim 1,
The instructions, the processor,
and display one or more images associated with the displayed at least one sample utterance on the touchscreen display. 6. The method of claim 5,
and the one or more images represent a sequence of states associated with the displayed at least one sample utterance. The method according to claim 1,
The instructions, the processor,
determining an order for displaying the at least one sample utterance;
display the at least one sample utterance according to the determined order. 8. The method of claim 7,
the response includes a value corresponding to each of the at least one sample utterance;
The instructions, the processor,
and determine the order of the magnitudes of the values in the order indicative of the at least one sample utterance. 8. The method of claim 7,
The instructions, the processor,
The electronic device of claim 1, wherein a sample utterance corresponding to a first order among the at least one sample utterance is visually displayed differently from other sample utterances. A method of operating an electronic device, comprising:
Receive a user utterance through a microphone of the electronic device, wherein the user utterance includes a request for performing a task with the electronic device,
transmitting data related to the user's utterance to an external server through a wireless communication circuit of the electronic device;
receive a response from the external server through the wireless communication circuit, wherein the response includes at least one path rule and a sample utterance corresponding to each of the at least one path rule, the at least one path A rule and a sample utterance corresponding to the at least one pass rule are selected or formed by the external server based on the user utterance,
performing an action on the received at least one pass rule,
receiving a feedback input corresponding to feedback on a result of the performed action through the microphone or the touch screen display of the electronic device;
In response to the feedback input being a negative input, displaying at least one sample utterance corresponding to at least one alternative pass rule received from the external server,
receiving a user input for selecting one of the displayed at least one sample utterance through the microphone or the touch screen display, and performing an action corresponding to a pass rule corresponding to the selected sample utterance . 11. The method of claim 10,
The method of operating an electronic device, wherein the displayed at least one sample utterance corresponding to the at least one alternative pass rule is included in the response received from the external server. 11. The method of claim 10,
according to the feedback input being the negative input, transmitting additional data associated with the user's utterance to the external server to receive the at least one alternative pass rule from the external server. 11. The method of claim 10,
and the response includes a plurality of sample utterances including a first sample utterance indicating a first intent and a second sample utterance indicating a second intent different from the first intent. 11. The method of claim 10,
and displaying one or more images associated with the displayed at least one sample utterance on the touch screen display. 15. The method of claim 14,
and the one or more images represent a sequence of states associated with the displayed at least one sample utterance. 11. The method of claim 10,
determining an order for displaying the at least one sample utterance;
and displaying the at least one sample utterance according to the determined order. 17. The method of claim 16,
the response includes a value corresponding to each of the at least one sample utterance;
and the electronic device determines the order of magnitudes of the values as the order of displaying the at least one sample utterance. A storage medium storing computer-readable instructions, wherein the instructions, when executed by a processor of an electronic device, cause the electronic device to:
Receive a user utterance through a microphone of the electronic device, wherein the user utterance includes a request for performing a task with the electronic device,
Transmitting data related to the user's utterance to an external server through a wireless communication circuit of the electronic device,
receive a response from the external server through the wireless communication circuit, wherein the response includes at least one path rule and a sample utterance corresponding to each of the at least one path rule, the at least one path A rule and a sample utterance corresponding to the at least one pass rule are selected or formed by the external server based on the user utterance,
performing an action on the received at least one pass rule,
receiving a feedback input corresponding to feedback on a result of the performed action through the microphone or the touch screen display of the electronic device;
according to the feedback input being a negative input, displaying at least one sample utterance corresponding to at least one alternative pass rule received from the external server,
receiving a user input for selecting one of the displayed at least one sample utterance through the microphone or the touch screen display, and performing an action corresponding to a pass rule corresponding to the selected sample utterance. 19. The method of claim 18,
and the indicated at least one sample utterance corresponding to the at least one alternative pass rule is included in the response received from the external server. 19. The method of claim 18,
When the instructions are executed by a processor of an electronic device, according to the feedback input being the negative input, additional data associated with the user utterance to the external server to receive the at least one alternative pass rule from the external server. A storage medium to transmit

Images