KR20240045031A - Electronic device, operating method, and storage medium for processing utterance not including a predicate - Google Patents

Abstract

An electronic device according to an embodiment of the present disclosure may include a microphone, a communication circuit, memory, and a processor. The memory can store instructions. The instructions, when executed by the processor, cause the electronic device to receive a voice command through the microphone, determine whether text corresponding to the voice command consists of a named entity, and , execution information for executing the voice command can be obtained based on the entity name, and the voice command can be executed using the execution information. The execution information may include domain information indicating an application executing the voice command or intent information indicating a function of the application requested by the voice command.

Description

Electronic device, operating method, and storage medium for processing utterances not containing predicates {ELECTRONIC DEVICE, OPERATING METHOD, AND STORAGE MEDIUM FOR PROCESSING UTTERANCE NOT INCLUDING A PREDICATE}

The present disclosure relates to an electronic device, operating method, and storage medium for processing utterances that do not contain predicates.

Voice recognition technology is a technology that analyzes the user's utterances and determines the meaning of the utterances. Speech recognition technology includes wake-up technology that recognizes speech for the first time, automatic speech recognition (ASR) technology that converts speech into identifiable text, and natural language understanding that extracts meaning from the converted text. (natural language understanding) skills may be included. Various electronic devices equipped with a voice assistant function that controls the device based on voice commands using voice recognition technology are being provided.

According to one embodiment, an electronic device may include a microphone, communication circuitry, memory, and a processor. The memory can store instructions. When executed by the processor, the memory may enable the electronic device to receive voice commands through the microphone. The instructions, when executed by the processor, may cause the electronic device to determine whether text corresponding to the voice command consists of a named entity. When executed by the processor, the instructions may cause the electronic device to obtain execution information for executing the voice command based on the entity name. The execution information may include domain information indicating an application executing the voice command or intent information indicating a function of the application requested by the voice command. When executed by the processor, the instructions may cause the electronic device to execute the voice command using the execution information.

According to one embodiment, a method of operating an electronic device may include receiving a voice command through a microphone of the electronic device. The method may include determining whether text corresponding to the voice command consists of one named entity. The method may include obtaining execution information for executing a voice command based on the entity name. The execution information may include domain information indicating an application executing the voice command or intent information indicating a function of the application requested by the voice command. The method may include executing the voice command using the execution information.

According to one embodiment, a non-transitory computer-readable storage medium may store a program for performing a method of operating an electronic device. The method of operating the electronic device may include receiving a voice command through a microphone of the electronic device. The method may include determining whether text corresponding to the voice command consists of one named entity. The method may include obtaining execution information for executing a voice command based on the entity name. The execution information may include domain information indicating an application executing the voice command or intent information indicating a function of the application requested by the voice command. The method may include executing the voice command using the execution information.

1 is a block diagram showing the configuration of an electronic device according to an example.
Figure 2 is a functional block diagram of an electronic device according to an example.
FIG. 3 is a diagram illustrating execution information acquired by an electronic device according to an example.
FIG. 4 is a diagram illustrating execution information acquired by an electronic device according to an example.
Figure 5 is a flowchart showing the operation of an electronic device according to an example.
FIG. 6 is a diagram illustrating a graphical user interface provided by an electronic device according to an example.
FIG. 7 is a diagram illustrating a graphical user interface provided by an electronic device according to an example.
FIG. 8 is a diagram illustrating a graphical user interface provided by an electronic device according to an example.
FIG. 9 is a diagram illustrating a graphical user interface provided by an electronic device according to an example.
Figure 10 is a flowchart showing the operation of an electronic device according to an example.
Figure 11 is a block diagram of an electronic device in a network environment according to an example.
Figure 12 is a block diagram showing an integrated intelligence system according to an example.
Figure 13 is a diagram showing how relationship information between concepts and actions is stored in a database, according to an example.
FIG. 14 is a diagram illustrating a user terminal displaying a screen for processing voice input received through an intelligent app, according to an example.
In relation to the description of the drawings, identical or similar reference numerals may be used for identical or similar components.

Hereinafter, with reference to the drawings, embodiments of the present disclosure will be described in detail so that those skilled in the art can easily practice them. However, the present disclosure may be implemented in many different forms and is not limited to the embodiments described herein. In relation to the description of the drawings, identical or similar reference numerals may be used for identical or similar components. Additionally, in the drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for clarity and brevity.

FIG. 1 is a block diagram 100 showing the configuration of an electronic device 101 according to an example.

Referring to FIG. 1, the electronic device 101 may include a microphone 110, a communication circuit 120, a memory 130, and a processor 140, but is not limited thereto. The electronic device 101 may further include other components (eg, an input device and/or a display) not shown in FIG. 1 .

The microphone 110 can receive sound from outside the electronic device 101. For example, the microphone 110 may receive a voice command from the user of the electronic device 101. For example, the microphone 110 may receive user speech.

The communication circuit 120 may support communication between the electronic device 101 and an external electronic device (eg, a server). The communication circuit 120 may support transmission and reception of data or signals between the electronic device 101 and an external electronic device. For example, the communication circuit 120 may transmit a voice command received through the microphone 110 of the electronic device 101 to the voice assistant server. Communication circuitry 120 may receive a response for executing a voice command from a voice assistant server. In the present disclosure, a response for executing a voice command may be referred to as execution information for executing a voice command. Execution information may include, for example, domain information indicating the application executing the voice command and intent information indicating the function of the application requested by the voice command.

The memory 130 may store one or more instructions for operating the electronic device 101. The memory 130 may store data or information necessary for the operation of the electronic device 101. For example, memory 130 may store at least one application. For example, the memory 130 may store usage history information of a named entity in at least one application. For example, entity name usage history information may include the number of times and/or time at which the entity name was used.

Processor 140 may be operatively connected to microphone 110, communication circuit 120, and memory 130. Being operatively connected between components may mean that the components are functionally connected or communicatively connected. For example, operationally connected components can exchange data with each other.

The processor 140 may execute one or more instructions stored in the memory 130. For example, processor 140 may provide a voice assistant function by executing one or more instructions stored in memory 130. The voice assistant function may provide responses to user requests included in voice commands. In other words, the processor 140 executes at least some of the functions performed by the voice assistant server (e.g., automatic speech recognition (ASR) and natural language understanding) by executing one or more instructions stored in the memory 130. , NLU)) can be implemented.

The processor 140 may receive a user's voice command through the microphone 110. The processor 140 may convert a voice command received through the microphone 110 into text based on automatic voice recognition. In one embodiment, the processor 140 may analyze the converted text based on natural language understanding to determine the domain representing the application executing the voice command and the intent representing the function of the application requested by the voice command.

The processor 140 may determine whether the text corresponding to the voice command consists of one named entity. For example, if the text corresponding to the voice command does not include a predicate but includes one entity name, the processor 140 may determine that the text consists of one entity name.

In one embodiment, when the text consists of one entity name, the processor 140 may search the database based on the entity name. Databases may contain public data and personal data. Public data refers to entity names used by the public, such as entity names of media contents, locations, organizations, or domain specific information (e.g., Internet of Things (IoT) mode). It can be included. For example, public data can be repeatedly updated with the latest data at specified times through an external server.

Personal data includes data specific to an individual (e.g., user of electronic device 101), such as activity, pet name, IoT device name, contact information, or personal media content information. It can be included. In one embodiment, the processor 140 may collect a registered entity name as personal data based on a user input received through an input device (eg, including the microphone 110) of the electronic device 101.

Public data and personal data may include category information, domain information, and/or intent information associated with the entity name.

In one embodiment, the processor 140 uses the entity name extracted from the entity name usage history in each application and the domain or extracted entity name corresponding to each application, based on the entity name usage history in each of the at least one application. The intent can be associated and stored in the database. Hereinafter, the entity name extracted from the entity name use history in each application and the domain or intent stored in connection with it may be referred to as designated entity name information. Designated entity name information may be treated as personal data, for example. For example, if the entity name included in the text corresponds to the entity name included in the specified entity name information, the processor 140 may obtain domain information or intent information included in the specified entity name information.

For example, data stored in a database may store not only string information of the entity name but also pronunciation string information. In one embodiment, when searching a database based on an entity name, the processor 140 may perform a search based on not only a string of the entity name but also a pronunciation sequence. The processor 140 can also search for entity names that have a pronunciation sequence similar to the entity name to be searched.

In one embodiment, the processor 140 may search a database based on an entity name to obtain execution information for executing a voice command. The execution information may include domain information indicating the application executing the voice command or intent information indicating the function of the application requested by the voice command. For example, execution information may include category information to which the entity name belongs. For example, the processor 140 may search a database based on the entity name to obtain at least one category, at least one domain, and/or at least one intent corresponding to the entity name.

In one embodiment, when a plurality of categories corresponding to an entity name are searched based on public data, the processor 140 may obtain ratio information for each category. For example, the ratio information for each category may indicate the ratio at which an entity name is used in each category among a plurality of categories. Ratio information for each category may, for example, indicate the probability that an entity name will be used in each category among a plurality of categories. For example, for the first entity name, two categories of 'TV shows' and 'Music' may be searched based on public data. For example, the processor 140 may obtain the ratio of the first entity name used in the 'TV shows' category and the ratio used in the 'Music' category.

In one embodiment, the processor 140 may determine a category corresponding to the entity name based on the obtained ratio information. For example, the processor 140 may determine that the category with the highest ratio is the category corresponding to the entity name. For example, for the first entity name, if the proportion used in the 'TV shows' category is about 60% and the proportion used in the 'Music' category is about 40%, the processor 140 selects the first entity name The category corresponding to can be determined as the 'TV shows' category.

In one embodiment, the processor 140 may obtain execution information corresponding to the determined category. For example, if the category of the first entity name is 'TV shows', the first domain (e.g., the first app) executes the voice command, and if the category of the first entity name is 'Music', the first domain (e.g., the first app) executes the voice command. A second domain (e.g., a second app) that is different from the first domain can execute voice commands. For example, by determining the category of the first entity name as 'TV shows', the processor 140 may obtain domain information indicating the first domain and intent information indicating the function of the first domain. For example, as the processor 140 determines the category of the first entity name as 'Music', the processor 140 may obtain domain information indicating the second domain and intent information indicating the function of the second domain.

In one embodiment, when a plurality of categories corresponding to an entity name are searched based on public data, the processor 140 may provide a user interface that asks the user which category among the plurality of categories he or she wants information belonging to. . For example, the processor 140 may provide a user interface by displaying a graphical user interface on the display of the electronic device 101. For example, for the first entity name, two categories of 'TV shows' and 'Music' may be searched based on public data. For example, the processor 140 may provide a user interface that asks the user which category of 'TV shows' or 'Music' the first entity name belongs to.

In one embodiment, if a plurality of categories corresponding to the entity name are searched and the category corresponding to the entity name cannot be determined as one based on ratio information for each category, the processor 140 allows the user to use the user interface through the user interface. You can check the intended category. For example, the proportion of the first entity name used in the 'TV shows' category may be about 52%, and the proportion used in the 'Music' category may be about 48%. In this case, the processor 140 may not be able to determine the category intended by the user because the difference in percentages used between the two categories is within a specified range (eg, 5%). For example, the processor 140 may provide a user interface that asks the user which category of 'TV shows' or 'Music' the user wants. The processor 140 may obtain execution information corresponding to the category selected by user input on the user interface.

In one embodiment, the processor 140 searches for a plurality of categories corresponding to the entity name based on public data, and if it is possible to specify one category among the plurality of categories based on personal data, it selects the specified category Corresponding execution information can be obtained. For example, if there is one category corresponding to the entity name as a result of a search based on personal data, the processor 140 may specify the category as the category corresponding to the entity name. For example, for the first entity name, two categories of 'TV shows' and 'Music' may be searched based on public data, and only the 'TV shows' category may be searched based on personal data. In this case, the processor 140 may specify the category corresponding to the first entity name as 'TV shows' and obtain execution information corresponding to 'TV shows'.

For example, if there are a plurality of categories corresponding to the entity name as a result of a search based on personal data, the processor 140 may specify the most frequently used category among the plurality of categories as the category corresponding to the entity name. . For example, for the first entity name, two categories, 'TV shows' and 'Music', are searched based on personal data, the number of uses of the 'TV shows' category is 2, and the number of uses of the 'Music' category are 2. It may be once. In this case, the processor 140 may specify the category corresponding to the first entity name as 'TV shows' and obtain execution information corresponding to 'TV shows'.

In one embodiment, the processor 140 may execute a voice command using the obtained execution information. For example, the processor 140 may execute a voice command using the domain indicated by the acquired domain information. For example, the processor 140 may execute a voice command by executing the function of a specific domain indicated by the acquired intent information. In the present disclosure, executing a voice command by the electronic device 101 or the processor 140 may be referred to as performing an action corresponding to the voice command or performing a function corresponding to the user's intention. You can.

In one embodiment, the processor 140 may determine whether an action performed by executing a voice command corresponds to the intention of the user of the electronic device 101. For example, when the processor 140 receives negative feedback from the user (e.g., inputting a 'cancel' or 'back' button, or re-uttering the same voice command), the processor 140 may control the electronic device 101 by executing the voice command. It may be determined that the action performed does not correspond to the user's intention. For example, processor 140 may receive positive feedback from the user (e.g., uttering a voice command to perform an action subsequent to the performed action, no response to keep the action performed, or a "thank you" or "well done"). When an utterance containing a positive word such as "I did it" is received, the action performed by the electronic device 101 by executing the voice command may be determined to correspond to the user's intention.

In one embodiment, if the action performed by executing the voice command corresponds to the intention of the user of the electronic device 101, the processor 140 stores the information used in executing the voice command as personal data. It can be saved in . For example, the information used to execute a voice command may include the entity name included in the voice command, the category corresponding to the entity name, the number of times the entity name was uttered, the domain in which the action was performed, or the action performed in the domain. It may include at least one of the following intentions: For example, information used to execute a voice command may be stored in association with the entity name and other information. Accordingly, when the processor 140 receives a voice command including the same entity name, the processor 140 may obtain information (eg, category, number of utterances, domain, and/or intent) associated with the entity name stored in the database. For example, the processor 140 may perform an operation corresponding to a voice command based on the acquired domain information or intent information.

For example, there may be no data retrieved from a database based on the entity name. In this case, the processor 140 may provide a user interface that includes a list of commonly used intents. The intent list may contain at least one intent. At least one intent included in the intent list may correspond to the function of at least one domain. At least one intent included in the intent list may be unrelated to the entity name. At least one intent included in the intent list is, for example, the most used by voice assistant service users (or electronic devices providing the voice assistant service) among the domains or intents supported by the electronic device 101. The domain or intention may be extracted. The processor 140 may perform an action corresponding to the intention selected by user input on the user interface.

For example, data retrieved from a database based on an entity name may contain multiple intents. In this case, processor 140 may provide a user interface that includes an intent list associated with the entity name. The intent list may include at least one intent associated with the entity name. At least one intent included in the intent list may correspond to the function of at least one domain associated with the entity name. The processor 140 may perform an operation corresponding to the intention selected by the user input to the user interface.

In one embodiment, when the text corresponding to the received voice command consists of one entity name, the processor 140 may determine whether the domain or intention can be specified based on the text. For example, when there is only one domain or intent corresponding to an entity name included in the text, the processor 140 may determine that the domain or intent can be specified based on the text. For example, when there are multiple domains or intents corresponding to an entity name included in the text, the processor 140 may determine that the domain or intent cannot be specified based on the text.

In one embodiment, when the processor 140 can specify a domain or intent based on text, the processor 140 may perform an action corresponding to a voice command based on the specified domain or intent. In one embodiment, if the domain or intent cannot be specified based on the text, the processor 140 may specify the domain or intent using execution information obtained by searching the database based on the entity name included in the text. You can. In one embodiment, the processor 140 may perform an action corresponding to a voice command based on a specified domain or intention using execution information.

In one embodiment, the processor 140 may receive a user input specifying the domain or intention in the domain to perform an action on at least one entity name through the input device of the electronic device 101. . Based on the received user input, the processor 140 may be set to perform an operation corresponding to the voice command according to a designated domain or designated intent when a voice command including one of at least one entity name is received. After the above setting, when the processor 140 receives a voice command consisting of an entity name with a specified domain or intent through the microphone 110, the processor 140 performs an operation corresponding to the voice command according to the domain or intent specified for the entity name. It can be done.

For example, the processor 140 may provide a user interface that specifies a domain in which to perform an operation for at least one entity name. At least one entity name may include a first entity name and a second entity name. For example, the processor 140 may receive a first user input designating a domain to perform an operation corresponding to a voice command for a first entity name as a first domain (eg, a first app). For example, the processor 140 may receive a second user input designating a domain to perform an operation corresponding to a voice command for a second entity name as a second domain (eg, a second app). The processor 140 may set domains to perform operations corresponding to voice commands for the first entity name and the second entity name based on the first user input and the second user input, respectively. Thereafter, when the processor 140 receives a voice command including the first entity name, the processor 140 may perform an operation corresponding to the voice command using the first domain. Additionally, when the processor 140 receives a voice command including a second entity name, the processor 140 may perform an operation corresponding to the voice command using the second domain.

For example, the processor 140 may store information about the domain or intent specified by user input for at least one entity name in a database. For example, the processor 140 may associate an entity name with a domain or intention designated corresponding to the entity name and store the associated entity name in a database.

The electronic device 101 according to one embodiment builds a database capable of detecting categories, domains, and/or intentions corresponding to entity names, and searches the database based on entity names to identify user utterances without predicates. Also, by identifying the user's desired domain or intention, an operation corresponding to the user's intention can be performed. The electronic device 101 according to one embodiment may identify intention information related to the user's utterance and provide it to the user even when the user's intention is unclear. The electronic device 101 according to one embodiment can identify the user's desired domain or intention using personal data and perform an operation corresponding to the user's intention, even when there are multiple categories corresponding to the user's utterance.

FIG. 2 is a functional block diagram 200 of an electronic device according to an example.

An electronic device (eg, the electronic device 101 of FIG. 1) according to an example may include at least some of the modules (or units) shown in FIG. 2. Each module (or unit) shown in FIG. 2 may be a set of instructions to implement each function. The operations of each module described below may be performed by at least a hardware component (eg, the processor 140 of FIG. 1) of the electronic device 101 shown in FIG. 1. For example, the electronic device 101 includes an ASR module 210, an NLU module 220, an execution information detection module 230, a predicate presence/absence determination unit 240, a user data collection unit 241, and/or a user It may include a data transmission unit 243.

The ASR module 210 may convert the user's voice command received through the microphone of the electronic device 101 (e.g., the microphone 110 of FIG. 1) into text.

The NLU module 220 may determine the domain or intent based on the converted text. A domain may represent an application that executes voice commands. The intent may indicate the functionality of the application that the voice command is requesting. The NLU module 220 may include a personalized intent processor 221, a domain classifier 223, a category resolver 225, or an intent classifier 227. .

The personalized intent processing unit 221 may call the predicate presence/absence determination unit 240 to determine whether the user's utterance is an utterance without a predicate. The predicate presence/absence determination unit 240 may determine whether the text converted from the received voice command includes the predicate. The predicate presence/absence determination unit 240 may determine whether the text converted from the received voice command consists of only the entity name and no predicate. The personalized intention processing unit 221 can obtain execution information for executing a voice command through the execution information detection module 230 only when the text corresponding to the voice command consists of a single entity name without a predicate. The personalized intention processing unit 221 may query the execution information detection module 230 or the execution information determination unit 232 of the execution information detection module 230 to obtain execution information for executing the voice command. Execution information may include, for example, category information, domain information, intent information, and/or action information. Operation information may be information indicating an operation corresponding to the intention or the function of the domain (e.g., application) indicated by the intention. The personalized intent processing unit 221 may transmit the entity name included in the text to the execution information detection module 230 or the execution information determination unit 232 when making a query.

The personalized intent processing unit 221 operates when the user's utterance includes only one entity name without a predicate and the domain to execute the voice command is clear (e.g. entity name If the domain corresponding to is determined (or specified) to be one, the electronic device 101 can execute a voice command using the corresponding domain. The personalized intent processing unit 221 executes the entity name when the user's utterance includes only one entity name without a predicate and the domain to execute the voice command is unclear (e.g., when there are multiple domains corresponding to the entity name). It can be transmitted to the information detection module 230 and the execution information for executing the voice command can be obtained from the execution information detection module 230. The electronic device 101 may perform an operation corresponding to the user's intention using the acquired execution information.

Even if the user's utterance includes a predicate, the personalized intention processing unit 221, if there is execution information obtained from the execution information detection module 230 or the execution information determination unit 232, sends the execution information to the domain classifier 223, It can be used by passing it to the category resolver 225 and/or the intent classifier 227.

The domain classifier 223 is a sub-module of the NLU module 220. The domain classifier 223 can deliver the utterance or text corresponding to the utterance to each domain processor. A domain may be referred to, for example, as an application area such as a phone, message, or gallery, or an application area such as media contents.

The category resolver 225 may determine the detailed domain (e.g., application) by the domain classifier 223, but the detailed domain may be determined by the domain classifier 223 to be an application area, such as media content. may not be decided. In this case, the category resolver 225 may need to determine in which detailed domain the utterance should be processed. For this decision, the category resolver 225 may use category information or domain information obtained through the personalized intent processing unit 221.

The intent classifier 227 can classify the user's intent by analyzing the utterance. For example, the intention classifier 227 may determine the user's intention by performing grammatical analysis and/or semantic analysis on the text corresponding to the utterance. The intent classifier 227 can determine the intent of an utterance whose intent is not identified in the personalized intent processing unit 221. For example, if the entity name is not a designated entity name whose execution information is detected through the execution information detection module 230, the intent of the utterance containing the entity name may not be recognized by the personalized intent processing unit 221. The intent classifier 227 may use an algorithm that can process more complex and larger-scale operations than the classification algorithm used in the personalized intent processing unit 221.

If the action performed in response to the user's utterance corresponds to the user's intention, the intent classifier 227 may transmit the utterance usability information to the execution information detection module 230 or the data collection unit 234. Utterance usability information is information used in the execution of a voice command, and includes the entity name included in the voice command, the category corresponding to the entity name, the number of times the entity name was uttered, the domain in which the action was performed, or the action performed in the domain. May contain at least one of the corresponding intents.

The execution information detection module 230 may include a processing unit 231, a search unit 233, and a DB 235. The processing unit 231 may include an execution information determination unit 232 and a data collection unit 234. The execution information determination unit 232 may search the DB 235 through the search unit 233 based on the entity name transmitted from the personalized intention processing unit 221. The search unit 233 can search using string information and pronunciation string information of the entity name. DB 235 may include personal data 235-1 and public data 235-2. Personal data 235-1 and public data 235-2 may be collected by the data collection unit 234 in different ways. Data stored in the DB 235 may also include pronunciation sequence information of the entity name.

The data collection unit 234 can repeatedly update the public data 235-2 with the latest data at a designated point in time. Public data 235-2 may include entity names used by the public, such as entity names of media content, location, organization, and/or domain specific information (e.g., IoT mode). You can. For example, the public data 235-2 may include an entity name, pronunciation sequence information of the entity name, and/or category information.

The data collection unit 234 may collect personal data 235-1 from the NLU module 220 or the user data transmission unit 243. Personal data 235-1 may be personal (e.g., user of electronic device 101), such as activity, pet name, IoT device name, contact information, or personal media content information. ) may include specialized data. Personal data 235-1 may include the user's app usage history, or the user's app usage history by time and/or location. For example, personal data 235-1 may be collected by the user data collection unit 241 within the electronic device 101. The user data transmission unit 243 may transmit the personal data 235-1 collected by the user data collection unit 241 to the data collection unit 234. For example, the data collection unit 234 may collect personal data 235-1 by receiving speech usability information from the NLU module 220. For example, the personal data 235-1 may include a user identifier, entity name, pronunciation sequence information of the entity name, category information, utterance count information, preference information, domain information, and/or intent information.

For example, the user data collection unit 241 may collect data when receiving a user input for registering or changing data. For example, when a user input for registering or changing a pet name is received in an IoT pet service, the user data collection unit 241 may collect pet name information.

For example, the user data collection unit 241 may collect information about the content when receiving a user input indicating a preference for the content. For example, when content is played for a long time (eg, longer than a threshold time), the user data collection unit 241 may collect information about the played content.

For example, the user data collection unit 241 may collect the selected intention information when receiving a user input for selecting one from an intention list including a plurality of intention information provided by a voice assistant. For example, even if the execution information determination unit 232 specifies the category corresponding to the entity name, the personalized intention processing unit 221 may be ambiguous as to what operation to perform. In this case, the personalized intent processing unit 221 may provide a user interface that allows the user to select a desired action. For example, when the user utters “Gangnam Station,” the execution information determination unit 232 may recognize that the entity name “Gangnam Station” belongs to the “location” category. The execution information determination unit 232 may transmit information indicating that 'Gangnam Station' belongs to the 'location' category to the personalized intention processing unit 221. Since there are a plurality of actions that can be performed using information in the 'location' category of “Gangnam Station,” the personalized intention processing unit 221 can provide candidates for a plurality of actions through the user interface. For example, the personalized intention processing unit 221 may say, ‘Tell me a restaurant near “Gangnam Station” in the first app,’ ‘Give me directions to “Gangnam Station” in the second app,’ and ‘A restaurant near “Gangnam Station” in the second app. A user interface including 'Tell me' may be provided. The personalized intention processing unit 221 may perform the selected action in response to receiving a user input for selecting one of the plurality of provided action candidates. The user data collection unit 241 may collect information about the selected action (e.g., domain information or intention information for executing the selected action) in response to receiving a user input for selecting one of the plurality of provided action candidates. there is.

For example, when receiving a user input specifying a domain corresponding to an entity name, the user data collection unit 241 may collect the entity name and execution information corresponding thereto. For example, the user data collection unit 241 may collect execution information corresponding to a designated entity name based on app usage history by time and location. Execution information may include domain information, intent information, or operation information. For example, the user data collection unit 241 transmits the designated entity name-execution information pair to the data collection unit 234, and the data collection unit 234 transmits the designated entity name-execution information pair to the DB 235. You can save it. The designated entity name-execution information pair may be stored as personal data 235-1 or may be stored as separate data.

The execution information determination unit 232 may be called by the personalized intention processing unit 221. The execution information determination unit 232 may search the DB 235 through the search unit 233 based on words or phrases included in the text corresponding to the user's utterance. For example, the execution information determination unit 232 may search the DB 235 based on the entity name included in the text through the search unit 233. For example, the execution information determination unit 232 responds to an utterance by determining not only tagging information (e.g., category information) indicating which category the entity name is included in, but also a domain or intention to perform an operation corresponding to the utterance. Execution information for performing an action can be determined. The execution information determination unit 232 may transmit the determined execution information to the personalized intention processing unit 221, allowing the personalized intention processing unit 221 to perform an operation corresponding to the utterance using the execution information.

For example, if there are two or more categories corresponding to the entity name searched in the DB 235, the execution information determination unit 232 may be unable to determine which category the utterance belongs to. In this case, the execution information determination unit 232 may return ratio information for each category to the personalized intention processing unit 221. For example, the ratio information for each category may indicate the ratio at which an entity name is used in each category among a plurality of categories. For example, if the personalized intent processing unit 221 cannot specify a category based on ratio information for each category, it may provide a user interface that asks the user which category of the plurality of categories he or she wants information about. .

For example, when an input for selecting a specific domain or an utterance input clearly including a domain such as “Play A in the first app” is received through the user interface, the user data collection unit 241 determines the entity name. Usability information about 'A' can be collected. For example, usability information for entity name 'A' may include domain information called 'first app' and intent information called 'play'. The usability information of an individual (user of the electronic device 101) collected by the user data collection unit 241 may be stored as personal data 235-1 of the DB 235 through the data collection unit 234. . After the personal data 235-1 is added, the execution information determination unit 232 can determine the user's intention using the added personal data 235-1 for the same entity name.

For example, when there is no personal usability information for an entity name, the execution information determination unit 232 personalizes a plurality of possible category information searched based on the public data 235-2 and ratio information for each category. It can be returned to the intention processing unit 221. For example, when personal usability information for an entity name is added, the execution information determination unit 232 combines the public data 235-2 and the personal data 235-1 to determine the user's desired domain or intent. You can decide. The execution information determination unit 232 may return execution information including meta information for performing an action corresponding to the utterance along with the determined domain or intent information to the personalized intent processing unit 221. The personalized intention processing unit 221 may perform an operation corresponding to the user's intention using execution information.

In the above example, the electronic device 101 includes all of the modules shown in FIG. 2, but the present invention is not limited thereto. According to various examples, the electronic device 101 may include only a portion of the modules shown in FIG. 2 .

As an example, the ASR module 210, NLU module 220, predicate presence/absence determination unit 240, user data collection unit 241, and user data transmission unit 243 are implemented and executed in the electronic device 101. The information detection module 230 may be implemented as a server external to the electronic device 101. In this case, the electronic device 101 may communicate with the execution information detection module 230 through the communication circuit 120. The personalized intent processing unit 221 of the electronic device 101 may query the execution information detection module 230 through the communication circuit 120 and receive a response.

As another example, the user data collection unit 241 and the user data transmission unit 243 are implemented in the electronic device 101, and the ASR 210, NLU 220, and predicate presence determination unit 240 are implemented in the electronic device 101. (101) may be implemented as an external first server, and the execution information detection module 230 may be implemented as a second server external to the electronic device 101. In this case, the electronic device 101 may communicate with the first server through the communication circuit 120 and transmit a voice command received through the microphone 110 to the first server through the communication circuit 120. there is. The electronic device 101 may communicate directly with the second server through the communication circuit 120 or indirectly through the first server. The personalized intent processing unit 221 of the first server may query the execution information detection module 230, receive a response, and process the response to be provided to the electronic device 101. The electronic device 101 may receive a processed response from the first server and execute a voice command based on the received response.

FIG. 3 is a diagram 300 showing execution information acquired by an electronic device according to an example. The operation of an electronic device described later (eg, the electronic device 101 of FIG. 1) may be performed by, for example, a processor (eg, the processor 140 of FIG. 1). Public data 310 of FIG. 3 may correspond to public data 235-2 of FIG. 2. The response 320 in FIG. 3 is a response date returned by the execution information determination unit (e.g., the execution information determination unit 232 in FIG. 2) to the personalized intention processing unit (e.g., the personalized intention processing unit 221 in FIG. 2). You can.

For example, the electronic device 101 may receive a voice command that includes a first entity name and does not include a predicate. The electronic device 101 may search a DB (eg, DB 235 in FIG. 2) based on the first entity name. For example, public data 310 related to the first entity name may exist in the DB 235, but personal data related to the first entity name may not exist. According to the public data 310, there may be two categories corresponding to the first entity name, the 'TV shows' category and the 'Music' category.

The electronic device 101 may not be able to determine which category of information the user wants among the 'TV shows' category and the 'Music' category in relation to the first entity name. In this case, the electronic device 101 can obtain usage ratio information for each category. For example, the electronic device 101 may obtain rate information for each category, indicating the rate at which the first entity name is used in the 'TV shows' category and the rate at which the first entity name is used in the 'Music' category. . For example, the electronic device 101 may obtain the response 320 as execution information for the first entity name. The response 320 may include category information corresponding to the first entity name. If there are multiple categories corresponding to the first entity name, the response 320 may further include ratio information for each category. The response 320 may include domain information or intent information for each category.

The electronic device 101 may determine the user's intention based on ratio information for each category. As in the example of FIG. 3, when the rate at which the first entity name is used in the 'TV shows' category and the rate at which the first entity name is used in the 'Music' category are the same, the electronic device 101 displays 'TV' for the first entity name. You can provide a user interface to query which category of information you want between the 'shows' category and the 'Music' category. For example, the electronic device 101 may receive a user input for selecting the 'TV shows' category or the 'Music' category. The electronic device 101 may determine the category selected by the user input as the category corresponding to the first entity name. The electronic device 101 may perform an operation corresponding to the user's intention using domain information and intention information according to the determined category.

FIG. 4 is a diagram 400 showing execution information acquired by an electronic device according to an example. The operation of an electronic device described later (eg, the electronic device 101 of FIG. 1) may be performed by, for example, a processor (eg, the processor 140 of FIG. 1). The public data 410 of FIG. 4 may correspond to the public data 235-2 of FIG. 2. Personal data 420 in FIG. 4 may correspond to personal data 235-1 in FIG. 2. The response 430 in FIG. 4 is a response date returned by the execution information determination unit (e.g., the execution information determination unit 232 in FIG. 2) to the personalized intention processing unit (e.g., the personalized intention processing unit 221 in FIG. 2). You can.

For example, the electronic device 101 may receive a voice command that includes a first entity name and does not include a predicate. The electronic device 101 may search a DB (eg, DB 235 in FIG. 2) based on the first entity name. For example, public data 410 and personal data 420 related to the first entity name may exist in the DB 235. According to the public data 410, there are two categories corresponding to the first entity name, the 'TV shows' category and the 'Music' category, and the ratio information for each category is about 50% for the 'TV shows' category and 'Music' category. ' Category may be about 50%. Ratio information for each category may, for example, indicate the probability that an entity name will be used in each category among a plurality of categories. According to the personal data 420, the 'TV shows' category was used for the first entity name, the number of times the first entity name was uttered (or used) in the 'TV shows' category was 1, and the number of times the first entity name was uttered (or used) in the 'TV shows' category was 1. 1 I have performed a search using the entity name as a search term.

The electronic device 101 may obtain a response 430 as execution information for the first entity name. Response 430 may be determined by combining public data 410 and personal data 420. According to the public data 410, the category corresponding to the first entity name may not be specified as one. When the public data 410 and the personal data 420 are combined, the category corresponding to the first entity name can be specified as 'TV shows'. The response 430 may include category information corresponding to the first entity name. When the category corresponding to the first entity name is determined based on the personal data 420, the response 430 executes a voice command including information on the number of times the first entity name was uttered in the determined category and the first entity name. It may contain domain information and intent information. The electronic device 101 may perform an operation corresponding to the user's intention using the domain information and intention information included in the response 430. For example, the electronic device 101 may perform a search in App A using the first entity name as a search word.

FIG. 5 is a flowchart 500 showing the operation of an electronic device according to an example. In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel. The operation of an electronic device described later (eg, the electronic device 101 of FIG. 1) may be performed by, for example, a processor (eg, the processor 140 of FIG. 1).

In operation 501, the electronic device 101 may receive a user's speech through a microphone (eg, the microphone 110 of FIG. 1).

In operation 503, the electronic device 101 may convert the received utterance into text. For example, operation 503 may be performed by ASR module 210 of FIG. 2 .

In operation 505, the electronic device 101 may determine whether the converted text consists of one entity name. For example, the electronic device 101 may determine whether the text includes only an entity name without a predicate. For example, operation 505 may be performed by the personalized intent processing unit 221 of FIG. 2 or the predicate presence/absence determination unit 240 called by the personalized intent processing unit 221. When the electronic device 101 determines that the text does not consist of one entity name, it may perform operations 521, 523, and 525. If the electronic device 101 determines that the text consists of one entity name, it may perform operation 507.

In operation 507, the electronic device 101 may determine whether the entity name constituting the text is a designated entity name. For example, the electronic device 101 may query the execution information determination unit whether the entity name constituting the text is a designated entity name. The execution information determination unit may be the execution information determination unit 232 of FIG. 2 . The designated entity name may be referenced by the execution information determination unit 232 as an entity name for which corresponding execution information can be obtained. Execution information may include domain information indicating the application executing the utterance (or voice command) or intent information indicating the function of the application requested by the utterance. When the electronic device 101 determines that the entity name constituting the text is a designated entity name, operation 509 may be performed. When the electronic device 101 determines that the entity name constituting the text is not the designated entity name, it may perform operations 521, 523, and 525.

In operation 521, the electronic device 101 may determine a domain corresponding to the text through text analysis. In operation 523, the electronic device 101 may determine the intent corresponding to the text through text analysis. For example, the electronic device 101 may perform grammatical analysis and/or semantic analysis by synthesizing entity names and/or predicates included in the converted text. Operation 521 may be performed, for example, by domain classifier 223 and category solver 225 of FIG. 2 . Operation 523 may be performed, for example, by intent classifier 227 of FIG. 2 . In operation 525, the electronic device 101 may execute an operation corresponding to the intention determined in operation 523.

For example, operation 509 and subsequent operations may be performed by the personalized intent processor 221 of FIG. 2 .

In operation 509, the electronic device 101 may obtain execution information from the execution information determination unit 232. For example, the execution information determination unit 232 may obtain execution information corresponding to the specified entity name searched from a database (eg, DB 235 in FIG. 2) based on the specified entity name.

In operation 511, the electronic device 101 may determine whether the user's intention is identified using the obtained execution information. The electronic device 101 may perform operation 519 when the user's intention is identified using execution information. For example, the electronic device 101 may determine that the user's intention is identified if the domain and intention can be specified using execution information. For example, when domain information or intent information included in execution information includes one domain or one intent, the electronic device 101 may specify the domain or intent.

The electronic device 101 may perform operation 513 when the user's intention is not identified using execution information. For example, if the electronic device 101 cannot specify the domain and intention using execution information, it may determine that the user's intention is not understood. For example, if the domain information or intent information included in the execution information includes multiple domains or includes multiple intents of the domain even if it includes one domain, the electronic device 101 specifies the domain or intent. You may not be able to do it.

In operation 513, the electronic device 101 may determine whether it is possible to obtain an intent list associated with the utterance using execution information. For example, when the intent information included in the execution information is related to the entity name, the electronic device 101 may determine that an intent list related to the utterance can be obtained. For example, if the intent information included in the execution information is not related to the entity name, the electronic device 101 may determine that it is impossible to obtain the intent list associated with the utterance. The electronic device 101 may perform operation 515 if it is possible to obtain an intent list associated with the utterance. The electronic device 101 may perform operation 517 when it is impossible to obtain the intent list associated with the utterance.

In operation 515, the electronic device 101 may return (or provide) a list of intentions associated with the utterance to the user and receive feedback from the user. For example, the user's feedback may include a user input for selecting one of at least one intent included in the intent list associated with the utterance. The electronic device 101 may determine the intention selected by the user input as the user's intention.

In operation 517, the electronic device 101 may return (or provide) a list of commonly used intentions to the user and receive feedback from the user. For example, the user's feedback may include user input for selecting one of at least one intention included in a list of popularly used intentions. For example, the user's feedback may include speech input that includes a specific domain and/or specific intent. The electronic device 101 may determine the user's intention using the intention selected by the user input, or the domain or intention specified by the speech input.

In operation 519, the electronic device 101 may execute an operation corresponding to the user's intention. For example, the electronic device 101 may execute an operation corresponding to the user's intention based on the domain and/or intention determined according to the process of FIG. 5.

FIG. 6 is a diagram 600 showing a graphical user interface provided by an electronic device according to an example. The second electronic device 602 shown in FIG. 6 may correspond to the electronic device 101 in FIG. 1, and the operation of the second electronic device 602, which will be described later, may be performed by, for example, a processor (e.g., the processor of FIG. 1). (140)). The first electronic device 601 shown in FIG. 6 may be an example electronic device for comparison with the second electronic device 602.

For example, the first electronic device 601 and the second electronic device 602 may receive the same utterance “XXX” from the user. For example, the utterance received by the first electronic device 601 and the second electronic device 602 may be an utterance consisting of only an entity name without a predicate. For example, “XXX” may be a content name. Referring to FIG. 6, the first electronic device 601 and the second electronic device 602 may provide different information even if they receive the same utterance.

For example, in the case of an utterance without a predicate, the first electronic device 601 may not be able to determine the intention, domain, or category of the utterance. For example, the first electronic device 601 may not determine that “XXX” is included in the ‘Media Contents’ category. Accordingly, the first electronic device 601 has no choice but to provide the first graphical user interface 610 including a popular utterance hint list (or intent list) unrelated to the received utterance. For example, the first graphical user interface 610 may include information indicating an utterance (or intention) that can be executed by at least one domain. For example, at least one domain or intent indicated by information included in the first graphical user interface 610 may be unrelated to the entity name “XXX” or the ‘Media Contents’ category.

For example, even if the second electronic device 602 is an utterance without a predicate, the second electronic device 602 searches a database containing public data and personal data based on the entity name included in the utterance to obtain execution information, and uses the execution information to obtain execution information for the user. An operation corresponding to the intention can be performed. For example, the second electronic device 602 recognizes that the entity name "XXX" is included in the 'Media Contents' category, and that a 'search' operation using "XXX" as a search term can be performed by app 'G'. The execution information indicated can be obtained. The second electronic device 602 can search for “XXX” in app ‘G’ using this execution information.

The second electronic device 602 may provide a second graphic user interface 620 that displays the result of performing an operation corresponding to the user's intention using execution information. For example, the second graphical user interface 620 may display the results of searching “XXX” in app ‘G’.

FIG. 7 is a diagram 700 showing a graphical user interface provided by an electronic device according to an example. The second electronic device 702 shown in FIG. 7 may correspond to the electronic device 101 of FIG. 1, and the operation of the second electronic device 702, which will be described later, may be performed by, for example, a processor (e.g., the processor of FIG. 1). (140)). The first electronic device 701 shown in FIG. 7 may be an example electronic device for comparison with the second electronic device 702.

For example, the first electronic device 701 and the second electronic device 702 may receive the same utterance “XXX” from the user. For example, the utterance received by the first electronic device 701 and the second electronic device 702 may be an utterance consisting of only an entity name without a predicate. For example, “XXX” may be a content name. Referring to FIG. 7, the first electronic device 701 and the second electronic device 702 may provide different information even if they receive the same utterance.

For example, when there are multiple categories corresponding to an entity name, the first electronic device 701 selects the entity name according to a predetermined priority (or ordering) regardless of the preference of the individual (user of the electronic device). The category corresponding to can be determined, and the domain or intent can be determined according to the determined category. For example, if a 'TV shows' category and a 'Music' category exist for the entity name "XXX", the first electronic device 701 responds to the entity name regardless of the preference of the first electronic device 701. You can decide the category to be 'Music'. For example, as the first electronic device 701 determines the 'Music' category, the domain is set to App 'A', a music streaming app, and the intention is to 'play' a song with "XXX" as the title. It can be decided that For example, the first electronic device 701 may display a first electronic device 701 that represents the result of performing an operation based on the domain and intention for the entity name “XXX”, which is determined regardless of the preference of the user of the first electronic device 701. A graphical user interface 710 may be provided. For example, the user's intention may be to play a TV program with “XXX” as the title. In this case, the first graphical user interface 710 may include results that do not correspond to the user's intention.

For example, when there are multiple categories corresponding to an entity name, the second electronic device 702 determines the category corresponding to the entity name by referring to the preferences of the individual (user of the electronic device), and determines the domain and domain according to the determined category. You can determine your intention. For example, the second electronic device 702 may store a history of playing a TV program with the entity name “XXX” as the title for more than a threshold time. This history information may indicate that the user of the second electronic device 702 prefers a TV program with the entity name “XXX” as the title. For example, specific category (eg, TV shows) preference information for the entity name “XXX” may be collected in the second electronic device 702 as personal data of the user of the second electronic device 702. For example, the second electronic device 702 responds to the user's intention based on the collected personal data among the 'TV shows' category and 'Music' category searched for the entity name "XXX" based on public data. The category may be determined to be the 'TV shows' category. For example, as the second electronic device 702 determines the 'TV shows' category, the domain is set to App 'G', an app capable of playing TV programs, and the intention is set to 'Play' a TV program with "XXX" as the title. You can decide to ‘play’. For example, the second electronic device 702 may display a second electronic device 702 that represents the result of performing an operation based on the domain and intention for the entity name “XXX”, which is determined with reference to the user's preference. A graphical user interface 720 may be provided. The second graphical user interface 720 may include results corresponding to the user's intent.

FIG. 8 is a diagram 800 showing a graphical user interface provided by an electronic device according to an example. The second electronic device 802 shown in FIG. 8 may correspond to the electronic device 101 in FIG. 1, and the operation of the second electronic device 802, which will be described later, may be performed by, for example, a processor (e.g., the processor of FIG. 1). (140)). The first electronic device 801 shown in FIG. 8 may be an example electronic device for comparison with the second electronic device 802.

For example, the first electronic device 801 and the second electronic device 802 may receive the same utterance, “Gangnam Station” from the user. For example, the utterance received by the first electronic device 801 and the second electronic device 802 may be an utterance consisting of only an entity name without a predicate. Referring to FIG. 8, the first electronic device 801 and the second electronic device 802 may provide different information even if they receive the same utterance.

For example, in the case of an utterance without a predicate, the first electronic device 801 may not be able to determine the category, domain, or intent of the utterance. For example, the first electronic device 801 may not determine that “Gangnam Station” is included in the ‘Location’ category. Accordingly, the first electronic device 601 has no choice but to provide the first graphical user interface 810 including a popular utterance hint list (or intent list) unrelated to the received utterance. For example, the first graphical user interface 810 may include information indicating an utterance (or intention) that can be executed by at least one domain. For example, at least one domain or intent indicated by the information included in the first graphical user interface 810 may be unrelated to the entity name “Gangnam Station” or the ‘Location’ category.

For example, even if the second electronic device 802 is an utterance without a predicate, the second electronic device 802 searches a database containing public data and personal data based on the entity name included in the utterance to obtain execution information, and uses the execution information to obtain execution information for the user. An operation corresponding to the intention can be performed. For example, the second electronic device 802 includes the entity name "Gangnam Station" in the 'Location' category, app 'B', app 'C', app 'D', and app 'E' based on public data. ', execution information indicating that each specified operation can be performed can be obtained. The second electronic device 802 may use this execution information to provide a second graphical user interface 820 including an intent list related to “Gangnam Station.” The intent list may include, for example, intent information executable by each of app 'B', app 'C', app 'D', and app 'E'.

The second electronic device 802 may receive a user input 825 for selecting one of a plurality of intentions included in the intention list. The second electronic device 802 may provide a third graphical user interface 830 in response to receiving the user input 825. For example, the intention selected by the user input 825 may be to perform a route guidance operation to “Gangnam Station” by app ‘B’. The second electronic device 802 may perform a route guidance operation from app 'B' to "Gangnam Station" in response to receiving the user input 825. For example, the third graphical user interface 830 may display a screen that provides route guidance from app 'B' to "Gangnam Station."

For example, in response to receiving the user input 825, the second electronic device 802 records a history of performing a navigation operation with “Gangnam Station” as the destination in app ‘B’ for the entity name “Gangnam Station”. You can save it. For example, domain information on which an action was performed for the entity name “Gangnam Station” or intent information corresponding to the performed action may be collected in the second electronic device 802 as personal data of the user of the second electronic device 802. You can. For example, the second electronic device 802 may select the collected apps among apps 'B', app 'C', app 'D', and app 'E' searched for the entity name "Gangnam Station" based on public data. Based on personal data, it can be determined that the domain corresponding to the user's intention is app 'B', and that the user's intention is to perform a navigation operation using app 'B'. For example, when the second electronic device 802 receives an utterance containing the entity name “Gangnam Station” and no predicate after collecting the personal data, the second electronic device 802 does not provide the second graphic user interface 820 and allows the user to A third graphical user interface 830 may be provided that displays a screen that performs an operation corresponding to the intention.

FIG. 9 is a diagram 900 showing a graphical user interface provided by an electronic device according to an example. The electronic device 901 shown in FIG. 9 may correspond to the electronic device 101 in FIG. 1, and the operation of the electronic device 901 described later may be performed by, for example, a processor (e.g., processor 140 in FIG. 1). It can be performed by .

Referring to FIG. 9 , the electronic device 901 may provide a graphical user interface 910 that specifies a domain in which to perform an operation for at least one entity name. For example, the graphical user interface 910 is a first UI (user interface) that includes a list of entity names included in a specific category and a button for specifying or changing the domain for at least one entity name included in the list. It may contain element 911. For example, the first UI element 911 may include a list of entity names included in the 'Media Contents' category. For example, entity names may be classified by type of content and included in the list.

For example, the graphical user interface 910 may include a second UI element 912 that includes a button for adding an entity name for which a domain is to be specified. In the example shown in FIG. 9, the second UI element 912 may be a button for adding an entity name included in the 'Media Contents' category.

For example, the graphical user interface 910 may be a user interface that specifies only a domain while the intent is specified. For example, the graphical user interface 910 may be a user interface that selects a domain corresponding to each entity name among domains that can execute the same or similar intent. In the example shown in FIG. 9 , the graphical user interface 910 may be a user interface that specifies (or selects) a domain for executing the intention of ‘playing content’ for each entity name related to content.

FIG. 10 is a flowchart 1000 showing the operation of an electronic device according to an example. In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel. The operation of an electronic device described later (eg, the electronic device 101 of FIG. 1) may be performed by, for example, a processor (eg, the processor 140 of FIG. 1).

In operation 1001, the electronic device 101 may receive a voice command. For example, the electronic device 101 may receive a voice command through a microphone (eg, microphone 110 in FIG. 1). For example, voice commands may be referred to as user utterances. For example, the electronic device 101 may convert a voice command received through the microphone 110 into text based on automatic voice recognition.

In operation 1003, the electronic device 101 may determine whether the text corresponding to the voice command consists of one entity name. For example, if the text corresponding to the voice command does not include a predicate but includes one entity name, the electronic device 101 may determine that the text is composed of one entity name.

In operation 1005, when the text consists of one entity name, the electronic device 101 may obtain execution information for executing a voice command based on the entity name. For example, the execution information may include domain information indicating the application executing the voice command and intent information indicating the function of the application requested by the voice command. For example, the electronic device 101 may search a database (eg, DB 235 in FIG. 2) based on the entity name to obtain execution information. For example, the electronic device 101 may search the database based on the string and pronunciation string of the entity name. For example, the database may include public data (e.g., public data 235-2 in FIG. 2) and personal data (e.g., personal data 235-1 in FIG. 2).

For example, public data may include entity names used by the public. For example, public data can be repeatedly updated with the latest data at specified times through an external server. For example, personal data may include data specific to an individual (e.g., a user of electronic device 101). For example, the electronic device 101 may collect registered entity names as personal data based on user input received through an input device (e.g., including the microphone 110 in FIG. 1) of the electronic device 101. You can. Public data and personal data may include category information, domain information, and/or intent information associated with the entity name.

For example, when a plurality of categories corresponding to an entity name are searched based on public data, the electronic device 101 may obtain usage ratio information for each category. For example, the usage rate information for each category may indicate the rate at which an entity name is used in each category among a plurality of categories. For example, the usage rate information for each category may indicate the probability that an entity name will be used in each category among a plurality of categories. The electronic device 101 may determine a category corresponding to the entity name based on ratio information. The electronic device 101 may obtain execution information corresponding to the determined category.

For example, when a plurality of categories corresponding to an entity name are searched based on public data, the electronic device 101 may provide a first user interface that asks the user which category of the plurality of categories he or she wants information belonging to. there is. For example, the electronic device 101 may provide a first user interface by displaying a graphical user interface on the display of the electronic device 101. For example, when a plurality of categories corresponding to an entity name are searched and the electronic device 101 cannot determine one category corresponding to the entity name based on usage ratio information for each category, the electronic device 101 displays the first user interface. You can check the category the user intended. The electronic device 101 may obtain execution information corresponding to the category selected by user input on the first user interface.

For example, the electronic device 101 searches for a plurality of categories corresponding to an entity name based on public data, and if it is possible to specify one category among the plurality of categories based on personal data, the electronic device 101 searches for a plurality of categories corresponding to the entity name, and selects one category among the plurality of categories based on personal data. Corresponding execution information can be obtained. For example, if there is one category corresponding to the entity name as a result of a search based on personal data, the electronic device 101 may specify the category as the category corresponding to the entity name. For example, if there are a plurality of categories corresponding to the entity name as a result of a search based on personal data, the electronic device 101 matches the most frequently used category or the category with the highest preference among the plurality of categories to the entity name. It can be specified by category.

For example, when there is no data to be searched from the database based on an entity name, the electronic device 101 may provide a second user interface including a list of popularly used intents. For example, the electronic device 101 may provide a second user interface by displaying a graphical user interface on the display of the electronic device 101.

For example, if the text corresponding to the voice command does not consist of a single entity name, the electronic device 101 may determine the intent through natural language processing of the text. The electronic device 101 may determine the domain and intent by analyzing the text (e.g., grammatical analysis and/or semantic analysis) based on natural language understanding. The electronic device 101 may perform a function corresponding to the determined intention.

For example, the electronic device 101 can perform operation 1005 only when the text consists of one entity name and there are two or more domains corresponding to the entity name. For example, when the text consists of one entity name and there is one domain corresponding to the entity name, the electronic device 101 determines the intent in the domain through natural language processing of the text and responds to the determined intent. It can perform the function:

In operation 1007, the electronic device 101 may execute a voice command using execution information. For example, the electronic device 101 may execute a voice command using the domain indicated by domain information included in execution information. For example, the electronic device 101 may execute a voice command by executing the function of the domain indicated by the intent information included in the execution information. For example, the electronic device 101 may perform an action corresponding to a voice command based on the domain and/or intent indicated by domain information or intent information included in execution information. For example, the electronic device 101 may perform an operation corresponding to a voice command based on a specified domain or intention using execution information. For example, the electronic device 101 may use execution information to perform an operation corresponding to the intention selected by user input on the provided user interface.

For example, if the action performed by executing the voice command corresponds to the intention of the user of the electronic device, the electronic device 101 may store the information used in executing the voice command as personal data. The information used to execute the voice command may include at least one of the entity name, the category corresponding to the entity name, the number of times the entity name was uttered, the domain in which the action was performed, or the intention corresponding to the action performed in the domain. there is. For example, the electronic device 101 may store information used to execute a voice command by associating the entity name with other information. Thereafter, when the electronic device 101 receives a voice command including the same entity name, it may obtain information (e.g., category, number of utterances, domain, and/or intent) associated with the entity name stored in the database. For example, the electronic device 101 may perform an operation corresponding to a voice command based on the acquired domain information and/or intent information.

For example, the electronic device 101 may generate an entity extracted from the entity name usage history in each application, based on the entity name usage history in each of at least one application stored (or installed) in the electronic device 101. The name and the domain corresponding to each application or the intent in which the extracted entity name was used can be linked and stored in the database. For example, if the entity name included in the voice command corresponds to an entity name that has previously been used, the electronic device 101 may perform an operation according to the domain or intention determined based on the usage history.

For example, the electronic device 101 may receive a user input that specifies the domain or intention in the domain to perform an operation on at least one entity name through the input device of the electronic device 101. Based on the received user input, the electronic device 101 may be set to perform an operation according to a designated domain or intent when a voice command including at least one entity name is received. For example, the electronic device 101 may associate an entity name with a correspondingly designated domain or intention and store it in a database. For example, if an entity name included in a voice command corresponds to an entity name whose domain or intent is specified by a user input, the electronic device 101 may perform an operation according to the specified domain or intent.

Hereinafter, the electronic device 1101 of FIG. 11 may correspond to the electronic device 101 of FIG. 1. The microphone 110 in FIG. 1 may correspond to the input module 1150 in FIG. 11 . The communication circuit 120 of FIG. 1 may correspond to the communication module 1190 of FIG. 11 . The memory 130 of FIG. 1 may correspond to the memory 1130 of FIG. 11 . The processor 940 of FIG. 1 may correspond to the processor 1120 of FIG. 11 . For example, the electronic device 101 of FIG. 1 may further include components (eg, a display module 1160) of the electronic device 1101 of FIG. 11 .

FIG. 11 is a block diagram of an electronic device 1101 in a network environment 1100, according to various examples. Referring to FIG. 11, in the network environment 1100, the electronic device 1101 communicates with the electronic device 1102 through a first network 1198 (e.g., a short-range wireless communication network) or a second network 1199. It is possible to communicate with at least one of the electronic device 1104 or the server 1108 through (e.g., a long-distance wireless communication network). According to one example, the electronic device 1101 may communicate with the electronic device 1104 through the server 1108. According to one example, the electronic device 1101 includes a processor 1120, a memory 1130, an input module 1150, an audio output module 1155, a display module 1160, an audio module 1170, and a sensor module 1176. ), interface 1177, connection terminal 1178, haptic module 1179, camera module 1180, power management module 1188, battery 1189, communication module 1190, subscriber identification module 1196, Alternatively, it may include an antenna module 1197. In some examples, at least one of these components (eg, the connection terminal 1178) may be omitted, or one or more other components may be added to the electronic device 1101. In some examples, some of these components (e.g., sensor module 1176, camera module 1180, or antenna module 1197) may be integrated into one component (e.g., display module 1160). You can.

The processor 1120, for example, executes software (e.g., program 1140) to operate at least one other component (e.g., hardware or software component) of the electronic device 1101 connected to the processor 1120. It can be controlled and various data processing or calculations can be performed. According to one example, as at least part of data processing or computation, processor 1120 stores commands or data received from another component (e.g., sensor module 1176 or communication module 1190) in volatile memory 1132. The commands or data stored in the volatile memory 1132 may be stored and processed, and the resulting data may be stored in the non-volatile memory 1134. According to one example, the processor 1120 is a main processor 1121 (e.g., a central processing unit or an application processor) or an auxiliary processor 1123 that can operate independently or together (e.g., a graphics processing unit, a neural network processing unit (NPU) : neural processing unit), image signal processor, sensor hub processor, or communication processor). For example, if the electronic device 1101 includes a main processor 1121 and a auxiliary processor 1123, the auxiliary processor 1123 may be set to use lower power than the main processor 1121 or be specialized for a designated function. You can. The auxiliary processor 1123 may be implemented separately from the main processor 1121 or as part of it.

The auxiliary processor 1123 may, for example, act on behalf of the main processor 1121 while the main processor 1121 is in an inactive (e.g., sleep) state, or while the main processor 1121 is in an active (e.g., application execution) state. ), together with the main processor 1121, at least one of the components of the electronic device 1101 (e.g., the display module 1160, the sensor module 1176, or the communication module 1190) At least some of the functions or states related to can be controlled. According to one example, coprocessor 1123 (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module 1180 or communication module 1190). . According to one example, the auxiliary processor 1123 (e.g., neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. Artificial intelligence models can be created through machine learning. For example, such learning may be performed in the electronic device 1101 itself on which the artificial intelligence model is performed, or may be performed through a separate server (e.g., server 1108). Learning algorithms may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but It is not limited. An artificial intelligence model may include multiple artificial neural network layers. Artificial neural networks include deep neural network (DNN), convolutional neural network (CNN), recurrent neural network (RNN), restricted boltzmann machine (RBM), belief deep network (DBN), bidirectional recurrent deep neural network (BRDNN), It may be one of deep Q-networks or a combination of two or more of the above, but is not limited to the examples described above. In addition to hardware structures, artificial intelligence models may additionally or alternatively include software structures.

The memory 1130 may store various data used by at least one component (eg, the processor 1120 or the sensor module 1176) of the electronic device 1101. Data may include, for example, input data or output data for software (e.g., program 1140) and instructions related thereto. Memory 1130 may include volatile memory 1132 or non-volatile memory 1134.

The program 1140 may be stored as software in the memory 1130 and may include, for example, an operating system 1142, middleware 1144, or application 1146.

The input module 1150 may receive commands or data to be used in a component of the electronic device 1101 (e.g., the processor 1120) from outside the electronic device 1101 (e.g., a user). The input module 1150 may include, for example, a microphone, mouse, keyboard, keys (eg, buttons), or digital pen (eg, stylus pen).

The sound output module 1155 may output sound signals to the outside of the electronic device 1101. The sound output module 1155 may include, for example, a speaker or receiver. Speakers can be used for general purposes such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. According to one example, the receiver may be implemented separately from the speaker or as part of it.

The display module 1160 can visually provide information to the outside of the electronic device 1101 (eg, a user). The display module 1160 may include, for example, a display, a hologram device, or a projector, and a control circuit for controlling the device. According to one example, the display module 1160 may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of force generated by the touch.

The audio module 1170 can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one example, the audio module 1170 acquires sound through the input module 1150, the sound output module 1155, or an external electronic device (e.g., electronic device) directly or wirelessly connected to the electronic device 1101. Sound may be output through a device 1102 (e.g., speaker or headphone).

The sensor module 1176 detects the operating state (e.g., power or temperature) of the electronic device 1101 or the external environmental state (e.g., user state) and generates an electrical signal or data value corresponding to the detected state. can do. According to one example, the sensor module 1176 includes, for example, a gesture sensor, a gyro sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, and a temperature sensor. It may include a sensor, humidity sensor, or light sensor.

The interface 1177 may support one or more designated protocols that can be used to directly or wirelessly connect the electronic device 1101 to an external electronic device (eg, the electronic device 1102). According to one example, the interface 1177 may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

The connection terminal 1178 may include a connector through which the electronic device 1101 can be physically connected to an external electronic device (eg, the electronic device 1102). According to one example, the connection terminal 1178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (eg, a headphone connector).

The haptic module 1179 can convert electrical signals into mechanical stimulation (e.g., vibration or movement) or electrical stimulation that the user can perceive through tactile or kinesthetic senses. According to one example, the haptic module 1179 may include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

The camera module 1180 can capture still images and moving images. According to one example, the camera module 1180 may include one or more lenses, image sensors, image signal processors, or flashes.

The power management module 1188 can manage power supplied to the electronic device 1101. According to one example, the power management module 1188 may be implemented as at least a part of, for example, a power management integrated circuit (PMIC).

The battery 1189 may supply power to at least one component of the electronic device 1101. According to one example, the battery 1189 may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

Communication module 1190 provides a direct (e.g., wired) communication channel or wireless communication channel between the electronic device 1101 and an external electronic device (e.g., electronic device 1102, electronic device 1104, or server 1108). It can support establishment and communication through established communication channels. Communication module 1190 operates independently of processor 1120 (e.g., an application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one example, the communication module 1190 may be a wireless communication module 1192 (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 1194 (e.g., It may include a local area network (LAN) communication module, or a power line communication module. Among these communication modules, the corresponding communication module is a first network 1198 (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network 1199 (e.g., legacy It may communicate with an external electronic device 1104 through a telecommunication network such as a cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., LAN or WAN). These various types of communication modules may be integrated into one component (e.g., a single chip) or may be implemented as a plurality of separate components (e.g., multiple chips). The wireless communication module 1192 uses subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module 1196 within a communication network such as the first network 1198 or the second network 1199. The electronic device 1101 can be confirmed or authenticated.

The wireless communication module 1192 may support 5G networks and next-generation communication technologies after 4G networks, for example, NR access technology (new radio access technology). NR access technology provides high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and access to multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low latency). -latency communications)) can be supported. The wireless communication module 1192 may support high frequency bands (e.g., mmWave bands), for example, to achieve high data rates. The wireless communication module 1192 uses various technologies to secure performance in high frequency bands, for example, beamforming, massive array multiple-input and multiple-output (MIMO), and full-dimensional multiplexing. It can support technologies such as input/output (FD-MIMO: full dimensional MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module 1192 may support various requirements specified in the electronic device 1101, an external electronic device (e.g., electronic device 1104), or a network system (e.g., second network 1199). According to one example, the wireless communication module 1192 supports peak data rate (e.g., 20 Gbps or more) for realizing eMBB, loss coverage (e.g., 164 dB or less) for realizing mmTC, or U-plane latency (e.g., 164 dB or less) for realizing URLLC. : Downlink (DL) and uplink (UL) each of 0.5 ms or less, or round trip 1 ms or less) can be supported.

The antenna module 1197 may transmit or receive signals or power to or from the outside (e.g., an external electronic device). According to one example, the antenna module 1197 may include an antenna including a radiator made of a conductor or a conductive pattern formed on a substrate (eg, PCB). According to one example, the antenna module 1197 may include a plurality of antennas (eg, an array antenna). In this case, at least one antenna suitable for the communication method used in the communication network, such as the first network 1198 or the second network 1199, is connected to the plurality of antennas by, for example, the communication module 1190. can be selected Signals or power may be transmitted or received between the communication module 1190 and an external electronic device through the selected at least one antenna. According to some examples, other components (eg, radio frequency integrated circuit (RFIC)) in addition to the radiator may be additionally formed as part of the antenna module 1197.

According to various examples, antenna module 1197 may form a mmWave antenna module. According to one example, a mmWave antenna module includes a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., bottom side) of the printed circuit board and capable of supporting a designated high frequency band (e.g., mmWave band), and It may include a plurality of antennas (e.g., array antennas) disposed on or adjacent to the second side (e.g., top or side) of the printed circuit board and capable of transmitting or receiving signals in the designated high frequency band. You can.

At least some of the components are connected to each other through a communication method between peripheral devices (e.g., bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)) and signal ( (e.g. commands or data) can be exchanged with each other.

According to one example, commands or data may be transmitted or received between the electronic device 1101 and the external electronic device 1104 through the server 1108 connected to the second network 1199. Each of the external electronic devices 1102 or 1104 may be of the same or different type as the electronic device 1101. According to one example, all or part of the operations performed in the electronic device 1101 may be executed in one or more of the external electronic devices 1102, 1104, or 1108. For example, when the electronic device 1101 needs to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device 1101 does not execute the function or service on its own. Alternatively, or additionally, one or more external electronic devices may be requested to perform at least part of the function or service. One or more external electronic devices that have received the request may execute at least part of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device 1101. The electronic device 1101 may process the result as is or additionally and provide it as at least part of a response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology can be used. The electronic device 1101 may provide an ultra-low latency service using, for example, distributed computing or mobile edge computing. In another example, the external electronic device 1104 may include an Internet of Things (IoT) device. Server 1108 may be an intelligent server using machine learning and/or neural networks. According to one example, an external electronic device 1104 or a server 1108 may be included in the second network 1199. The electronic device 1101 may be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

Hereinafter, the user terminal 1201 of FIG. 12 may correspond to the electronic device 101 of FIG. 1 and/or the electronic device 1101 of FIG. 11. The microphone 110 in FIG. 1 may correspond to the microphone 1270 in FIG. 12 . The communication circuit 120 of FIG. 1 may correspond to the communication interface 1290 of FIG. 12 . Memory 130 of FIG. 1 may correspond to memory 1230 of FIG. 12 . The processor 140 of FIG. 1 may correspond to the processor 1220 of FIG. 12 . For example, the electronic device 101 of FIG. 1 may further include components (e.g., a display 1260) of the user terminal 1201 of FIG. 12. The automatic voice recognition module 1321 of FIG. 12 is shown in FIG. It may correspond to the ASR module 210 of Figure 2. The natural language understanding module 1323 of Figure 12 may correspond to the NLU module 220 of Figure 2. The electronic device 101 of Figure 1 may correspond to the intelligent All or at least part of the modules of the server 1300 (e.g., the automatic voice recognition module 1321 and/or the natural language understanding module 1323) may be implemented in the electronic device 101. The electronic device 101 in FIG. ) may perform all or part of the functions of the intelligent server 1300 of FIG. 12 (e.g., server 1108 of FIG. 11) by the electronic device 101 itself.

Figure 12 is a block diagram showing an integrated intelligence system according to an example.

Referring to FIG. 12, an example integrated intelligence system may include a user terminal 1201, an intelligent server 1300, and a service server 1400 (eg, server 1108 in FIG. 11).

The user terminal 1201 (e.g., the electronic device 1101 in FIG. 11) may be a terminal device (or electronic device) capable of connecting to the Internet, for example, a mobile phone, a smartphone, or a personal digital assistant (PDA). assistant), a laptop computer, a television (TV), a white appliance, a wearable device, a head mounted device (HMD), or a smart speaker.

According to the example shown, the user terminal 1201 may include a communication interface 1290, a microphone 1270, a speaker 1255, a display 1260, a memory 1230, and/or a processor 1220. . The components listed above may be operatively or electrically connected to each other.

An example communication interface 1290 (e.g., communication module 1190 in FIG. 11) may be connected to an external device and configured to transmit and receive data. The microphone 1270 (e.g., the audio module 1170 in FIG. 11) may receive sound (e.g., a user's speech) and convert it into an electrical signal. The speaker 1255 (e.g., the sound output module 1155 of FIG. 11) may output an electrical signal as sound (e.g., voice).

One example display 1260 (e.g., display module 1160 of FIG. 11) may be configured to display images or video. The example display 1260 may also display a graphic user interface (GUI) of an app (or application program) being executed.

The display 1260 in one embodiment may receive a touch input through a touch sensor. For example, the display 1260 may receive text input through a touch sensor in the on-screen keyboard area displayed within the display 1260.

An example memory 1230 (e.g., memory 1130 of FIG. 11) may store a client module 1231, a software development kit (SDK) 1233, and a plurality of applications. The client module 1231 and SDK 1233 may form a framework (or solution program) for performing general functions. Additionally, the client module 1231 or SDK 1233 may configure a framework for processing user input (eg, voice input, text input, or touch input).

The plurality of applications (eg, 1235a, 1235b) may be programs for performing designated functions. According to one example, the plurality of applications may include a first app 1235a and/or a second app 1235b. According to one example, each of a plurality of applications may include a plurality of operations to perform a designated function. For example, the applications may include an alarm app, a messaging app, and/or a schedule app. According to one example, a plurality of applications may be executed by the processor 1220 to sequentially execute at least some of the plurality of operations.

In one example, the processor 1220 may control the overall operation of the user terminal 1201. For example, the processor 1220 may be electrically connected to the communication interface 1290, the microphone 1270, the speaker 1255, and the display 1260 to perform a designated operation. For example, processor 1220 may include at least one processor.

The example processor 1220 may also execute a program stored in the memory 1230 to perform a designated function. For example, the processor 1220 may execute at least one of the client module 1231 or the SDK 1233 and perform the following operations to process user input. The processor 1220 may control the operation of multiple applications through, for example, the SDK 1233. The following operations described as operations of the client module 1231 or SDK 1233 may be operations performed by execution of the processor 1220.

One example client module 1231 can receive user input. For example, the client module 1231 may receive a voice signal corresponding to a user utterance detected through the microphone 1270. Alternatively, the client module 1231 may receive a touch input detected through the display 1260. Alternatively, the client module 1231 may receive text input detected through a keyboard or visual keyboard. In addition, various types of user input detected through an input module included in the user terminal 1201 or an input module connected to the user terminal 1201 can be received. The client module 1231 may transmit the received user input (eg, voice signal) to the intelligent server 1300. The client module 1231 may transmit status information of the user terminal 1201 to the intelligent server 1300 along with the received user input. The status information may be, for example, execution status information of an app.

As an example, the client module 1231 may receive a result corresponding to the received user input from the intelligent server 1300. For example, if the intelligent server 1300 can calculate a result corresponding to the received user input, the client module 1231 may receive a result corresponding to the received user input. The client module 1231 may display the received result on the display 1260. Additionally, the client module 1231 may output the received result as audio through the speaker 1255.

The example client module 1231 may receive a plan corresponding to the received user input. The client module 1231 can display the results of executing a plurality of operations of the app according to the plan on the display 1260. For example, the client module 1231 may sequentially display execution results of a plurality of operations on a display and output audio through the speaker 1255. For another example, the user terminal 1201 may display only a partial result of executing a plurality of operations (e.g., the result of the last operation) on the display and output audio through the speaker 1255.

According to one example, the client module 1231 may receive a request from the intelligent server 1300 to obtain information necessary to calculate a result corresponding to the user input. According to one example, the client module 1231 may transmit the necessary information to the intelligent server 1300 in response to the request.

As an example, the client module 1231 may transmit information as a result of executing a plurality of operations according to the plan to the intelligent server 1300. The intelligent server 1300 can use the result information to confirm that the received user input has been processed correctly.

One example client module 1231 may include a voice recognition module. According to one example, the client module 1231 may recognize voice input performing a limited function through the voice recognition module. For example, the client module 1231 may execute an intelligent app for processing voice input by performing an organic action in response to a designated voice input (e.g., wake up!).

The example intelligent server 1300 receives information related to the user's voice input from the user terminal 1201 through the network 1299 (e.g., the first network 1198 and/or the second network 1199 in FIG. 11). can do. According to one example, the intelligent server 1300 may change data related to the received voice input into text data. According to one example, the intelligent server 1300 may generate at least one plan for performing a task corresponding to a user's voice input based on the text data.

According to one example, the plan may be generated by an artificial intelligence (AI) system. An artificial intelligence system may be a rule-based system, a neural network-based system (e.g., a feedforward neural network (FNN)), and/or a recurrent neural network. network(RNN))). Alternatively, it may be a combination of the above or a different artificial intelligence system. According to one example, a plan may be selected from a set of predefined plans or may be generated in real time in response to a user request. For example, an artificial intelligence system can select at least one plan from a plurality of predefined plans.

As an example, the intelligent server 1300 may transmit a result according to the generated plan to the user terminal 1201 or transmit the generated plan to the user terminal 1201. According to one example, the user terminal 1201 may display results according to the plan on the display 1260. According to one example, the user terminal 1201 may display the results of executing an operation according to the plan on the display 1260.

An example intelligent server 1300 includes a front end 1310, a natural language platform 1320, a capsule database 1330, an execution engine 1340, an end It may include an end user interface (1350), a management platform (1360), a big data platform (1370), or an analytic platform (1380).

In one example, the front end 1310 may receive a voice input received by the user terminal 1201 from the user terminal 1201 . The front end 1310 may transmit a response corresponding to the voice input to the user terminal 1201.

According to one example, the natural language platform 1320 includes an automatic speech recognition module (ASR module) 1321, a natural language understanding module (NLU module) 1323, and a planner module. module) (1325), a natural language generator module (NLG module) (1327), and/or a text to speech module (TTS module) (1329).

As an example, the automatic voice recognition module 1321 may convert voice input received from the user terminal 1201 into text data. As an example, the natural language understanding module 1323 may determine the user's intention using text data of voice input. For example, the natural language understanding module 1323 may determine the user's intention by performing syntactic analysis and/or semantic analysis on user input in the form of text data. As an example, the natural language understanding module 1323 uses linguistic features (e.g., grammatical elements) of morphemes or phrases to determine the meaning of a word extracted from the user input, and matches the meaning of the identified word to the intention to determine the user's intention. can be decided. The natural language understanding module 1323 can acquire intent information corresponding to the user's utterance. Intention information may be information indicating the user's intention determined by interpreting text data. Intent information may include information indicating an action or function that the user wishes to perform using the device.

As an example, the planner module 1325 may generate a plan using the intent and parameters determined by the natural language understanding module 1323. According to one example, the planner module 1325 may determine a plurality of domains necessary to perform a task based on the determined intention. The planner module 1325 may determine a plurality of operations included in each of the plurality of domains determined based on the intention. According to one example, the planner module 1325 may determine parameters required to execute the determined plurality of operations or result values output by executing the plurality of operations. The parameters and the result values may be defined as concepts of a specified type (or class). Accordingly, the plan may include a plurality of operations and/or a plurality of concepts determined by the user's intention. The planner module 1325 may determine the relationship between the plurality of operations and the plurality of concepts in a stepwise (or hierarchical) manner. For example, the planner module 1325 may determine the execution order of a plurality of operations determined based on the user's intention based on a plurality of concepts. In other words, the planner module 1325 may determine the execution order of the plurality of operations based on parameters required for execution of the plurality of operations and results output by executing the plurality of operations. Accordingly, the planner module 1325 may generate a plan that includes association information (eg, ontology) between a plurality of operations and a plurality of concepts. The planner module 1325 can create a plan using information stored in the capsule database 1330, which stores a set of relationships between concepts and operations.

As an example, the natural language generation module 1327 may change designated information into text form. The information changed to the text form may be in the form of natural language speech. As an example, the text-to-speech conversion module 1329 can change information in text form into information in voice form.

According to one example, some or all of the functions of the natural language platform 1320 may also be implemented in the user terminal 1201. For example, the user terminal 1201 may include an automatic speech recognition module and/or a natural language understanding module. After the user terminal 1201 recognizes the user's voice command, it can transmit text information corresponding to the recognized voice command to the intelligent server 1300. For example, the user terminal 1201 may include a text-to-speech module. The user terminal 1201 may receive text information from the intelligent server 1300 and output the received text information as voice.

The capsule database 1330 may store information about the relationship between a plurality of concepts and operations corresponding to a plurality of domains. A capsule according to an example may include a plurality of action objects (or action information) and/or concept objects (or concept information) included in the plan. According to one example, the capsule database 1330 may store a plurality of capsules in the form of CAN (concept action network). According to one example, a plurality of capsules may be stored in a function registry included in the capsule database 1330.

The capsule database 1330 may include a strategy registry in which strategy information necessary for determining a plan corresponding to user input is stored. The strategy information may include standard information for determining one plan when there are multiple plans corresponding to user input. According to one example, the capsule database 1330 may include a follow up registry in which information on follow-up actions is stored to suggest follow-up actions to the user in a specified situation. The follow-up action may include, for example, follow-up speech. According to one example, the capsule database 1330 may include a layout registry that stores layout information of information output through the user terminal 1201. According to one example, the capsule database 1330 may include a vocabulary registry where vocabulary information included in capsule information is stored. According to one example, the capsule database 1330 may include a dialogue registry in which information about a dialogue (or interaction) with a user is stored. The capsule database 1330 can update stored objects through a developer tool. The developer tool may include, for example, a function editor for updating operation objects or concept objects. The developer tool may include a vocabulary editor for updating the vocabulary. The developer tool may include a strategy editor that creates and registers a strategy for determining the plan. The developer tool may include a dialogue editor that creates a dialogue with the user. The developer tool may include a follow up editor that can edit follow-up utterances to activate follow-up goals and provide hints. The subsequent goal may be determined based on currently set goals, user preferences, or environmental conditions. In one example, the capsule database 1330 may also be implemented within the user terminal 1201.

The example execution engine 1340 may calculate a result using the generated plan. The end user interface 1350 may transmit the calculated result to the user terminal 1201. Accordingly, the user terminal 1201 can receive the result and provide the received result to the user. An example management platform 1360 may manage information used by the intelligent server 1300. One example big data platform 1370 may collect user data. An example analytics platform 1380 may manage quality of service (QoS) of the intelligent server 1300. For example, analytics platform 1380 may manage the components and processing speed (or efficiency) of intelligent server 1300.

In one example, the service server 1400 may provide a designated service (eg, food ordering or hotel reservation) to the user terminal 1201. According to one example, the service server 1400 may be a server operated by a third party. As an example, the service server 1400 may provide the intelligent server 1300 with information for creating a plan corresponding to the received voice input. The provided information may be stored in the capsule database 1330. Additionally, the service server 1400 may provide result information according to the plan to the intelligent server 1300. The service server 1400 may communicate with the intelligent server 1300 and/or the user terminal 1201 through the network 1299. The service server 1400 can communicate with the intelligent server 1300 through a separate connection. Although the service server 1400 is shown as one server in FIG. 12, the embodiments of this document are not limited thereto. At least one of the services 1401, 1402, and 1403 of the service server 1400 may be implemented as a separate server.

In the integrated intelligence system described above, the user terminal 1201 can provide various intelligent services to the user in response to user input. The user input may include, for example, input through a physical button, touch input, or voice input.

In one example, the user terminal 1201 may provide a voice recognition service through an internally stored intelligent app (or voice recognition app). In this case, for example, the user terminal 1201 recognizes a user utterance or voice input received through the microphone 1270 and provides a service corresponding to the recognized voice input to the user. can do.

In one example, the user terminal 1201 may perform a designated operation alone or together with the intelligent server 1300 and/or the service server 1400 based on the received voice input. For example, the user terminal 1201 may run an app corresponding to a received voice input and perform a designated operation through the executed app.

In one example, when the user terminal 1201 provides a service together with the intelligent server 1300 and/or the service server 1400, the user terminal 1201 uses the microphone 1270 to make a user speech. may be detected, and a signal (or voice data) corresponding to the detected user utterance may be generated. The user terminal 1201 may transmit the voice data to the intelligent server 1300 using the communication interface 1290.

The intelligent server 1300 according to one example is a response to a voice input received from the user terminal 1201, a plan for performing a task corresponding to the voice input, or a result of performing an operation according to the plan. can be created. For example, the plan may include a plurality of operations for performing a task corresponding to a user's voice input and/or a plurality of concepts related to the plurality of operations. The concept may define parameters input to the execution of the plurality of operations or result values output by the execution of the plurality of operations. The plan may include association information between a plurality of operations and/or a plurality of concepts.

As an example, the user terminal 1201 may receive the response using the communication interface 1290. The user terminal 1201 outputs a voice signal generated inside the user terminal 1201 to the outside using the speaker 1255, or outputs an image generated inside the user terminal 1201 to the outside using the display 1260. It can be output as .

Figure 13 is a diagram showing how relationship information between concepts and actions is stored in a database, according to an example.

The capsule database (e.g., capsule database 1330) of the intelligent server (e.g., intelligent server 1300 of FIG. 12) may store capsules in CAN (concept action network) format. The capsule database may store operations for processing tasks corresponding to the user's voice input, and parameters necessary for the operations in CAN (concept action network) format.

The capsule database may store a plurality of capsules (capsule A 1331, capsule B 1334) corresponding to each of a plurality of domains (eg, applications). According to one example, one capsule (eg, capsule A 1331) may correspond to one domain (eg, location (geo), application). Additionally, one capsule contains at least one capsule of a service provider (e.g., CP 1 (1332), CP 2 (1333), CP3 (1335), and/or CP4 (1336) to perform functions for the domain associated with the capsule. )) can correspond. According to one example, one capsule may include at least one operation 1330a and at least one concept 1330b for performing a designated function.

The natural language platform (e.g., natural language platform 1320 in FIG. 12) may generate a plan for performing a task corresponding to the received voice input using the capsule stored in the capsule database 1330. For example, the planner module of the natural language platform (e.g., planner module 1325 in FIG. 12) can create a plan using capsules stored in the capsule database. For example, create a plan 1337 using the operations 1331a, 1332a and concepts 1331b, 1332b of capsule A 1331 and the operations 1334a and concept 1334b of capsule B 1334. can do.

Figure 14 is a diagram illustrating a screen in which a user terminal processes voice input received through an intelligent app according to an example.

The user terminal 1201 may run an intelligent app to process user input through an intelligent server (e.g., the intelligent server 1300 in FIG. 12).

According to one example, in the first screen 1210, when the user terminal 1201 recognizes a designated voice input (e.g., wake up!) or receives an input through a hardware key (e.g., a dedicated hardware key), the user terminal 1201 inputs the voice input. You can run intelligent apps to process. For example, the user terminal 1201 may run an intelligent app while executing a schedule app. According to one example, the user terminal 1201 may display an object (e.g., an icon) 1411 corresponding to an intelligent app on a display (e.g., the display 1260 of FIG. 12). According to one example, the user terminal 1201 may receive voice input through a user's utterance. For example, the user terminal 1201 may receive a voice input saying “Tell me this week’s schedule!” According to one example, the user terminal 1201 may display a user interface (UI) 1213 (e.g., input window) of an intelligent app displaying text data of a received voice input on the display.

According to one example, in the second screen 1215, the user terminal 1201 may display a result corresponding to the received voice input on the display. For example, the user terminal 1201 may receive a plan corresponding to the received user input and display 'this week's schedule' on the display according to the plan.

Electronic devices may provide a voice assistant function that controls devices based on voice commands. In order for an electronic device to provide a voice assistant function, it must identify the domain and intention of the action to be performed through the user's speech. For example, as the range of speech processing by the voice assistant function increases, there may be limitations in deriving the results desired by the user simply based on the content of the speech. For example, in order to specify the domain and intent simply based on the content of the utterance, an utterance containing a predicate may be required, or an utterance containing a specific domain may be required. For example, when receiving an utterance that does not include a predicate or a specific domain, the electronic device may not be able to perform an operation corresponding to the user's intention.

For example, when an electronic device receives an utterance that does not include a specific domain but includes a predicate and an entity name, it can determine the user's intention based on a rule. For example, if the entity name is related to media content, the electronic device may estimate the domain by calling a named entity search server (NES server) to determine the category corresponding to the entity name. However, when the electronic device receives an utterance without a predicate, it may not call the entity name search server. In this case, since the electronic device cannot determine the user's intention, it can only provide a list of popular speech hints that are unrelated to the speech content.

For example, even if the electronic device receives an utterance containing a predicate and determines a category corresponding to the entity name through an entity name search server, it may still not be able to determine the user's intention if there are multiple categories. In this case, the electronic device may perform operations based on its own ordering of the voice assistant function. The result of performing the operation may not be what the user wants.

The embodiment of the present disclosure can identify the user's desired domain (application-level domain) and the user's intention even for utterances without predicates.

Embodiments of the present disclosure can identify and provide intention information related to the utterance to the user even for utterances in which the user's intention is unclear.

An embodiment of the present disclosure can determine the user's intention using personal data even if there are multiple categories corresponding to the utterance.

An electronic device (101;602;702;802;901;1101;1201) according to an embodiment of the present disclosure includes a microphone (110;1150;1270), a communication circuit (120;1190;1290), a memory (130); 1130; 1230), and processors 140; 1120; 1220. The memory can store instructions. The instructions, when executed by the processor, may cause the electronic device to receive voice commands through the microphone. The instructions, when executed by the processor, may cause the electronic device to determine whether text corresponding to the voice command consists of a named entity. When executed by the processor, the instructions may cause the electronic device to obtain execution information for executing the voice command based on the entity name. The execution information may include domain information indicating an application executing the voice command or intent information indicating a function of the application requested by the voice command. When executed by the processor, the instructions may cause the electronic device to execute the voice command using the execution information.

According to an embodiment of the present disclosure, the instructions, when executed by the processor, cause the electronic device to: If the text consists of one entity name and there are two or more domains corresponding to the entity name, The execution information can be obtained based on the entity name. The instructions, when executed by the processor, cause the electronic device to, if the text consists of one entity name and there is one domain corresponding to the entity name, determine the intent in the domain, and It can be made to perform a function corresponding to the determined intention.

According to an embodiment of the present disclosure, when the instructions are executed by the processor, the electronic device determines the intent through natural language processing of the text if the text does not consist of one entity name. And, the function corresponding to the determined intention can be performed.

According to an embodiment of the present disclosure, when the instructions are executed by the processor, the electronic device collects public data 235-2 and personal data based on the string and pronunciation sequence of the entity name. The database 235 including (personal data) 235-1 can be searched.

According to an embodiment of the present disclosure, when the instructions are executed by the processor, if the electronic device searches for a plurality of categories corresponding to the entity name based on the public data, the instructions determine the ratio for each category. Information can be obtained. When executed by the processor, the instructions may cause the electronic device to determine the category corresponding to the entity name based on the ratio information. When executed by the processor, the instructions may cause the electronic device to obtain the execution information corresponding to the determined category.

According to an embodiment of the present disclosure, when the instructions are executed by the processor, the electronic device selects one of the plurality of categories when a plurality of categories corresponding to the entity name are searched based on the public data. A first user interface may be provided to query the user whether he/she wants information belonging to the category. When executed by the processor, the instructions may cause the electronic device to obtain the execution information corresponding to the category selected by the user input on the first user interface.

According to an embodiment of the present disclosure, when the instructions are executed by the processor, the electronic device searches for a plurality of categories corresponding to the entity name based on the public data and searches for a plurality of categories corresponding to the entity name based on the personal data. Thus, when one category among a plurality of categories can be specified, the execution information corresponding to the specified one category can be obtained.

According to an embodiment of the present disclosure, the instructions, when executed by the processor, are intended to be popularly used when the electronic device does not have data to be retrieved from the database based on the entity name. A second user interface including a list may be provided. When executed by the processor, the instructions may cause the electronic device to perform an action corresponding to the intention selected by the user input to the second user interface.

According to an embodiment of the present disclosure, when executed by the processor, the instructions determine whether the electronic device can specify a domain or intent based on the text when the text consists of one entity name. You can decide. When the instructions are executed by the processor, if the electronic device cannot specify the domain or the intent, the instructions may be used to specify the domain and the intent using the obtained execution information. When executed by the processor, the instructions may cause the electronic device to perform an action corresponding to the voice command based on the specified domain or intent.

According to an embodiment of the present disclosure, the electronic device may further include input devices (1150; 1260; 1270). The instructions, when executed by the processor, cause the electronic device to, through the input device, receive a user input specifying a domain to perform an action on at least one entity name or an intention in the domain. You can receive it. The instructions are, when executed by the processor, the electronic device, based on the received user input, when a voice command including any one of the at least one entity name is received, the specified domain or intent. Accordingly, it can be set to perform an operation corresponding to the voice command.

According to an embodiment of the present disclosure, the instructions, when executed by the processor, cause the electronic device to: Information used to execute voice commands can be stored as the personal data. The information used to execute the voice command is at least one of the entity name, the category corresponding to the entity name, the number of times the entity name is uttered, the domain that performed the operation, or the intention corresponding to the operation in the domain. It can contain one.

According to one embodiment of the present disclosure, the memory may store at least one application. The instructions, when executed by the processor, cause the electronic device to correspond to each application with an entity name extracted from the entity name usage history in each application, based on the entity name usage history in each of the at least one application. The domain or the extracted entity name may be associated with the intent of use and stored in the database.

According to an embodiment of the present disclosure, a method of operating an electronic device (101;602;702;802;901;1101;1201) includes receiving a voice command through a microphone (110;1150;1270) of the electronic device. Can include actions. The method may include determining whether text corresponding to the voice command consists of one named entity. The method may include obtaining execution information for executing a voice command based on the entity name. The execution information may include domain information indicating an application executing the voice command or intent information indicating a function of the application requested by the voice command. The method may include executing the voice command using the execution information.

According to an embodiment of the present disclosure, the operation of acquiring the execution information can be performed when the text consists of one entity name and there are two or more domains corresponding to the entity name. The method may include an operation of determining intent in the domain when the text consists of one entity name and there is only one domain corresponding to the entity name. The method may include an operation of performing a function corresponding to the determined intention.

According to an embodiment of the present disclosure, the method may include an operation of determining intent through natural language processing of the text when the text does not consist of a single entity name. The method may include an operation of performing a function corresponding to the determined intention.

According to an embodiment of the present disclosure, the operation of acquiring the execution information includes collecting public data 235-2 and personal data 235-1 based on the string and pronunciation string of the entity name. ) may include an operation of searching the database 235 including.

According to an embodiment of the present disclosure, the operation of acquiring the execution information may include obtaining ratio information for each category when a plurality of categories corresponding to the entity name are searched based on the public data. You can. Obtaining the execution information may include determining the category corresponding to the entity name based on the ratio information. The operation of acquiring the execution information may include obtaining the execution information corresponding to the determined category.

According to an embodiment of the present disclosure, the operation of obtaining the execution information includes, when a plurality of categories corresponding to the entity name are searched based on the public data, the user is asked which category among the plurality of categories he/she wants information belonging to. The method may include providing a first user interface for querying. The operation of acquiring the execution information may include obtaining the execution information corresponding to the category selected by the user input on the first user interface.

According to an embodiment of the present disclosure, the operation of obtaining the execution information includes searching for a plurality of categories corresponding to the entity name based on the public data, and searching for one category among the plurality of categories based on the personal data. If it is possible to specify, it may include an operation of acquiring the execution information corresponding to the specified one category.

According to an embodiment of the present disclosure, the method includes providing a second user interface including a popularly used intent list when there is no data to be retrieved from the database based on the entity name. It can be included. The method may include performing an action corresponding to the intention selected by user input on the second user interface.

According to an embodiment of the present disclosure, the method may include an operation of determining whether a domain or intention can be specified based on the text when the text consists of one entity name. The method may include an operation of specifying the domain or the intent using the obtained execution information when the domain or the intent cannot be specified. The method may include performing an action corresponding to the voice command based on the specified domain or intention.

According to an embodiment of the present disclosure, the method includes a domain or intention in the domain to perform an action on at least one entity name through an input device (1150; 1260; 1270) of the electronic device. It may include the action of receiving user input that specifies . The method is configured to perform an operation corresponding to the voice command according to the specified domain or intent when a voice command containing any one of the at least one entity name is received based on the received user input. It may include actions such as:

According to an embodiment of the present disclosure, the method includes, when the operation performed by executing the voice command corresponds to the intention of the user of the electronic device, the information used in executing the voice command is treated as the personal data. It may include a saving operation. The information used to execute the voice command is at least one of the entity name, the category corresponding to the entity name, the number of times the entity name is uttered, the domain that performed the operation, or the intention corresponding to the operation in the domain. It can contain one.

According to an embodiment of the present disclosure, the method is based on the entity name usage history in each of at least one application stored in the electronic device, and the entity name extracted from the entity name usage history in each application corresponds to each application. It may include an operation of associating the domain or the intention of using the extracted entity name and storing it in the database.

According to an embodiment of the present disclosure, a non-transitory computer-readable storage medium may store a program for performing a method of operating an electronic device. The method of operating the electronic device may include receiving a voice command through a microphone (110; 1150; 1270) of the electronic device. The method may include determining whether text corresponding to the voice command consists of one named entity. The method may include obtaining execution information for executing a voice command based on the entity name. The execution information may include domain information indicating an application executing the voice command or intent information indicating a function of the application requested by the voice command. The method may include executing the voice command using the execution information.

Electronic devices according to various embodiments disclosed in this document may be of various types. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to embodiments of this document are not limited to the above-described devices.

The various embodiments of this document and the terms used herein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various changes, equivalents, or replacements of the embodiments. In connection with the description of the drawings, similar reference numbers may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the above items, unless the relevant context clearly indicates otherwise. As used herein, “A or B”, “at least one of A and B”, “at least one of A or B”, “A, B or C”, “at least one of A, B and C”, and “A Each of phrases such as “at least one of , B, or C” may include any one of the items listed together in the corresponding phrase, or any possible combination thereof. Terms such as "first", "second", or "first" or "second" may be used simply to distinguish one component from another, and to refer to that component in other respects (e.g., importance or order) is not limited. One (e.g., first) component is said to be “coupled” or “connected” to another (e.g., second) component, with or without the terms “functionally” or “communicatively.” When mentioned, it means that any of the components can be connected to the other components directly (e.g. wired), wirelessly, or through a third component.

The term “module” used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and is interchangeable with terms such as logic, logic block, component, or circuit, for example. It can be used as A module may be an integrated part or a minimum unit of the parts or a part thereof that performs one or more functions. For example, according to one embodiment, the module may be implemented in the form of an application-specific integrated circuit (ASIC).

Various embodiments of this document are one or more instructions stored in a storage medium (e.g., built-in memory 1136 or external memory 1138) that can be read by a machine (e.g., electronic device 1101). It may be implemented as software (e.g., program 1140) including these. For example, a processor (e.g., processor 1120) of a device (e.g., electronic device 1101) may call at least one command among one or more commands stored from a storage medium and execute it. This allows the device to be operated to perform at least one function according to the at least one instruction called. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. A storage medium that can be read by a device may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' only means that the storage medium is a tangible device and does not contain signals (e.g. electromagnetic waves), and this term refers to cases where data is semi-permanently stored in the storage medium. There is no distinction between temporary storage cases.

According to one embodiment, methods according to various embodiments disclosed in this document may be included and provided in a computer program product. Computer program products are commodities and can be traded between sellers and buyers. The computer program product may be distributed in the form of a machine-readable storage medium (e.g. compact disc read only memory (CD-ROM)) or through an application store (e.g. Play Store ^™ ) or on two user devices (e.g. It can be distributed (e.g. downloaded or uploaded) directly between smart phones) or online. In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily created in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

According to various embodiments, each component (e.g., module or program) of the above-described components may include a single or plural entity, and some of the plurality of entities may be separately placed in other components. there is. According to various embodiments, one or more of the components or operations described above may be omitted, or one or more other components or operations may be added. Alternatively or additionally, multiple components (eg, modules or programs) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each component of the plurality of components in the same or similar manner as those performed by the corresponding component of the plurality of components prior to the integration. . According to various embodiments, operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, or omitted. Alternatively, one or more other operations may be added.

Claims (20)

In the electronic device (101;602;702;802;901;1101;1201),
mic(110;1150;1270);
communication circuit (120;1190;1290);
a memory (130;1130;1230) for storing instructions; and
Includes a processor (140; 1120; 1220),
The instructions, when executed by the processor, cause the electronic device to:
Receive voice commands through the microphone,
Determine whether the text corresponding to the voice command consists of one named entity,
When the text consists of one entity name, execution information for executing the voice command is obtained based on the entity name, and the execution information is domain information indicating an application executing the voice command or the voice command. Contains intent information indicating the requested function of the application,
An electronic device that executes the voice command using the execution information. In claim 1,
The instructions, when executed by the processor, cause the electronic device to:
If the text consists of one entity name and there are two or more domains corresponding to the entity name, the execution information is obtained based on the entity name,
When the text consists of one entity name and there is one domain corresponding to the entity name, an electronic device determines the intent in the domain and performs a function corresponding to the determined intent. The method according to any one of claims 1 to 2,
The instructions, when executed by the processor, cause the electronic device to:
An electronic device that, when the text does not consist of one entity name, determines intent through natural language processing of the text and performs a function corresponding to the determined intent. The method according to any one of claims 1 to 3,
The instructions, when executed by the processor, cause the electronic device to:
An electronic device that searches a database 235 containing public data 235-2 and personal data 235-1 based on the string and pronunciation string of the entity name. The method according to any one of claims 1 to 4,
The instructions, when executed by the processor, cause the electronic device to:
If a plurality of categories corresponding to the entity name are searched based on the public data, obtain ratio information for each category,
Determine the category corresponding to the entity name based on the ratio information,
An electronic device configured to obtain the execution information corresponding to the determined category. The method according to any one of claims 1 to 5,
The instructions, when executed by the processor, cause the electronic device to:
When a plurality of categories corresponding to the entity name are searched based on the public data, providing a first user interface that asks the user which category of the plurality of categories he or she wants information belonging to,
An electronic device configured to obtain the execution information corresponding to a category selected by a user input on the first user interface. The method according to any one of claims 1 to 6,
The instructions, when executed by the processor, cause the electronic device to:
If a plurality of categories corresponding to the entity name are searched based on the public data, and one category among the plurality of categories can be specified based on the personal data, the execution information corresponding to the specified one category is retrieved. An electronic device that allows acquisition. The method according to any one of claims 1 to 7,
The instructions, when executed by the processor, cause the electronic device to:
If there is no data to be searched in the database based on the entity name, providing a second user interface including a list of popularly used intents,
An electronic device that performs an action corresponding to an intention selected by a user input to the second user interface. The method according to any one of claims 1 to 8,
The instructions, when executed by the processor, cause the electronic device to:
If the text consists of one entity name, determine whether the domain or intention can be specified based on the text,
If the domain or the intention cannot be specified, the domain or the intention is specified using the obtained execution information,
An electronic device that performs an action corresponding to the voice command based on the specified domain or intent. The method according to any one of claims 1 to 9,
The electronic device further includes an input device (1150; 1260; 1270),
The instructions, when executed by the processor, cause the electronic device to:
Receiving, through the input device, a user input specifying a domain to perform an action on at least one entity name or an intention in the domain,
Based on the received user input, when a voice command including any one of the at least one entity name is received, the electronic device is set to perform an operation corresponding to the voice command according to the designated domain or intent. Device. The method according to any one of claims 1 to 10,
The instructions, when executed by the processor, cause the electronic device to:
If the operation performed by executing the voice command corresponds to the intention of the user of the electronic device, store the information used in executing the voice command as the personal data,
The information used to execute the voice command is at least one of the entity name, the category corresponding to the entity name, the number of times the entity name is uttered, the domain that performed the operation, or the intention corresponding to the operation in the domain. An electronic device, including one. The method according to any one of claims 1 to 11,
The memory stores at least one application,
The instructions, when executed by the processor, cause the electronic device to:
Based on the entity name usage history in each of the at least one application, the entity name extracted from the entity name usage history in each application is associated with the domain corresponding to each application or the intention for using the extracted entity name and stored in the database. An electronic device that allows storage. In a method of operating an electronic device (101;602;702;802;901;1101;1201),
Receiving a voice command through a microphone (110; 1150; 1270) of the electronic device;
An operation of determining whether text corresponding to the voice command consists of one named entity;
When the text consists of one entity name, execution information for executing a voice command is obtained based on the entity name, and the execution information is domain information indicating an application executing the voice command or a request for the voice command. An operation including intent information indicating a function of the application to perform; and
A method comprising executing the voice command using the execution information. In claim 13,
The operation of obtaining the execution information is performed when the text consists of one entity name and there are two or more domains corresponding to the entity name,
In the method, when the text consists of one entity name and there is one domain corresponding to the entity name,
determining intent in the domain; and
A method comprising performing a function corresponding to the determined intent. The method of any one of claims 13 to 14,
The method includes the operation of determining intent through natural language processing for the text when the text does not consist of a single entity name, and performing a function corresponding to the determined intent. The method of any one of claims 13 to 15,
The operation of acquiring the execution information involves creating a database 235 containing public data 235-2 and personal data 235-1 based on the string and pronunciation string of the entity name. A method, including the operation of searching. The method of any one of claims 13 to 16,
The operation of obtaining the execution information is,
When a plurality of categories corresponding to the entity name are searched based on the public data, obtaining ratio information for each category;
determining the category corresponding to the entity name based on the ratio information; and
A method comprising obtaining the execution information corresponding to the determined category. The method of any one of claims 13 to 17,
The operation of obtaining the execution information is,
When a plurality of categories corresponding to the entity name are searched based on the public data, providing a first user interface to ask the user which of the plurality of categories information belongs to; and
A method comprising obtaining the execution information corresponding to a category selected by a user input on the first user interface. The method of any one of claims 13 to 18,
The operation of obtaining the execution information is,
If a plurality of categories corresponding to the entity name are searched based on the public data, and one category among the plurality of categories can be specified based on the personal data, the execution information corresponding to the specified one category is retrieved. A method, comprising the operation of acquiring. The method of any one of claims 13 to 19,
If no data is retrieved from the database based on the entity name, providing a second user interface including a list of popularly used intents; and
The method further includes performing an action corresponding to the intention selected by the user input on the second user interface.

Images