KR20220141137A - Electronic apparatus for processing user utterance and controlling method thereof - Google Patents

Abstract

In accordance with one embodiment of the present disclosure, an electronic device includes: a memory configured to store at least one piece of voiceprint data corresponding to at least one user; a communication circuit transmitting a signal for controlling at least one external device; and at least one processor electrically coupled with the memory and the communication circuit, wherein the at least one processor can identify voiceprint data corresponding to the user among the at least one piece of voiceprint data based on received user voice data, identify a general voice command included in the received user voice data, determine user preference information of the user based on the identified voiceprint data, determine a control operation for determining an operation to be performed by at least one external device or the electronic device based on the general voice command identified in the received user voice data, determine a personalized voice command based on at least one of the control operation and the user preference information, and transmit an audio signal corresponding to the personalized voice command such that the at least one external device outputs the audio signal through the communication circuit or output the audio signal through a speaker included in the electronic device. Therefore, the present invention is capable of improving the accuracy of a voice recognition service.

Description

An electronic device for processing a user's utterance, and a control method of the electronic device

Various embodiments relate to a technology for recognizing a user's voice command and performing an operation corresponding to the user's voice command.

An artificial intelligence system (or integrated intelligence system) is a computer system that implements human-level intelligence. It is a system in which a machine learns and judges on its own, and the recognition rate improves the more it is used. Artificial intelligence technology consists of machine learning (deep learning) technology that uses an algorithm that classifies/learns the characteristics of input data by itself, and element technology that uses machine learning algorithms to simulate functions such as cognition and recognition of the human brain.

The element technologies are, for example, linguistic understanding technology for recognizing human language/text, visual understanding technology for recognizing objects as if they were human eyes, reasoning/prediction technology for logically reasoning and predicting by judging information, human experience information It may include at least one of a knowledge expression technology that processes the data as knowledge data, an autonomous driving of a vehicle, and a motion control technology that controls the movement of a robot. Among the above-described element technologies, linguistic understanding is a technology for recognizing and applying/processing human language/text, and includes natural language processing, machine translation, dialogue system, question answering, voice recognition/synthesis, and the like.

On the other hand, the electronic device equipped with the artificial intelligence system analyzes voice data input by the user's utterance, determines a function to be performed by the electronic device or an external electronic device connected to the electronic device based on the analyzed result, and controls the An operation may be supported to be performed by an electronic device or an external electronic device.

Recently, the number of electronic devices providing a voice recognition function is increasing. The electronic device, which recognizes the user's utterance, may provide a service to the user by executing a function built into the electronic device (eg, an application installed in the electronic device), and may use various third party service providers. may also provide related services.

Recently, speech recognition services are being developed based on a technology for processing natural language. The technology for processing natural language is a technology for recognizing the intention of a user's utterance and providing a result that matches the intention to the user.

On the other hand, various electronic devices that provide a voice recognition function can provide various services to users by executing functions built into the electronic devices, but have a disadvantage in that it is difficult for the user to fully understand and use the service usage methods. In addition, there is a disadvantage that a predetermined utterance must be performed in order to use various voice recognition services.

In addition, when it is necessary to utter a predetermined voice command to execute the function of the voice recognition service, there is a disadvantage in that it is difficult for various users to control the device based on the voice command personalized for each user.

From the point of view of a user who uses the voice recognition service, when there is a command that can be continuously performed from a voice command according to a user's utterance, the usability of the voice recognition service may be increased by the user being aware of the command.

Depending on the situation of a user using the voice recognition service, it may be necessary to provide a different service even for the same text-to-speech command.

The electronic device according to an embodiment includes a memory configured to store at least one glottal data corresponding to at least one user, a communication circuit for transmitting a signal for controlling at least one external device, and a memory and electrically connected to the communication circuit at least one processor, wherein the at least one processor identifies voiceprint data corresponding to the user from among the at least one voiceprint data based on the received user voice data, and identifies a general voice command included in the received user voice data. a control operation for determining user preference information of the user based on the identified voiceprint data, and determining an operation to be performed in the at least one external device or electronic device based on a general voice command identified in the received user voice data determines, determines a personalized voice command based on at least one of a control operation and user preference information, and transmits or includes an audio signal corresponding to the personalized voice command to be output by at least one external device through a communication circuit or included in an electronic device An audio signal can be output through a built-in speaker.

According to an embodiment, a method of operating an electronic device including a memory configured to store at least one voiceprint data corresponding to at least one user and a communication circuit for transmitting a signal for controlling at least one external device includes a user voice Receiving data, identifying voiceprint data corresponding to the user from among the at least one voiceprint data, identifying a general voice command included in the received user voice data, user preference of the user based on the identified voiceprint data Determining information and determining a personalized voice command based on at least one of an operation for determining a control operation based on an operation to be performed in at least one external device or an electronic device based on a general voice command, a control operation, and user preference information The operation may include transmitting an audio signal corresponding to the operation and the personalized voice command to be output to at least one external device through a communication circuit or outputting the audio signal through a speaker included in the electronic device.

An electronic device and an operating method thereof according to various embodiments of the present disclosure may output a voice command for using a voice recognition service as an audio signal so that the user is familiar with it so that the user can more easily use the voice command service.

In an electronic device and an operating method thereof according to various embodiments of the present disclosure, based on user preference information, as a voice recognition service can be used through a personalized voice command corresponding to a general voice command, users can use the personalized voice command. The usability can be increased by being able to control the device based on it.

An electronic device and an operating method thereof according to various embodiments of the present disclosure provide a voice recognition service based on state information associated with a user using a voice recognition service, thereby providing a voice recognition service based on only a user's utterance. It is possible to improve the accuracy of the voice recognition service.

An electronic device and an operating method thereof according to various embodiments of the present disclosure may provide a concise voice recognition service by performing an authentication operation based on a user's voice data in relation to at least one external device.

In addition, various effects directly or indirectly identified through this document may be provided.

1 is a block diagram illustrating an integrated intelligence system according to an embodiment.
2 is a diagram illustrating a form in which relation information between a concept and an action is stored in a database, according to an embodiment.
3 is a block diagram of an electronic device according to an exemplary embodiment.
4 is a diagram illustrating an on-device natural language understanding platform according to an embodiment.
5 is a diagram illustrating a connection system between an electronic device and an external electronic device according to an exemplary embodiment.
6 illustrates various modules related to personalized voice commands and additional control operations according to an embodiment.
7 is a flowchart illustrating outputting a personalized voice command signal and performing an operation of an electronic device according to an exemplary embodiment.
8 is a flowchart illustrating an operation of determining user preference information of an electronic device according to an exemplary embodiment.
9 is a flowchart illustrating an operation of determining a personalized voice command of an electronic device according to an embodiment.
10 is a flowchart illustrating an operation of determining an additional control operation of an electronic device according to an exemplary embodiment.
11 is a flowchart illustrating an operation of an electronic device according to an exemplary embodiment.
12 is a diagram illustrating an electronic device displaying a screen for processing a voice input received through an intelligent app, according to an embodiment.
13A is a diagram illustrating an operation of controlling an external device through a voice command, according to an embodiment.
13B is a diagram illustrating an operation of outputting a personalized voice command based on content being reproduced, according to an embodiment.
13C is a diagram illustrating an operation of outputting a personalized command list through an external device, according to an embodiment.
13D is a diagram illustrating an operation of controlling the use of an external device based on a voice command, according to an embodiment.
14 is a block diagram of an electronic device in a network environment according to an embodiment.
In connection with the description of the drawings, the same or similar reference numerals may be used for the same or similar components.

1 is a block diagram illustrating an integrated intelligence system according to an embodiment.

Referring to FIG. 1 , the integrated intelligent system according to an embodiment may include a user terminal 100 , an intelligent server 200 , and a service server 290 .

The user terminal 100 of an embodiment may be a terminal device (or electronic device) connectable to the Internet, for example, a mobile phone, a smart phone, a personal digital assistant (PDA), a notebook computer, a TV (television), It may be a white goods appliance, a wearable device, a head mounted device (HMD), or a smart speaker.

According to the illustrated embodiment, the user terminal 100 may include a communication interface 110 , a microphone 120 , a speaker 130 , a display 140 , a memory 150 , or a processor 160 . The components listed above may be operatively or electrically connected to each other.

The communication interface 110 according to an embodiment may be configured to transmit and receive data by being connected to an external device. The microphone 120 according to an embodiment may receive a sound (eg, a user's utterance) and convert it into an electrical signal. The speaker 130 according to an exemplary embodiment may output an electrical signal as a sound (eg, voice). Display 140 of an embodiment may be configured to display an image or video. The display 140 according to an embodiment may also display a graphic user interface (GUI) of an executed app (or an application program).

The memory 150 according to an embodiment may store a client module 151 , a software development kit (SDK) 153 , and a plurality of apps 155 . The client module 151 and the SDK 153 may constitute a framework (or solution program) for performing general functions. In addition, the client module 151 or the SDK 153 may configure a framework for processing a voice input.

The plurality of apps 155 may be programs for performing a specified function. According to an embodiment, the plurality of apps 155 may include a first app 155_1 and/or a second app 155_3 . According to an embodiment, each of the plurality of apps 155 may include a plurality of operations for performing a specified function. For example, the apps may include an alarm app, a message app, and/or a schedule app. According to an embodiment, the plurality of apps 155 may be executed by the processor 160 to sequentially execute at least some of the plurality of operations.

The processor 160 according to an embodiment may control the overall operation of the user terminal 100 . For example, the processor 160 may be electrically connected to the communication interface 110 , the microphone 120 , the speaker 130 , and the display 140 to perform a specified operation. For example, the processor 160 may include at least one processor.

The processor 160 according to an embodiment may also execute a program stored in the memory 150 to perform a designated function. For example, the processor 160 may execute at least one of the client module 151 and the SDK 153 to perform the following operation for processing a voice input. The processor 160 may control the operation of the plurality of apps 155 through, for example, the SDK 153 . The following operations described as operations of the client module 151 or the SDK 153 may be operations performed by the execution of the processor 160 .

The client module 151 according to an embodiment may receive a voice input. For example, the client module 151 may receive a voice signal corresponding to the user's utterance sensed through the microphone 120 . The client module 151 may transmit a received voice input (eg, a voice signal) to the intelligent server 200 . The client module 151 may transmit the state information of the user terminal 100 to the intelligent server 200 together with the received voice input. The state information may be, for example, execution state information of an app.

The client module 151 according to an embodiment may receive a result corresponding to the received voice input. For example, when the intelligent server 200 can calculate a result corresponding to the received voice input, the client module 151 may receive a result corresponding to the received voice input. The client module 151 may display the received result on the display 140 .

The client module 151 according to an embodiment may receive a plan corresponding to the received voice input. The client module 151 may display a result of executing a plurality of operations of the app according to the plan on the display 140 . The client module 151 may, for example, sequentially display execution results of a plurality of operations on the display. As another example, the user terminal 100 may display only a partial result of executing a plurality of operations (eg, a result of the last operation) on the display.

According to an embodiment, the client module 151 may receive a request for obtaining information necessary for calculating a result corresponding to a voice input from the intelligent server 200 . According to an embodiment, the client module 151 may transmit the necessary information to the intelligent server 200 in response to the request.

The client module 151 according to an embodiment may transmit result information of executing a plurality of operations according to a plan to the intelligent server 200 . The intelligent server 200 may confirm that the received voice input has been correctly processed using the result information.

The client module 151 according to an embodiment may include a voice recognition module. According to an embodiment, the client module 151 may recognize a voice input performing a limited function through the voice recognition module. For example, the client module 151 may execute an intelligent app for processing a voice input by performing an organic operation in response to a specified voice input (eg, wake up!).

The intelligent server 200 according to an embodiment may receive information related to a user's voice input from the user terminal 100 through a communication network. According to an embodiment, the intelligent server 200 may change data related to the received voice input into text data. According to an embodiment, the intelligent server 200 may generate at least one plan for performing a task corresponding to a user's voice input based on the text data.

According to one embodiment, the plan may be generated by an artificial intelligent (AI) system. The artificial intelligence system may be a rule-based system, a neural network-based system (eg, a feedforward neural network (FNN)), and/or a recurrent neural network network(RNN))). Alternatively, it may be a combination of the above or other artificial intelligence systems. According to an embodiment, the plan may be selected from a set of predefined plans or may be generated in real time in response to a user request. For example, the artificial intelligence system may select at least one plan from among a plurality of predefined plans.

The intelligent server 200 of an embodiment may transmit a result according to the generated plan to the user terminal 100 or transmit the generated plan to the user terminal 100 . According to an embodiment, the user terminal 100 may display a result according to the plan on the display. According to an embodiment, the user terminal 100 may display the result of executing the operation according to the plan on the display.

The intelligent server 200 of an embodiment includes a front end 210, a natural language platform 220, a capsule database 230, an execution engine 240, It may include an end user interface 250 , a management platform 260 , a big data platform 270 , or an analytics platform 280 .

The front end 210 according to an embodiment may receive a voice input received from the user terminal 100 . The front end 210 may transmit a response corresponding to the voice input to the user terminal 100 .

According to an embodiment, the natural language platform 220 includes an automatic speech recognition module (ASR module) 221 , a natural language understanding module (NLU module) 223 , a planner module ( planner module 225 , a natural language generator module (NLG module) 227 , and/or a text to speech module (TTS module) 229 .

The automatic voice recognition module 221 according to an embodiment may convert a voice input received from the user terminal 100 into text data. The natural language understanding module 223 according to an embodiment may recognize the user's intention by using text data of the voice input. For example, the natural language understanding module 223 may determine the user's intention by performing syntactic analysis or semantic analysis. The natural language understanding module 223 according to an embodiment recognizes the meaning of a word extracted from a voice input using a linguistic feature (eg, a grammatical element) of a morpheme or phrase, and matches the meaning of the identified word with the intention of the user. You can decide your intentions.

The planner module 225 according to an embodiment may generate a plan using the intent and parameters determined by the natural language understanding module 223 . According to an embodiment, the planner module 225 may determine a plurality of domains required to perform a task based on the determined intention. The planner module 225 may determine a plurality of operations included in each of the plurality of domains determined based on the intention. According to an embodiment, the planner module 225 may determine a parameter required to execute the determined plurality of operations or a result value output by the execution of the plurality of operations. The parameter and the result value may be defined as a concept of a specified format (or class). Accordingly, the plan may include a plurality of actions and/or a plurality of concepts determined by the user's intention. The planner module 225 may determine the relationship between the plurality of operations and the plurality of concepts in stages (or hierarchically). For example, the planner module 225 may determine the execution order of the plurality of operations determined based on the user's intention based on the plurality of concepts. In other words, the planner module 225 may determine the execution order of the plurality of operations based on parameters required for execution of the plurality of operations and results output by the execution of the plurality of operations. Accordingly, the planner module 225 may generate a plan including related information (eg, ontology) between a plurality of operations and a plurality of concepts. The planner module 225 may generate a plan using information stored in the capsule database 230 in which a set of relationships between concepts and operations is stored.

The natural language generation module 227 according to an embodiment may change the specified information into a text form. The information changed to the text form may be in the form of natural language utterance. The text-to-speech conversion module 229 according to an embodiment may change information in a text format into information in a voice format.

According to an embodiment, some or all of the functions of the natural language platform 220 may be implemented in the user terminal 100 .

The capsule database 230 may store information on relationships between a plurality of concepts and operations corresponding to a plurality of domains. A capsule according to an embodiment may include a plurality of action objects (or action information) and a concept object (or concept information) included in the plan. According to an embodiment, the capsule database 230 may store a plurality of capsules in the form of a concept action network (CAN). According to an embodiment, the plurality of capsules may be stored in a function registry included in the capsule database 230 .

The capsule database 230 may include a strategy registry in which strategy information necessary for determining a plan corresponding to a voice input is stored. The strategy information may include reference information for determining one plan when there are a plurality of plans corresponding to the voice input. According to an embodiment, the capsule database 230 may include a follow up registry in which information on a subsequent operation for suggesting a subsequent operation to the user in a specified situation is stored. The subsequent operation may include, for example, a subsequent utterance. According to an embodiment, the capsule database 230 may include a layout registry that stores layout information of information output through the user terminal 100 . According to an embodiment, the capsule database 230 may include a vocabulary registry in which vocabulary information included in the capsule information is stored. According to an embodiment, the capsule database 230 may include a dialog registry (dialog registry) in which dialog (or interaction) information with the user is stored. The capsule database 230 may update a stored object through a developer tool. The developer tool may include, for example, a function editor for updating an action object or a concept object. The developer tool may include a vocabulary editor for updating the vocabulary. The developer tool may include a strategy editor for creating and registering strategies for determining plans. The developer tool may include a dialog editor that creates a conversation with the user. The developer tool can include a follow up editor that can edit subsequent utterances that activate follow-up goals and provide hints. The subsequent goal may be determined based on a currently set goal, a user's preference, or an environmental condition. In an embodiment, the capsule database 230 may be implemented in the user terminal 100 .

The execution engine 240 according to an embodiment may calculate a result using the generated plan. The end user interface 250 may transmit the calculated result to the user terminal 100 . Accordingly, the user terminal 100 may receive the result and provide the received result to the user. The management platform 260 according to an embodiment may manage information used in the intelligent server 200 . The big data platform 270 according to an embodiment may collect user data. The analysis platform 280 of an embodiment may manage the quality of service (QoS) of the intelligent server 200 . For example, the analysis platform 280 may manage the components and processing speed (or efficiency) of the intelligent server 200 .

The service server 290 according to an embodiment may provide a specified service (eg, food order or hotel reservation) to the user terminal 100 . According to an embodiment, the service server 290 may be a server operated by a third party. The service server 290 according to an embodiment may provide information for generating a plan corresponding to the received voice input to the intelligent server 200 . The provided information may be stored in the capsule database 230 . In addition, the service server 290 may provide result information according to the plan to the intelligent server 200 .

In the integrated intelligent system described above, the user terminal 100 may provide various intelligent services to the user in response to a user input. The user input may include, for example, an input through a physical button, a touch input, or a voice input.

In an embodiment, the user terminal 100 may provide a voice recognition service through an intelligent app (or a voice recognition app) stored therein. In this case, for example, the user terminal 100 may recognize a user utterance or a voice input received through the microphone, and provide a service corresponding to the recognized voice input to the user. .

In an embodiment, the user terminal 100 may perform a specified operation alone or together with the intelligent server and/or the service server, based on the received voice input. For example, the user terminal 100 may execute an app corresponding to the received voice input and perform a specified operation through the executed app.

In an embodiment, when the user terminal 100 provides a service together with the intelligent server 200 and/or the service server, the user terminal detects a user's utterance using the microphone 120 , and the A signal (or voice data) corresponding to the sensed user's utterance may be generated. The user terminal may transmit the voice data to the intelligent server 200 using the communication interface 110 .

The intelligent server 200 according to an embodiment is a plan for performing a task corresponding to the voice input as a response to the voice input received from the user terminal 100, or performs an operation according to the plan. results can be generated. The plan may include, for example, a plurality of actions for performing a task corresponding to a user's voice input and/or a plurality of concepts related to the plurality of actions. The concept may define parameters input to the execution of the plurality of operations or result values output by the execution of the plurality of operations. The plan may include association information between a plurality of actions and/or a plurality of concepts.

The user terminal 100 according to an embodiment may receive the response using the communication interface 110 . The user terminal 100 outputs a voice signal generated inside the user terminal 100 to the outside using the speaker 130 or an image generated inside the user terminal 100 using the display 140 to the outside. can be output as

2 is a diagram illustrating a form in which relation information between a concept and an operation is stored in a database, according to various embodiments of the present disclosure;

The capsule database (eg, the capsule database 230 ) of the intelligent server 200 may store the capsule in the form of a concept action network (CAN). The capsule database may store an operation for processing a task corresponding to a user's voice input and parameters necessary for the operation in the form of a concept action network (CAN).

The capsule database may store a plurality of capsules (capsule(A) 401, capsule(B) 404) corresponding to each of a plurality of domains (eg, applications). According to an embodiment, one capsule (eg, capsule(A) 401 ) may correspond to one domain (eg, location (geo), application). Also, at least one service provider (eg, CP service A 291 or CP service B 292 ) for performing a function for a domain related to the capsule may correspond to one capsule. According to an embodiment, one capsule may include at least one operation 410 and at least one concept 420 for performing a specified function.

The natural language platform 220 may generate a plan for performing a task corresponding to the received voice input using the capsule stored in the capsule database. For example, the planner module 225 of the natural language platform may generate a plan using a capsule stored in a capsule database. For example, using operations 4011 , 4013 and concepts 4012 , 4014 of capsule A 410 and operations 4041 and concept 4042 of capsule B 404 to generate plan 407 . can do.

3 is a block diagram of an electronic device according to an exemplary embodiment.

Referring to FIG. 3 , the electronic device 300 (eg, the user terminal 100 , the intelligent server 200 or the service server 290 of FIG. 1 ) includes at least one processor 310 , a memory 320 , and a microphone. 330 , a speaker 340 , and a communication circuit 350 . The electronic device 300 according to an embodiment may be the intelligent server 200 or the service server 290 . According to another embodiment, the electronic device 300 may represent the user terminal 100 . The configuration of the electronic device 300 according to an embodiment may vary depending on whether the electronic device 300 is a server or the user terminal 100 . For example, when the electronic device 300 represents the service server 290 , the electronic device 300 may include a memory 320 and a communication circuit 350 .

The operations to be described below of the electronic device 300 according to the present disclosure may be performed in different ways depending on the configuration of the electronic device 300 . For example, when the electronic device 300 does not include the microphone 330 and the speaker 340 , the electronic device 300 performs the same operation as the external device including the microphone and the speaker using the communication circuit 350 . can be controlled to perform.

According to an embodiment, components included in the electronic device 300 may be operatively or electrically connected to each other. Components of the electronic device 300 illustrated in FIG. 3 may be partially modified, deleted, or added as an example. According to an embodiment, the electronic device 300 may include all or some of the components of the user terminal 100 of FIG. 1 . According to various embodiments, the electronic device 300 may communicate with the intelligent server 200 of FIG. 1 and may perform operations according to various embodiments alone or jointly.

According to another embodiment, the electronic device 300 may include all or some of the components of the intelligent server 200 or the service server 290 of FIG. 1 .

According to an embodiment, the processor 310 may be electrically or operatively connected to the memory 320 , the microphone 330 , the speaker 340 , and the communication circuit 350 . According to an embodiment, the processor 310 may execute an operation or data processing related to control and/or communication of at least one other component of the electronic device 300 using instructions stored in the memory 320 . According to an embodiment, the processor 310 includes a central processing unit (CPU), a graphics processing unit (GPU), a micro controller unit (MCU), a sensor hub, a supplementary processor, a communication processor, and an application. It may include at least one of a processor (application processor), application specific integrated circuit (ASIC), and field programmable gate arrays (FPGA), and may have a plurality of cores.

According to an embodiment, the processor 310 may receive a voice input by the user's utterance through the microphone 330 and process the received voice input. Also, the processor 310 may convert the received voice input into text data and perform a function corresponding to the received voice input based on the text data. For example, the processor 310 may convert a received voice input into text data and perform a function corresponding to a general voice command or a personalized voice command extracted through the conversion. According to another embodiment, the processor 310 may output voice data related to function performance through the speaker 340 in response to the received voice input. In order to perform the above-described function, the processor 310 may include a hardware structure specialized for processing a voice recognition model and/or instructions.

According to another embodiment, the electronic device 300 may receive user voice data including a general voice command. For example, the electronic device 300 may receive voice data acquired through the microphone 120 of the user terminal 100 through the communication circuit 350 .

According to an embodiment, the memory 320 provides instructions for the processor 310 to process data or control components of the electronic device 300 to perform an operation of the electronic device 300 when executed. ) can be stored. According to an embodiment, the memory 320 may store at least one application capable of performing various functions. According to an embodiment, the memory 320 may store commands and data related to processing of a voice input. For example, when a command related to processing of a voice input is executed by the processor 310 , the processor 310 may perform a function related to processing of a voice input that is a function corresponding to the command.

According to various embodiments, the memory 320 may store various information related to at least one user of the electronic device 300 . According to an embodiment, the memory 320 may store data related to a voice command. For example, the memory 320 may store voiceprint data corresponding to each of at least one user of the electronic device 300 . According to an embodiment, the processor 310 may identify the user of the electronic device 300 through the voiceprint data. Also, the memory 320 may store voice command history information corresponding to each of at least one user of the electronic device 300 , age information, or preference information input from each of the at least one user. This will be described later with reference to FIG. 6 .

According to an embodiment, the memory 320 may store device information on each of at least one external device connected to the electronic device 300 . For example, the memory 320 may store state information of at least one external device connected to the electronic device 300 . According to an embodiment, at least one external device connected to the electronic device 300 will be described later with reference to FIG. 5 .

According to an embodiment, the microphone 330 may receive a user's utterance and convert it into an electrical signal. For example, the microphone 330 may receive a user utterance including a general voice command from at least one user of the electronic device 300 and convert it into an electrical signal. According to an embodiment, the processor 310 may determine the general voice command received from the user by using the electrical signal received through the microphone 330 . According to another embodiment, the microphone 330 may receive a user utterance including a personalized voice command from at least one user of the electronic device 300 and convert it into an electrical signal. According to an embodiment, the processor 310 may determine the voice command received from the user using the electrical signal received through the microphone 330 . According to various embodiments, the processor 310 identifies the voice command included in the user's utterance will be described below with reference to FIG. 4 .

According to another embodiment, the electronic device 300 may receive user voice data converted into an electrical signal using the communication circuit 350 . According to various embodiments, the user voice data may include a general voice command or a personalized voice command.

According to an embodiment, the speaker 340 may output sound. For example, the speaker 340 may receive an electrical signal corresponding to voice data to be output from the processor 310 , convert the received electrical signal into an audio signal, and output the received electrical signal. According to an embodiment, the speaker 340 may receive electrical signals corresponding to various voice data from the processor 310 . For example, the speaker 340 may receive an electrical signal corresponding to the personalized voice command from the processor 310 , and may convert the electrical signal into an audio signal and output it. For another example, the speaker 340 may receive an electrical signal corresponding to a notification related to the function execution of the electronic device 300 from the processor 310 , convert it into an audio signal, and output the received signal.

According to another embodiment, the electronic device 300 may generate an audio signal and transmit the generated audio signal to an external device including a speaker using the communication circuit 350 . For example, the processor 310 may transmit an audio signal corresponding to the voice data to be output through the communication circuit 350 so that the external device may output the audio signal.

According to an embodiment, the electronic device 300 may communicate with the intelligent server 200 and/or the service server 290 described with reference to FIG. 1 through the communication circuit 350 . The electronic device 300 may transmit voice data based on the user's utterance to the intelligent server 200 through the communication circuit 350 . Also, the electronic device 300 may transmit the voice data based on the user's utterance from the intelligent server 200 to the communication circuit 350 . can be used to receive a response. According to another embodiment, the electronic device 300 may communicate with the user terminal 100 described with reference to FIG. 1 through the communication circuit 350 .

According to an embodiment, the electronic device 300 may communicate with at least one external device connected to the electronic device 300 through the communication circuit 350 . For example, the electronic device 300 may receive status information of the external device for each of the at least one external device through the communication circuit 350 . According to an embodiment, the processor 310 may generate a control signal for controlling the at least one external device and transmit the control signal through the communication circuit 350 . According to an embodiment, the electronic device 300 transmits/receives data to and from at least one external device through the communication circuit 350 will be described below in detail with reference to FIG. 5 .

According to an embodiment, the processor 310, for example, the processor 310 receives user voice data including a general voice command, and identifies voiceprint data corresponding to the user based on the received user voice data. can In an embodiment, the processor 310 may use at least one voiceprint data stored in the memory 320 to identify the voiceprint data. According to an embodiment, the processor 310 may determine the age of the user through the received voice. For example, the processor 310 may identify voiceprint data through the received voice, and determine the age of the user corresponding to the identified voiceprint data. The age of the user corresponding to the voiceprint data may be determined based on the user's input data or characteristics of the voiceprint data. According to various embodiments, the age of the user may be classified in various ways. For example, the age of the user may be divided into whether the user is a child or an adult.

According to an embodiment, the processor 310 may determine the user's preference information based on the identified voiceprint data. The preference information according to various embodiments may be determined based on various types of information. For example, the processor 310 may determine user preference information based on voice command history information stored in the memory 320 , age information, or preference information input from the user. According to an embodiment, the processor 310 determines user preference information based on preference information input from the user, and if there is no preference information input from the user, user preference information based on the user's voice command history information can judge According to an embodiment, when the user's voice command history information is not stored in the memory 320 , the processor 310 may determine user preference information based on the user's age information. According to an embodiment, the processor 310 may receive various pieces of information through an external server to determine user preference information based on user age information. For example, the processor 310 may determine user preference information based on a database based on user preference information for each age group stored in an external server or a database based on a viewer share based on age for each time zone.

According to another embodiment, when a general voice command by a user's speech using the electronic device 300 is received for the first time, the processor 310 may not identify the voiceprint data. According to an embodiment, when the processor 310 cannot identify the user's voiceprint data, the processor 310 may perform a limited operation. For example, when the operation corresponding to the general voice command requires security authentication, the processor 310 may perform at least some operations in response to the inability to identify the voiceprint data.

According to an embodiment, the processor 310 may determine a personalized voice command and a control operation based on at least the determined user preference information and a general voice command. According to an embodiment, the control operation determined by the processor 310 may refer to the plan described with reference to FIGS. 1 and 2 . According to an embodiment, the personalized voice command may indicate a command for performing an operation corresponding to a general voice command. For example, an operation performed by the electronic device 300 based on a general voice command and an operation performed by the electronic device 300 based on a personalized voice command may include the same operation. According to an embodiment, the personalized voice command may determine a theme based on the user's user preference information, and may indicate a voice command corresponding to the general voice command in the determined theme. According to various embodiments, the theme may include various types. For example, the theme may include various types without limitation, such as animation, movie, entertainment program, entertainer, and the like. The content of the personalized voice command will be described later through the personalized command set 621 of FIG. 6 .

According to an embodiment, the control operation may indicate an operation to be performed by the electronic device 300 based on a general voice command received from the user. For example, when the general voice command received from the user is “Tell me about today’s schedule”, the control operation is performed by the processor 310 of the electronic device 300 based on data about today’s schedule stored in the memory 320 through the speaker ( 340) may represent outputting an audio signal for today's schedule. According to various embodiments, the personalized voice command and control operation may be determined in various ways.

According to an embodiment, the processor 310 may output an audio signal corresponding to the determined personalized voice command through the speaker 340 . For example, when the personalized voice command corresponding to the general voice command "Turn on the desk light" is "ABC", the processor 310 generates an audio signal corresponding to "If you shout ABC, you can turn on the light on the desk!" It may output through the speaker 340 .

4 is a diagram illustrating an on-device natural language understanding platform according to an embodiment.

According to an embodiment, the processor 310 of the electronic device 300 may include an on-device natural language understanding platform. According to an embodiment, the processor 310 may implement a natural language understanding platform based on a program stored in the memory 320 .

According to an embodiment, the processor 310 may include an automatic speech recognition module 311 , a natural language understanding module 312 , a planner module 313 , a natural language generation module 314 , and a text-to-speech conversion module 315 . have. Hereinafter, the configuration of the processor 310 including the automatic speech recognition module 311 , the natural language understanding module 312 , the planner module 313 , the natural language generation module 314 , and the text-to-speech conversion module 315 is on-device natural language It can be expressed as an understanding platform. The on-device natural language understanding platform may process the received user utterance complementary to the natural language platform 220 described with reference to FIG. 1 . According to an embodiment, the on-device natural language understanding platform of the electronic device 300 may process a voice input that may be processed by the natural language platform 220 .

According to an embodiment, the intelligent server 200 described with reference to FIG. 1 may change the voice input processed by the on-device natural language platform of the electronic device 300 . According to an embodiment, the processor of the intelligent server 200 may determine whether to change (or update) the natural language understanding model used in the natural language understanding module 312 of the electronic device 300 .

According to an embodiment, the automatic speech recognition module 311, the natural language understanding module 312, the planner module 313, and the text-to-speech module 315 may be embedded modules for performing a specified function. have. According to an embodiment, the electronic device 300 may receive a voice input. According to an embodiment, the electronic device 300 may process a voice input received through the automatic voice recognition module 311 , the natural language understanding module 312 , and the planner module 313 . According to an embodiment, the automatic speech recognition module 311 , the natural language understanding module 312 , and the planner module 313 of the electronic device 300 include the automatic speech recognition module 221 of the intelligent server 200 , the natural language A voice input may be processed similarly to the understanding module 223 and the planner module 225 . According to an embodiment, the natural language understanding module 312 may determine the intention of the received voice input. The natural language understanding module 312 may determine an intention corresponding to the voice input by using the natural language understanding model. According to an embodiment, the electronic device 300 may process a voice input corresponding to an intention through the automatic voice recognition module 311 , the natural language understanding module 312 , and the planner module 313 .

According to an embodiment, the automatic voice recognition module 311 may convert a voice input received through the microphone 330 into text data. For example, the automatic speech recognition module 311 may include a speech recognition module. The speech recognition module may include an acoustic model and a language model. For example, the acoustic model may include information related to vocalization, and the language model may include information on a combination of unit phoneme information and unit phoneme information. The speech recognition module may convert the user's speech into text data using information related to speech and information related to unit phonemes.

The natural language understanding module 312 may determine the user's intention by using text data of the voice input. For example, the natural language understanding module 312 may determine the user's intention by performing syntactic analysis or semantic analysis. According to an embodiment, the natural language understanding module 312 recognizes the meaning of a word extracted from text data by using a linguistic feature (eg, a grammatical element) of a morpheme or phrase, and matches the meaning of the identified word to an intention. to determine the user's intentions.

The planner module 313 may generate a plan using the intent and parameters determined by the natural language understanding module 312 . According to an embodiment, the planner module 313 may determine a plurality of domains required to perform a task (or function) based on the determined intention. The planner module 313 may determine a plurality of operations included in each of the plurality of domains determined based on the intention. According to an embodiment, the planner module 313 may determine a parameter required to execute the determined plurality of operations or a result value output by the execution of the plurality of operations. The parameter and the result value may be defined as a concept of a specified format (or class). Accordingly, the plan may include a plurality of actions and a plurality of concepts determined by the user's intention. The planner module 313 may determine the relationship between the plurality of operations and the plurality of concepts in stages (or hierarchically). For example, the planner module 313 may determine the execution order of the plurality of operations determined based on the user's intention based on the plurality of concepts. In other words, the planner module 313 may determine the execution order of the plurality of operations based on parameters necessary for the execution of the plurality of operations and results output by the execution of the plurality of operations. Accordingly, the planner module 313 may generate a plan including related information (eg, ontology) between a plurality of operations and a plurality of concepts.

The natural language generation module 314 may change the specified information into text form. The information changed to the text form may be in the form of natural language utterance. The specified information may be, for example, information guiding completion of an operation (or performing a control operation) corresponding to a voice input by a user's utterance or information guiding a user's additional input (eg, feedback information on user input). can According to an embodiment, the specified information may include a personalized voice command corresponding to the user's general voice command. Alternatively, the designated information may include a subsequent personalized command corresponding to a subsequent voice command expected after the user's general voice command. The information changed in the text format may be transmitted to the text-to-speech conversion module 315 and changed into information in the voice format.

The text-to-speech conversion module 315 may change information in a text format into information in a voice format. For example, the text-to-speech module 315 may change text data into voice data. The text-to-speech conversion module 315 may receive text-type information from the natural language generation module 314 , convert the text-type information into speech-type information (voice data), and output it to the speaker 340 .

As described above, the processor 310 converts the voice input received through the microphone 330 into text data through the automatic speech recognition module 311 and uses the converted text data through the natural language understanding module 312 . to determine the user's intention, and determine a control operation (or task) to be performed based on the identified user's intention through the planner module 313 . In addition, the processor 310 may output a response to the user's utterance corresponding to the voice input received through the microphone 330 through the speaker 340 . For example, the processor 310 changes information on a personalized voice command corresponding to a general voice command included in the voice input received as the response into text form through the natural language generation module 314, and converts the information changed into text form It may be converted into voice data through the text-to-speech conversion module 315 and output through the speaker 340 . The personalized voice command may be determined based on at least user preference information of the user and a general voice command of the user.

5 is a diagram illustrating a connection system between an electronic device and an external electronic device according to an exemplary embodiment.

Referring to FIG. 5 , a system according to an embodiment may include at least one external device 530 , a server 510 , and an electronic device 300 . At least one of the at least one external device 530 and the electronic device 300 may be connected to the server 510 through a system network 520 (eg, wi-fi or a cellular network). According to an embodiment, the at least one external device 530 may correspond to an Internet of things (IoT) device. According to an embodiment, the server 510 may represent an intelligent server (eg, the intelligent server 200 of FIG. 1 ) using machine learning and/or a neural network.

In an embodiment, the at least one external device 530 may represent a user device. According to an embodiment, the at least one external device 530 may include various devices. For example, in addition to the first device 531 , at least one of a second device 532 , a third device 533 , and a fourth device 534 may be included. For example, the first device 531 may be an image output electronic device such as a TV. The second device 532 may be a lighting device such as a lamp or a stand. The third device 533 may be an audio output electronic device such as a Bluetooth speaker. The fourth device 534 may be a home appliance such as a washing machine.

According to an embodiment, at least one external device 530 may be connected by an IoT hub device. According to an embodiment, the IoT hub device may perform voice recognition on a user's voice input to control a control operation of the IoT device to be performed. According to an embodiment, the electronic device 300 may correspond to an IoT hub device. Accordingly, when the at least one external device 530 is first connected to the electronic device 300 , the electronic device 300 may store information on the type and location of the at least one external device 530 .

According to an embodiment, at least one external device 530 may be connected to each other using a predetermined communication protocol. For example, it may be connected to at least one of the first to fourth devices 531 to 534 through a local area network. For example, a network (eg, a local area network) for establishing a connection between at least one external device 530 may be appropriately selected. For example, Bluetooth Low Energy (BLE), Wi-Fi direct, near field communication (NFC), ultra-wide band (UWB) communication, or infrared (Infra-red) communication with or instead of Bluetooth at least one It can be used to establish a connection between the external devices 530 of the. Also, according to an embodiment, the at least one external device 530 may establish a connection using a mesh network (eg, zigbee or Z-Wave) through short-range wireless communication.

According to an embodiment, at least one of the first to fourth devices 531 to 534 may be directly connected to the server 510 through the system network 520 . According to an embodiment, a method of connecting the at least one external device 530 to each other may vary according to device information (eg, device components).

According to an embodiment, the electronic device 300 may transmit/receive data to and from at least one external device 530 through the communication circuit 350 . According to an embodiment, the electronic device 300 may generate a control signal based on a general voice command received from the user and an additional control operation determined through the processor 310 . For example, the electronic device 300 may generate a plan based on a general voice command received from the user and determine an additional control operation. The electronic device 300 may generate a control signal based on the plan and the additional control operation, and transmit the control signal to an external device to perform an operation according to the control signal among at least one external device 530 . According to an embodiment, the external device receiving the control signal may perform an operation according to the plan and the additional control operation based on the control signal. For example, the electronic device 300 may receive a general voice command (eg, "Turn on TV") included in the user's utterance, and determine user preference information based on the user's voiceprint data. The electronic device 300 may determine a personalized voice command and an additional control operation corresponding to a general voice command based on user preference information. The operation of determining the additional control operation will be described later in detail with reference to FIG. 6 . According to an embodiment, the electronic device 300 generates a control signal based on a general voice command and an additional control operation, and an external device (eg, the first device 531 ) to perform an operation corresponding to the general voice command. can transmit a control signal. According to an embodiment, the external device (eg, the first device 531 ) that has received the control signal may perform an operation corresponding to the additional control operation (eg, output a set background music and output a set screen).

6 illustrates various modules related to personalized voice commands and additional control operations according to an embodiment.

Referring to FIG. 6 , the processor 310 of the electronic device 300 according to an embodiment may include a user data analysis module 610 , a personalized voice command analysis module 620 , and an additional control operation analysis module 630 . can According to an embodiment, the processor 310 may implement the module based on the program stored in the memory 320 .

According to an embodiment, the user data analysis module 610 may determine the user's user preference information by using information (eg, user data 611 ) stored in the memory 320 . According to various embodiments, the user data 611 may include information about a user of the electronic device 300 . For example, the user data 611 may include voiceprint data of each of at least one user of the electronic device 300 . In addition, the user data 611 may store information on the speech command utterance and/or content reproduction history information of each of at least one user. According to various embodiments, the content reproduction history may include a reproduction history of various data. For example, the content reproduction history may include content reproduction histories using at least one external device 530 . According to an embodiment, when the content reproduction operation is performed through the at least one external device 530 , the processor 310 may store information on the content reproduction history in the memory 320 . According to an embodiment, when the electronic device 300 receives an input of user preference information from a user, the user data 611 may include preference information corresponding to each of at least one user. According to various embodiments, the user data 611 is not limited to the example described in the present disclosure, and may include various information based on determining user preference information.

According to an embodiment, when a general voice command is received from the user through the microphone 330 , the processor 310 may determine the user's user preference information through the user data analysis module 610 . For example, the user analysis module 610 may identify voiceprint data corresponding to the general voice command from among voiceprint data of each of at least one user stored in the user data 611 . The user analysis module 610 may determine the user based on the identified voiceprint data. When there is preference information input from the user, the user analysis module 610 may determine user preference information based on the preference information. Alternatively, the user analysis module 610 may determine user preference information based on the voice command utterance and/or content reproduction history information corresponding to the user from among the voice command utterance and/or content reproduction history information corresponding to at least one user. have. According to an embodiment, the voice command utterance information may include information about a control operation used by the user or content uttered by the user. In an embodiment, the content reproduction history information may include information on content reproduced through at least one external device 530 . For example, information on a TV program reproduced through the first device 531 or a sound source reproduced through the third device 533 may be included. According to one embodiment, the user analysis module 610 is, when there is no preference information input from the user, voice command utterance and/or content reproduction history information in the user data 611, the age of the user analyzed through the voiceprint data based on the user preference information may be determined. For example, the user analysis module 610 may receive user preference information information for each age group through the server, and determine user preference information based on the age of the user.

According to an embodiment, the personalized voice command analysis module 620 may determine a personalized voice command corresponding to the user's general voice command based on the user preference information determined through the user data analysis module 610 . According to an embodiment, the memory 320 may store the personalized instruction set 621 . According to an embodiment, the personalized command set 621 may determine at least one theme and include at least one personalized voice command corresponding to each of the at least one theme. For example, the personalized command set 621 may include a list corresponding to each of at least one theme, and the list may include personalized voice commands corresponding to various general voice commands. According to an embodiment, the personalized voice command analysis module 620 may receive a user input for setting a personalized voice command corresponding to a general voice command. For example, the electronic device 300 may set a personalized voice command based on a user's input. According to an embodiment, the personalized voice command analysis module 620 may store the personalized voice command input by the user in association with the general voice command in the personalized command set 621 . Accordingly, when the general voice command is received from the user, the personalized voice command analysis module 620 may determine the personalized voice command stored in association with the general voice command without determining user preference information.

According to an embodiment, when a general voice command is received from the user through the microphone 330 , the personalized voice command analysis module 620 may determine the personalized voice command as needed. For example, as a result of identifying the user's voiceprint data, if the user uses the personalized voice command service, the personalized voice command analysis module 620 may determine the personalized voice command. As another example, when it is determined that the user is an age group of a teenager or younger as a result of identifying the user's voiceprint data, the personalized voice command analysis module 620 may determine the personalized voice command.

According to an embodiment, the personalized voice command analysis module 620 may determine the personalized voice command based on user preference information of the user determined through the user data analysis module 610 . For example, the personalized voice command analysis module 620 includes a theme corresponding to user preference information of the user among at least one theme included in the personalized command set 621, and personalization corresponding to a general voice command received from the user. When a voice command is included, the personalized voice command may be determined. According to an embodiment, the determined personalized voice command may be converted into an audio signal and output through the speaker 340 through the on-device natural language understanding platform described with reference to FIG. 4 .

According to an embodiment, the additional control operation analysis module 630 generates a plan based on a general voice command received from the user, and the additional control operation to be performed when the electronic device 300 performs an operation included in the plan. can be decided According to an embodiment, the additional control operation analysis module 630 may use the additional control operation set 631 stored in the memory 320 when determining the additional control operation. According to an embodiment, the additional control operation set 631 may include at least one additional control operation corresponding to at least one control operation classified according to each of the at least one theme included in the personalized instruction set 621 . have. For example, the additional control operation set 621 may include a list corresponding to each of at least one theme, and each list may include additional control operations corresponding to a control operation to be performed by the personalized voice command.

According to an embodiment, when a general voice command is received from the user through the microphone 330 and a corresponding plan is generated, the additional control operation analysis module 630 is an additional control operation to be performed together with the operation included in the plan. can be determined as needed. For example, the processor 310 may receive status information of the external device from each of the at least one external device 530 in order to determine the additional control operation. According to an embodiment, as a result of determining the state information of the external device, the additional control operation analysis module 630 may not determine the additional control operation when the user is playing content through at least one external device 530 . . Alternatively, the additional control operation analysis module 630 may not determine the additional control operation when an input for limiting execution of the additional control operation is received from the user of the electronic device 300 .

According to an embodiment, the additional control operation analysis module 630 may determine the additional control operation based on at least one of a control operation determined based on a general voice command and/or user preference information or state information associated with the user. . For example, if the general voice command is “Turn on the light on the desk lighting device,” the additional control operation analysis module 630 adds a control operation for adjusting the brightness of the lighting device on the desk to a brightness that can increase concentration. It can be determined by the control action. The electronic device 300 may generate a plan including an operation of turning on a light on the desk lighting device based on the general voice command, and perform an operation by adding an additional control operation while performing an operation corresponding to the plan. . For example, the electronic device 300 may turn on the desk lighting device by setting the brightness to a level sufficient to increase concentration. According to various embodiments, the state information associated with the user may include various types of information. For example, the type of control operation corresponding to the general voice command, the type of device to perform the operation based on the control operation, the time at which the general voice command is received, the location at which the voice command is received, or at least one external device 530 . Status information of each external device may be included.

According to an embodiment, the electronic device 300 may perform the operation by adding the additional control operation determined through the additional control operation analysis module 630 while performing the control operation corresponding to the general voice command.

7 is a flowchart 700 for explaining output of a personalized voice command signal and performing an operation of an electronic device according to an exemplary embodiment.

Referring to FIG. 7 , in operation 701 , the electronic device 300 may receive a general voice command from the user. According to an embodiment, the electronic device 300 may change a general voice command received through the on-device natural language understanding platform or the intelligent server 200 into text data. According to an embodiment, the electronic device 300 may identify voiceprint data in response to receiving user voice data including a general voice command. For example, the electronic device 300 may identify voiceprint data corresponding to the user's voice data from among at least one voiceprint data corresponding to at least one user stored in the memory 320 .

According to an embodiment, in operation 703 , the electronic device 300 may determine user preference information and a control operation of the user corresponding to the general voice command. According to an embodiment, the electronic device 300 may identify the user's voiceprint data through the user data analysis module 610 and determine user preference information. The user preference information may include, for example, a type of TV program that the user enjoys, and a type of music that the user enjoys listening to. Also, when the voiceprint data is not identified and the general speech processing is set, the electronic device 300 may provide a general voice service. For convenience of explanation, the user preference information is exemplified, but the user preference information may include various other contents and is not limited to the present disclosure. An operation of determining user preference information according to various embodiments will be described below in detail with reference to FIG. 8 .

According to an embodiment, the control operation may be determined based on a general voice command. For example, the electronic device 300 may change a general voice command into text data and generate at least one plan for performing a task corresponding to the command based on the text data. According to an embodiment, the plan may be generated by an artificial intelligence system. The electronic device 300 may perform an operation based on the generated plan. According to an embodiment, when performing a control operation based on a general voice command, the electronic device 300 may add an additional control operation to perform the control operation. Alternatively, the electronic device 300 may include the additional control operation in the plan. According to various embodiments, the contents of the electronic device 300 performing an operation based on a general voice command will be described below in detail with reference to FIG. 10 . According to an embodiment, in operation 705 , the electronic device 300 may determine a personalized voice command associated with a control operation. For example, the electronic device 300 may determine a personalized voice command and determine whether to output an audio signal based on the determination result. For example, the electronic device 300 may determine a personalized voice command corresponding to a general voice command through the personalized voice command analysis module 620 . When there is no personalized voice command corresponding to the general voice command, or when a general voice command is received based on a user rather than a target for outputting the personalized voice command, the electronic device 300 may not output the personalized voice command. An operation in which the electronic device 300 determines a personalized voice command and an operation in which an audio signal is output will be described in detail below with reference to FIG. 9 .

According to an embodiment, in operation 707 , the electronic device 300 transmits an audio signal corresponding to the personalized voice command to be output by at least one external device through a communication circuit or a speaker 340 included in the electronic device 300 . ) can be printed.

8 is a flowchart 703 for explaining an operation of determining user preference information of an electronic device according to an exemplary embodiment.

Referring to FIG. 8 , in operation 801 , the electronic device 300 may extract voiceprint data based on user voice data received from the user. According to an embodiment, the electronic device 300 may extract the user's voiceprint data by utilizing a module related to voice recognition. For example, the user's glottal data may represent an electrical signal generated based on the user's audio signal.

According to an embodiment, in operation 803 , the electronic device 300 may determine whether voiceprint data corresponding to the received user voice data exists among at least one voiceprint recognition data stored in the memory 320 . For example, the electronic device 300 may determine whether the extracted voiceprint data exists in the voiceprint recognition data corresponding to each of at least one user of the electronic device 300 stored in the memory 320 . According to an embodiment, when the extracted voiceprint data exists among voiceprint recognition data corresponding to each of at least one user, the electronic device 300 may determine that the user is an existing user. Alternatively, the electronic device 300 may determine that the regular user is an authenticated user. For another example, when there is no voiceprint data extracted from voiceprint recognition data corresponding to each of at least one user stored in the memory 320 , the electronic device 300 may determine the user as a new user. Accordingly, the voiceprint data corresponding to the new user may be stored in the memory 320 . Also, the electronic device 300 may determine that the new user is an unauthenticated user, and limit some of the operations based on the general voice command of the new user.

According to an embodiment, when the extracted voiceprint data does not exist among the voiceprint recognition data corresponding to each of at least one user, in operation 805 , the electronic device 300 processes the user's voice data by a general speech processing method. can do. That is, when the user's voiceprint data does not exist, the electronic device 300 may provide a general voice service. For example, when the voiceprint data does not exist, the electronic device 300 may perform an operation based on a general voice command included in the received voice data. That is, the electronic device 300 may determine a control operation based on a general voice command without determining user preference information.

According to one embodiment, according to an embodiment, when there is voiceprint data extracted from voiceprint recognition data corresponding to each of at least one user, in operation 807 , the electronic device 300 determines based on the user's input. It may be determined whether there is user preference information. According to an embodiment, when the extracted voiceprint data exists, user data 611 related to a user corresponding to the extracted voiceprint data may be included in the memory 320 . Accordingly, the processor 310 of the electronic device 300 may determine user preference information using the user data 611 . In an embodiment, when there is user preference information determined based on a user input, the electronic device 300 may determine user preference information based on the user input in operation 811 .

According to an embodiment, when there is no user preference information determined based on a user input, in operation 809 , the electronic device 300 may determine user preference information based on a history of uttering/reproducing a user's voice command. For example, when the user executes a plurality of utterances of a voice command related to a specific content or, as a result of analyzing a playback history through at least one external device 530, the electronic device 300 performs a plurality of playback of specific content, The specific content may be determined as user preference information.

According to an embodiment, the electronic device 300 may determine the user preference information based on at least one of the user's voice command history information, the user's age information, or the preference information input from the user.

According to various embodiments, the operation of the electronic device 300 determining user preference information may further include various operations. Also, in the operation of the electronic device 300 determining the user preference information, at least some operations of operations 801 to 809 may be omitted.

9 is a flowchart 705 for explaining an operation of determining a personalized voice command of an electronic device according to an embodiment.

Referring to FIG. 9 , in operation 901 , the electronic device 300 may determine whether there is a personalized instruction set based on user preference information. For example, the electronic device 300 may determine whether a personalized instruction set based on the user preference information determined in operation 703 is stored in the memory 320 . According to an embodiment, if there is no personalized instruction set based on user preference information in operation 901 , the electronic device 300 may perform operation 707 without performing another operation in operation 705 .

According to an embodiment, in operation 903 , the electronic device 300 may determine whether there is a personalized voice command corresponding to a general voice command in the personalized command set. For example, even if the personalized command set corresponding to the user preference information is included in the memo 320, the personalized command set may not include the personalized voice command corresponding to the general voice command received from the user. When the personalized voice command does not exist, the electronic device 300 may perform operation 707 without performing another operation in operation 705 .

According to an embodiment, in operation 905 , when there is a personalized voice command, the electronic device 300 transmits an audio signal corresponding to the personalized voice command to be output by at least one external device through the communication circuit 350, or An audio signal corresponding to the personalized voice command may be output through the speaker 340 included in the electronic device 300 . The electronic device 300 may extract text-to-speech data corresponding to the personalized voice command through the on-device natural language understanding platform included in the processor 310 and output an audio signal through the speaker 340 based thereon. For example, in response to a user's general voice command of "play a bright song" being received, the electronic device 300 may "play a bright song" with the same voice as that of a specific animation character according to user preference information of the user. It can output an audio signal of "You just have to memorize AAA". That is, the electronic device 300 may output an audio signal corresponding to a personalized voice command (eg, AAA) corresponding to a general voice command (eg, play a bright song) through the speaker 340 .

10 is a flowchart 707 for explaining an operation of determining an additional control operation of an electronic device according to an exemplary embodiment.

Referring to FIG. 10 , in operation 1001 , the electronic device 300 may determine whether there is an additional control operation corresponding to a general voice command in the personalized command set 621 . According to an embodiment, in response to receiving the general voice command received from the user, the electronic device 300 determines whether there is a personalized command set corresponding to the determined user preference information, and the personalized command set corresponds to the general voice command. It may be determined whether an additional control operation is included. For example, the personalized command set may include at least one personalized voice command corresponding to the general voice command and an additional control operation corresponding to the general voice command according to each of the at least one theme.

According to an embodiment, when there is no additional control operation, the electronic device 300 may perform an operation corresponding to a general voice command in operation 1007 . According to an embodiment, when there is an additional control operation, the electronic device 300 may determine the additional control operation based on at least one of a general voice command or user-related state information in operation 1003 . According to an embodiment, the additional control operation corresponding to the general voice command may include at least one effect. For example, when a control operation is determined according to a general voice command and an operation based on the control operation is performed, an effect of outputting additional background music and/or an effect of adjusting the color of the output screen or light may be included. have. According to an embodiment, the additional control operation may be determined according to a type of device to perform an operation based on the control operation. For example, even if the additional control operation includes an effect of adjusting the color of the output screen, the control operation may be omitted if the device to perform the operation based on the control operation does not include the screen output device. According to an embodiment, the additional control operation may be determined based on state information associated with the user. According to an embodiment, the state information associated with the user may include various context information. For example, the state information associated with the user may include information on the location, time, day, and date at which the electronic device 300 receives a general voice command, and information on a device to perform an operation based on a control operation. For example, when the time at which the electronic device 300 receives the general voice command is bedtime and the user's general voice command is "Turn off the light", the additional control operation is performed based on the time at which the general voice command is received, It may include an effect of outputting a good song to listen to at bedtime.

According to an embodiment, the electronic device 300 may perform a control operation and an additional control operation in operation 1005 . For example, the electronic device 300 may perform a control operation corresponding to a general voice command, including an additional control operation. In an embodiment, the electronic device 300 may provide user convenience when performing an operation corresponding to a general voice command by using at least one effect. According to an embodiment, the processor 310 may add an additional control operation when generating a plan corresponding to a general voice command. According to an embodiment, when performing an operation based on a control operation according to a general voice command using an external device, the electronic device 300 may transmit a control signal generated based on the control operation and the additional control operation to the external device. can The external device according to an embodiment may receive a control signal and perform an operation based on a control operation and an additional control operation based on the control signal. For example, if the user's general voice command is "Turn off the lights" and the additional control action is the effect of outputting a good song to listen to at bedtime, an external device (eg, lighting device) turns off the lights and a good song to listen to at bedtime can be printed out.

11 is a flowchart 1100 for explaining an operation of an electronic device according to an embodiment.

Referring to FIG. 11 , in operation 1101 , the electronic device 300 may receive voice data including a general voice command from the user of the electronic device 300 . According to an embodiment, the electronic device 300 may identify the voiceprint data corresponding to the user in operation 1103 by using the voiceprint data corresponding to each of at least one user stored in the memory 320 .

According to an embodiment, in operation 1105 , the electronic device 300 may determine user preference information of the user based on the identified user's voiceprint data. The operation of determining the user preference information according to an embodiment may include an operation similar to the operation included in the flowchart 703 for describing the operation of determining the user preference information of the electronic device described with reference to FIG. 8 .

According to an embodiment, in operation 1107 , the electronic device 300 may determine a personalized voice command and a control operation based on user preference information and a general voice command. According to an embodiment, the electronic device 300 may determine the personalized voice command using the personalized voice command analysis module 620 . According to an embodiment, the control operation may include operations included in a plan generated based on a general voice command.

According to an embodiment, in operation 1109 , the electronic device 300 may determine an additional control operation based on at least one of a control operation, user preference information, and device information of an external device that is a control target of the control operation. Also, in operation 1111 , the electronic device 300 may transmit an audio signal corresponding to the personalized voice command to be output by at least one external device or may output the audio signal through the speaker 340 . According to various embodiments, when there are no additional control operations and personalized voice commands, operations 1109 and 1111 may be omitted. According to an embodiment, operation 1109 may include operations similar to operations included in the flowchart 707 for describing an operation of determining an additional control operation.

According to an embodiment, the electronic device 300 may perform a control operation and an additional control operation in operation 1113 . For example, the electronic device 300 may perform an operation corresponding to a general voice command including an additional control operation. In an embodiment, when the additional control operation is not determined, the electronic device 300 may perform only an operation corresponding to a general voice command. For example, when performing an operation corresponding to a voice command including an additional control operation is restricted based on a user input, the electronic device 300 performs an operation corresponding to the voice command except for the additional control operation. can do. As another example, the electronic device 300 performs additional control when there is an external device that is playing content at the time of inputting a general voice command, based on the state information of the external device received through the at least one external device 530 . An operation corresponding to the voice command may be performed except for the operation.

12 is a diagram illustrating an electronic device displaying a screen for processing a voice input received through an intelligent app, according to an embodiment.

12 is a diagram illustrating a screen on which an electronic device processes a voice input received through an intelligent app according to various embodiments of the present disclosure;

The electronic device 300 (eg, the user terminal 100 of FIG. 1 ) may execute an intelligent app to process a user input through the intelligent server 200 .

According to an embodiment, on screen 1210 , when the electronic device 300 recognizes a specified voice input (eg, wake up!) or receives an input through a hardware key (eg, a dedicated hardware key), the electronic device 300 processes the voice input. You can run intelligent apps for The electronic device 300 may, for example, execute the intelligent app while the schedule app is running. According to an embodiment, the electronic device 300 may include a display (not shown). The electronic device 300 may display an object (eg, an icon) 1211 corresponding to an intelligent app on a display (eg, the display 140 of FIG. 1 ). According to an embodiment, the electronic device 300 may receive a voice input by a user's utterance. For example, the electronic device 300 may receive a voice input saying "Tell me about this week's schedule!" According to an embodiment, the electronic device 300 may display a user interface (UI) 1212 (eg, an input window) of an intelligent app on which text data of the received voice input is displayed on the display.

According to an embodiment, on the screen 1220, the electronic device 300 may display a result corresponding to the received voice input on the display. For example, the electronic device 300 may receive a plan corresponding to the received user input, and display 'this week's schedule' on the display according to the plan.

13A is a diagram illustrating an operation of controlling an external device through a voice command, according to an embodiment.

Referring to FIG. 13A , the electronic device 1320 (eg, the user terminal 100 of FIG. 1 ) may receive a first general voice command 1311 from the user 1310 . For example, the electronic device 1320 may receive the first general voice command 1311 "Turn on the light" from the user 1310 .

According to an embodiment, the electronic device 1320 determines user preference information of the user 1310 based on the reception of the first general voice command 1311 , and includes the user preference information and the first general voice command 1311 . Based on , the first personalized voice command 1321 may be determined. According to an embodiment, the electronic device 1320 may output the first personalized voice command 1321 as an audio signal. For example, the electronic device 1320 determines the first personalized voice command 1321 corresponding to the first general voice command 1311, and an audio signal corresponding to "Hello! You can turn on the light when you shout ABCD" can be printed out. That is, by outputting the first personalized voice command 1321 (eg, ABCD) as an audio signal, the electronic device 1320 may increase the usability of the voice service of the user 1310 .

According to an embodiment, the electronic device 1320 may control the operation of the first external device 1331 based on the first general voice command 1311 and the first personalized voice command 1321 . For example, the electronic device 1320 generates a control signal (eg, a signal to turn on a light) based on the first general voice command 1311 and the first personalized voice command 1321 , and the generated control signal may be transmitted to the first external device 1331 .

According to another embodiment, the electronic device 1320 may receive the first personalized voice command 1321 from the user 1310 . In response to receiving the first personalized voice command 1321 , the electronic device 1320 may perform an operation corresponding to the first personalized voice command 1321 . For example, when the electronic device 1320 receives the first personalized voice command 1321 (eg, “ABCD”) from the user 1310 , a control signal corresponding thereto (eg, a signal to turn on a light) can create The electronic device 1320 may transmit a control signal to the first external device 1331 . According to an embodiment, the first general voice command 1311 may perform the same operation that can be replaced with the first personalized voice command 1321 . That is, the electronic device 1320 may perform the same operation based on the reception of the first general voice command 1311 or the first personalized voice command 1321 .

According to an embodiment, when performing an operation based on the first general voice command 1311 , the electronic device 1320 may perform the operation by adding an additional control operation. According to an embodiment, the additional control operation may be determined through the additional control operation analysis module 630 . For example, the electronic device 1320 may determine the additional control operation based on the first general voice command 1311 (eg, “turn on the light”) and user preference information. The electronic device 1320 may perform an operation based on an additional control operation including at least one effect based on user preference information while performing an operation to turn on the light. For example, the electronic device 1320 transmits a control signal to the first external device 1331, and the first external device 1331 turns on a light and outputs background music based on user preference information. action can be performed.

According to an embodiment, the electronic device 1320 may determine an additional control operation based on state information associated with the user 1310 . For example, the state information associated with the user 1310 may include the user's first general voice command 1311 utterance time. According to an embodiment, when the time at which the electronic device 1320 receives the first general voice command 1311 is in the morning, the electronic device 1320 additionally controls the effect of outputting a light having an appropriate brightness for waking up. It can be decided by action.

According to an embodiment, the electronic device 1320 may determine a subsequent command based on the first general voice command 1311 . According to various embodiments, the subsequent instruction may be determined in various ways. For example, the electronic device 1320 may determine a command corresponding to a control operation matching a control operation corresponding to the first general voice command as a subsequent command. Alternatively, the electronic device 1320 may determine a command that is highly likely to be executed after the first general voice command 1311 as a subsequent command. The electronic device 1320 may determine the subsequent command based on the voice command utterance and/or content reproduction history information stored in the memory 320 . According to another embodiment, the electronic device 1320 may determine a command that can be used by applying the first general voice command 1311 as a subsequent command. For example, when the first general voice command 1311 is “Turn on the light”, the electronic device 1320 may determine to change the lighting color as a subsequent command.

According to an embodiment, the electronic device 1320 may determine a subsequent personalized command corresponding to the subsequent command. For example, the electronic device 1320 may determine a subsequent personalization command based on user preference information. In an embodiment, the electronic device 1320 may output an audio signal corresponding to a subsequent personalization command. Even before receiving the general voice command from the user, the electronic device 1320 may output the subsequent personalized command as an audio signal based on the existence of the subsequent personalized command.

According to an embodiment, the electronic device 1320 may receive a personalized voice command setting input corresponding to a general voice command from the user 1310 . For example, the user 1310 may set the personalized voice command corresponding to the first general voice command 1311 as the first personalized voice command 1321 . When receiving the input for setting the first personalized voice command 1321 , the electronic device 1320 stores the first general voice command 1311 in association with the first personalized voice command 1321 in the memory 320 . can Accordingly, the electronic device 1320 may perform an operation corresponding to the first general voice command 1311 through the first personalized voice command 1321 received from the user 1310 .

13B is a diagram illustrating an operation of outputting a personalized voice command based on content being reproduced, according to an embodiment.

Referring to FIG. 13B , based on the content being reproduced through the second external device 1332 , the electronic device 1320 may output the second personalized voice command 1322 as an audio signal. According to an embodiment, the user 1310 may play content through the second external device 1332 . For example, the user 1310 may watch a broadcast program through the second external device 1332 .

According to an embodiment, the electronic device 1320 may determine that there is a second external device 1332 playing content among at least one external device 530 . The electronic device 1320 may determine at least one personalized voice command (eg, the second personalized voice command 1322) based on the content being reproduced, and output the at least one personalized voice command as an audio signal. . For example, when a broadcast program is being played on the second external device 1332 , the electronic device 1320 may generate a second personalized voice command based on information (eg, cast lines, etc.) included in the currently playing broadcast program. can decide

According to an embodiment, when outputting the personalized voice command, the electronic device 1320 may determine whether there is an external device playing content among at least one external device 530 based on the state information of the external device. The electronic device 1320 may control the size of the audio signal output corresponding to the personalized voice command based on the determination result. For example, when the electronic device 1320 outputs the second personalized voice command 1322 , the user 1310 is watching the content through the second external device 1332 , so as not to interfere with the content viewing. The second personalized voice command 1322 may be output with a minimum size. According to various embodiments of the present disclosure, when there is content being played from at least one external device 530 , etc., which may disturb the user 1310 , the electronic device 1320 controls the output size of the personalized voice command and the additional control operation. can be controlled

13C is a diagram illustrating an operation of outputting a personalized command list through an external device, according to an embodiment.

Referring to FIG. 13C , the electronic device 1320 according to an embodiment transmits at least one personalized command list corresponding to a personalized voice command to an external device including a display, and the external device sends the at least one personalized command list A control signal may be transmitted to output .

According to an embodiment, the electronic device 1320 may receive the third general voice command 1312 from the user 1310 . For convenience of description, although different general voice commands are used throughout FIGS. 13A to 13D , the same general voice command may perform an operation in the same manner.

According to an embodiment, in response to receiving the third general voice command 1312 , the electronic device 1320 transmits at least one corresponding to the third general voice command 1312 to the second external device 1332 including a display. A control signal for outputting one instruction set may be transmitted.

According to an embodiment, the personalized voice command stored in the memory 320 corresponding to the third general voice command 1312 may vary. According to an embodiment, the second external device 1332 may output the third personalized command list 1332_1 based on the control signal. Accordingly, the user 1310 may obtain information on the personalized voice command through an external device as well as an audio signal through the electronic device 1320 .

13D is a diagram illustrating an operation of controlling the use of an external device based on a voice command, according to an embodiment.

Referring to FIG. 13D , the electronic device 1320 according to an embodiment may determine whether a control operation corresponding to a plan generated by a general voice command includes a restricted operation. The electronic device 1320 according to an embodiment may output a notification regarding the limitation of the operation corresponding to the general voice command in response to that the operation for which the control operation is restricted is included.

According to an embodiment, the electronic device 1320 may receive the fourth general voice command 1313 from the user 1310 . According to an embodiment, the electronic device 1320 may limit control operations of some of the control operations that the user 1310 can use through the voice command service. For example, when the age of the user 1310 is low, the electronic device 1320 may limit some functions of the voice command service of the user 1310 .

According to an embodiment, the electronic device 1320 may receive a fourth general voice command 1313 including a limited operation from the user 1310 . In response to receiving the fourth general voice command 1313 , the electronic device 1320 may determine whether a restricted operation is included in the operation corresponding to the fourth general voice command 1313 . According to an embodiment, the electronic device 1320 may limit the operation of the third external device 1333 among at least one external device 530 . The electronic device 1320 may determine whether an operation using the third external device 1333 is included in the operation corresponding to the fourth general voice command 1313 .

In an embodiment, when an operation using the third external device 1333 is included, the electronic device 1320 may output a usage restriction notification 1323 . For example, in response to receiving the fourth general voice command 1313 including the operation of using the third external device 1333 (eg, washing machine) from the user 1310 , the electronic device 1320 may say “User, It is possible to output a usage restriction notification 1323 including "cannot perform the washing machine spin-drying operation corresponding to the voice command."

According to an embodiment, in response to receiving the fourth general voice command 1313 from the user 1310 , the electronic device 1320 performs an operation using the fourth general voice command 1313 . A control signal that provides a notification to each of the available external devices may be transmitted. For example, in response to receiving the fourth general voice command 1313 from the user 1310 , the electronic device 1320 selects the user 1310 from among at least one external device 530 connected to the electronic device 1320 . A notification for the available external device 530 may be provided. For example, the electronic device 1320 may transmit a control signal to an external device available to the user 1310 among at least one external device 530 to provide a notification through sound.

14 is a block diagram of an electronic device in a network environment according to an embodiment.

Referring to FIG. 14 , in a network environment 1400 , the electronic device 1401 communicates with the electronic device 1402 through a first network 1498 (eg, a short-range wireless communication network) or a second network 1499 . It may communicate with at least one of the electronic device 1404 and the server 1408 through (eg, a long-distance wireless communication network). According to an embodiment, the electronic device 1401 may communicate with the electronic device 1404 through the server 1408 . According to an embodiment, the electronic device 1401 includes a processor 1420 , a memory 1430 , an input module 1450 , a sound output module 1455 , a display module 1460 , an audio module 1470 , and a sensor module ( 1476), interface 1477, connection terminal 1478, haptic module 1479, camera module 1480, power management module 1488, battery 1489, communication module 1490, subscriber identification module 1496 , or an antenna module 1497 . In some embodiments, at least one of these components (eg, the connection terminal 1478 ) may be omitted or one or more other components may be added to the electronic device 1401 . In some embodiments, some of these components (eg, sensor module 1476 , camera module 1480 , or antenna module 1497 ) are integrated into one component (eg, display module 1460 ). can be

The processor 1420, for example, executes software (eg, a program 1440) to execute at least one other component (eg, hardware or software component) of the electronic device 1401 connected to the processor 1420. It can control and perform various data processing or operations. According to one embodiment, as at least part of data processing or computation, the processor 1420 converts commands or data received from other components (eg, the sensor module 1476 or the communication module 1490 ) to the volatile memory 1432 . , process commands or data stored in the volatile memory 1432 , and store the resulting data in the non-volatile memory 1434 . According to an embodiment, the processor 1420 is the main processor 1421 (eg, a central processing unit or an application processor) or a secondary processor 1423 (eg, a graphics processing unit, a neural network processing unit) a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor). For example, when the electronic device 1401 includes a main processor 1421 and a sub-processor 1423 , the sub-processor 1423 uses less power than the main processor 1421 or is set to be specialized for a specified function. can The coprocessor 1423 may be implemented separately from or as part of the main processor 1421 .

The coprocessor 1423 may be, for example, on behalf of the main processor 1421 while the main processor 1421 is in an inactive (eg, sleep) state, or the main processor 1421 is active (eg, executing an application). ), together with the main processor 1421, at least one of the components of the electronic device 1401 (eg, the display module 1460, the sensor module 1476, or the communication module 1490) It is possible to control at least some of the related functions or states. According to one embodiment, the coprocessor 1423 (eg, image signal processor or communication processor) may be implemented as part of another functionally related component (eg, camera module 1480 or communication module 1490). have. According to an embodiment, the auxiliary processor 1423 (eg, a neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. Artificial intelligence models can be created through machine learning. Such learning may be performed, for example, in the electronic device 1401 itself on which the artificial intelligence model is performed, or may be performed through a separate server (eg, the server 1408). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but in the above example not limited The artificial intelligence model may include a plurality of artificial neural network layers. Artificial neural networks include deep neural networks (DNNs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), restricted boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), It may be one of deep Q-networks or a combination of two or more of the above, but is not limited to the above example. The artificial intelligence model may include, in addition to, or alternatively, a software structure in addition to the hardware structure.

The memory 1430 may store various data used by at least one component of the electronic device 1401 (eg, the processor 1420 or the sensor module 1476 ). The data may include, for example, input data or output data for software (eg, a program 1440 ) and commands related thereto. The memory 1430 may include a volatile memory 1432 or a non-volatile memory 1434 .

The program 1440 may be stored as software in the memory 1430 , and may include, for example, an operating system 1442 , middleware 1444 , or an application 1446 .

The input module 1450 may receive a command or data to be used by a component (eg, the processor 1420 ) of the electronic device 1401 from the outside (eg, a user) of the electronic device 1401 . The input module 1450 may include, for example, a microphone, a mouse, a keyboard, a key (eg, a button), or a digital pen (eg, a stylus pen).

The sound output module 1455 may output a sound signal to the outside of the electronic device 1401 . The sound output module 1455 may include, for example, a speaker or a receiver. The speaker can be used for general purposes such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from or as part of the speaker.

The display module 1460 may visually provide information to the outside (eg, a user) of the electronic device 1401 . The display module 1460 may include, for example, a display, a hologram device, or a projector and a control circuit for controlling the corresponding device. According to an embodiment, the display module 1460 may include a touch sensor configured to sense a touch or a pressure sensor configured to measure the intensity of a force generated by the touch.

The audio module 1470 may convert a sound into an electric signal or, conversely, convert an electric signal into a sound. According to an embodiment, the audio module 1470 acquires a sound through the input module 1450 or an external electronic device (eg, a sound output module 1455 ) directly or wirelessly connected to the electronic device 1401 . The electronic device 1402) (eg, a speaker or headphones) may output a sound.

The sensor module 1476 detects an operating state (eg, power or temperature) of the electronic device 1401 or an external environmental state (eg, user state), and generates an electrical signal or data value corresponding to the sensed state. can do. According to one embodiment, the sensor module 1476 may include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, It may include a temperature sensor, a humidity sensor, or an illuminance sensor.

The interface 1477 may support one or more designated protocols that may be used for the electronic device 1401 to directly or wirelessly connect with an external electronic device (eg, the electronic device 1402 ). According to an embodiment, the interface 1477 may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

The connection terminal 1478 may include a connector through which the electronic device 1401 can be physically connected to an external electronic device (eg, the electronic device 1402 ). According to an embodiment, the connection terminal 1478 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (eg, a headphone connector).

The haptic module 1479 may convert an electrical signal into a mechanical stimulus (eg, vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic sense. According to an embodiment, the haptic module 1479 may include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

The camera module 1480 may capture still images and moving images. According to one embodiment, the camera module 1480 may include one or more lenses, image sensors, image signal processors, or flashes.

The power management module 1488 may manage power supplied to the electronic device 1401 . According to an embodiment, the power management module 1488 may be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

The battery 1489 may supply power to at least one component of the electronic device 1401 . According to one embodiment, battery 1489 may include, for example, a non-rechargeable primary cell, a rechargeable secondary cell, or a fuel cell.

The communication module 1490 is a direct (eg, wired) communication channel or a wireless communication channel between the electronic device 1401 and an external electronic device (eg, the electronic device 1402, the electronic device 1404, or the server 1408). It can support establishment and communication performance through the established communication channel. The communication module 1490 may include one or more communication processors that operate independently of the processor 1420 (eg, an application processor) and support direct (eg, wired) communication or wireless communication. According to one embodiment, the communication module 1490 may include a wireless communication module 1492 (eg, a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 1494 (eg : It may include a local area network (LAN) communication module, or a power line communication module). A corresponding communication module among these communication modules is a first network 1498 (eg, a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network 1499 (eg, legacy). It may communicate with the external electronic device 1404 through a cellular network, a 5G network, a next-generation communication network, the Internet, or a telecommunication network such as a computer network (eg, LAN or WAN). These various types of communication modules may be integrated into one component (eg, a single chip) or may be implemented as a plurality of components (eg, multiple chips) separate from each other. The wireless communication module 1492 uses subscriber information (eg, International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module 1496 within a communication network, such as the first network 1498 or the second network 1499 . The electronic device 1401 may be identified or authenticated.

The wireless communication module 1492 may support a 5G network after a 4G network and a next-generation communication technology, for example, a new radio access technology (NR). NR access technology includes high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and access to multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency) -latency communications)). The wireless communication module 1492 may support a high frequency band (eg, mmWave band) to achieve a high data rate, for example. The wireless communication module 1492 uses various technologies for securing performance in a high-frequency band, for example, beamforming, massive multiple-input and multiple-output (MIMO), all-dimensional multiplexing. It may support technologies such as full dimensional MIMO (FD-MIMO), an array antenna, analog beam-forming, or a large scale antenna. The wireless communication module 1492 may support various requirements specified in the electronic device 1401 , an external electronic device (eg, the electronic device 1404 ), or a network system (eg, the second network 1499 ). According to an embodiment, the wireless communication module 1492 includes a peak data rate (eg, 20 Gbps or more) for realization of eMBB, loss coverage for realization of mMTC (eg, 164 dB or less), or U-plane latency (for URLLC realization) ( Example: Downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) can be supported.

The antenna module 1497 may transmit or receive a signal or power to the outside (eg, an external electronic device). According to an embodiment, the antenna module 1497 may include an antenna including a conductor formed on a substrate (eg, a PCB) or a radiator formed of a conductive pattern. According to an embodiment, the antenna module 1497 may include a plurality of antennas (eg, an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network such as the first network 1498 or the second network 1499 is connected from the plurality of antennas by, for example, the communication module 1490 . can be selected. A signal or power may be transmitted or received between the communication module 1490 and an external electronic device through the selected at least one antenna. According to some embodiments, other components (eg, a radio frequency integrated circuit (RFIC)) other than the radiator may be additionally formed as a part of the antenna module 1497 .

According to various embodiments, the antenna module 1497 may form a mmWave antenna module. According to one embodiment, the mmWave antenna module comprises a printed circuit board, an RFIC disposed on or adjacent to a first side (eg, bottom side) of the printed circuit board and capable of supporting a designated high frequency band (eg, mmWave band); and a plurality of antennas (eg, an array antenna) disposed on or adjacent to a second side (eg, top or side) of the printed circuit board and capable of transmitting or receiving signals of the designated high frequency band. can do.

At least some of the components are connected to each other through a communication method between peripheral devices (eg, a bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)) and a signal ( e.g. commands or data) can be exchanged with each other.

According to an embodiment, the command or data may be transmitted or received between the electronic device 1401 and the external electronic device 1404 through the server 1408 connected to the second network 1499 . Each of the external electronic devices 1402 and 1404 may be the same or a different type of the electronic device 1401 . According to an embodiment, all or a part of operations executed in the electronic device 1401 may be executed in one or more external electronic devices 1402 , 1404 , or 1408 . For example, when the electronic device 1401 needs to perform a function or service automatically or in response to a request from a user or other device, the electronic device 1401 may perform the function or service itself instead of executing the function or service itself. Alternatively or additionally, one or more external electronic devices may be requested to perform at least a part of the function or the service. One or more external electronic devices that have received the request may execute at least a part of the requested function or service, or an additional function or service related to the request, and transmit a result of the execution to the electronic device 1401 . The electronic device 1401 may process the result as it is or additionally and provide it as at least a part of a response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device 1401 may provide an ultra-low latency service using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device 1404 may include an Internet of things (IoT) device. The server 1408 may be an intelligent server using machine learning and/or neural networks. According to an embodiment, the external electronic device 1404 or the server 1408 may be included in the second network 1499 . The electronic device 1401 may be applied to an intelligent service (eg, smart home, smart city, smart car, or health care) based on 5G communication technology and IoT-related technology.

The electronic device according to various embodiments disclosed in this document may have various types of devices. The electronic device may include, for example, a portable communication device (eg, a smart phone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a home appliance device. The electronic device according to the embodiment of the present document is not limited to the above-described devices.

The various embodiments of this document and terms used therein are not intended to limit the technical features described in this document to specific embodiments, but it should be understood to include various modifications, equivalents, or substitutions of the embodiments. In connection with the description of the drawings, like reference numerals may be used for similar or related components. The singular form of the noun corresponding to the item may include one or more of the item, unless the relevant context clearly dictates otherwise. As used herein, "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C", and "A , B, or C," each of which may include any one of the items listed together in the corresponding one of the phrases, or all possible combinations thereof. Terms such as "first", "second", or "first" or "second" may simply be used to distinguish an element from other elements in question, and may refer elements to other aspects (e.g., importance or order) is not limited. It is said that one (eg, first) component is “coupled” or “connected” to another (eg, second) component, with or without the terms “functionally” or “communicatively”. When referenced, it means that one component can be connected to the other component directly (eg by wire), wirelessly, or through a third component.

The term “module” used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and is interchangeable with terms such as, for example, logic, logic block, component, or circuit. can be used as A module may be an integrally formed part or a minimum unit or a part of the part that performs one or more functions. For example, according to an embodiment, the module may be implemented in the form of an application-specific integrated circuit (ASIC).

Various embodiments of the present document include one or more instructions stored in a storage medium (eg, internal memory 1436 or external memory 1438) readable by a machine (eg, electronic device 1401). may be implemented as software (eg, a program 1440) including For example, a processor (eg, processor 1420 ) of a device (eg, electronic device 1401 ) may call at least one command among one or more commands stored from a storage medium and execute it. This makes it possible for the device to be operated to perform at least one function according to the called at least one command. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' only means that the storage medium is a tangible device and does not contain a signal (eg, electromagnetic wave), and this term is used in cases where data is semi-permanently stored in the storage medium and It does not distinguish between temporary storage cases.

According to one embodiment, the method according to various embodiments disclosed in this document may be provided in a computer program product (computer program product). Computer program products may be traded between sellers and buyers as commodities. The computer program product is distributed in the form of a machine-readable storage medium (eg compact disc read only memory (CD-ROM)), or via an application store (eg Play Store ^TM ) or on two user devices ( It can be distributed (eg downloaded or uploaded) directly, online between smartphones (eg: smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created in a machine-readable storage medium such as a memory of a server of a manufacturer, a server of an application store, or a relay server.

According to various embodiments, each component (eg, a module or a program) of the above-described components may include a singular or a plurality of entities, and some of the plurality of entities may be separately disposed in other components. have. According to various embodiments, one or more components or operations among the above-described corresponding components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (eg, a module or a program) may be integrated into one component. In this case, the integrated component may perform one or more functions of each component of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. . According to various embodiments, operations performed by a module, program, or other component are executed sequentially, in parallel, repeatedly, or heuristically, or one or more of the operations are executed in a different order, or omitted. , or one or more other operations may be added.

As described above, the electronic device (eg, the electronic device 300 of FIG. 3 ) according to an exemplary embodiment has a memory (eg, the memory of FIG. 3 ) configured to store at least one voiceprint data corresponding to at least one user. 320)), a communication circuit for transmitting a signal for controlling at least one external device (eg, the communication circuit 350 of FIG. 3 ) and at least one processor electrically connected to the memory and the communication circuit (eg, FIG. 3 ) 3, the processor 310), wherein the at least one processor identifies the voiceprint data corresponding to the user from among the at least one voiceprint data based on the received user voice data, identify an included general voice command, determine user preference information of the user based on the identified voiceprint data, and determine the at least one external device based on the general voice command identified in the received user voice data or Determine a control operation for determining an operation to be performed in the electronic device, determine a personalized voice command based on at least one of the control operation and the user preference information, and communicate an audio signal corresponding to the personalized voice command The at least one external device may transmit the output signal through a circuit or output the audio signal through a speaker included in the electronic device.

According to an embodiment, the memory stores the control operation in association with at least one of the personalized voice command or the user preference information, and the at least one processor responds to the determination of the control operation and the user preference information. Thus, the personalized voice command stored in association with at least one of the control operation and the user preference information may be searched.

In an embodiment, the at least one processor determines an additional control operation based on at least one of the control operation, the user preference information, or device information of a first external device that is a target of the control operation, and performs the personalization. The memory may be controlled to store the additional control operation in association with a voice command.

According to an embodiment, the at least one processor generates a control signal for controlling the first external device to perform the control operation and the additional control operation, and transmits the control signal to the first external device through the communication circuit. The control signal may be transmitted.

The at least one processor in an embodiment stores, in the memory, voice command history information corresponding to each of the at least one user, age information, or preference information input from each of the at least one user, in the memory, The user preference information may be determined based on at least one of voice command history information of the user, age information of the user, and preference information input from the user among the number of users.

According to an embodiment, the at least one processor may include at least one of the control operation, device information of the first external device, a time at which the user voice data is received, and location information of an electronic device when the user voice data is received. The additional control operation may be determined based on one.

According to an embodiment, the at least one processor may receive status information of the external device for each of the at least one external device using the communication circuit.

In an embodiment, the at least one processor determines whether there is an external device playing content among the at least one external device based on the state information of the external device, and outputs the audio signal based on the determination result You can control the size.

According to an embodiment, the at least one processor determines whether there is a device playing content among the at least one external device based on the state information of the external device, and provides a content voice command based on the content being played. is determined, and an audio signal corresponding to the content voice command is transmitted through the communication circuit so that the at least one external device is outputted or outputted through the speaker, and the content voice command is information included in the content being reproduced. can be determined based on

In an embodiment, the at least one processor determines a subsequent command to be used after the general voice command based on the general voice command, and determines a subsequent personalized command based on the subsequent command and the user preference information, , transmits a subsequent audio signal corresponding to the subsequent personalized command to be output by the at least one external device through the communication circuit or output through the speaker, wherein the subsequent command is uttered by the user after the general voice command It may contain commands to be

According to an embodiment, the at least one processor determines whether the control operation includes a restricted operation requiring an authentication operation for a user of the electronic device, and responds that the control operation includes a restricted operation as a result of the determination Thus, it is possible to control the speaker to output a notification about the limitation of the operation corresponding to the general voice command.

According to an embodiment, the at least one processor receives a user input for setting at least one personalized voice command corresponding to at least one general voice command stored in the memory, and executes the at least one personalized voice command. The memory may be controlled to store in association with the at least one general voice command.

In an embodiment, the at least one processor transmits at least one personalized command list corresponding to the general voice command to an external device including a display among the at least one external device using the communication circuit, and An external device including a display may transmit a list output signal to output the at least one personalized command list.

As described above, a memory set to store at least one voiceprint data corresponding to at least one user (eg, the memory 320 in FIG. 3 ) and a communication circuit for transmitting a signal for controlling at least one external device ( For example, a method of operating an electronic device (eg, the electronic device 300 of FIG. 3 ) including the communication circuit 350 of FIG. 3 ) includes an operation of receiving user voice data and the user from among the at least one voiceprint data. The operation of identifying voiceprint data corresponding to the operation of identifying a general voice command included in the received user voice data, determining the user preference information of the user based on the identified voiceprint data, and responding to the general voice command determining a control operation based on an operation to be performed in the at least one external device or the electronic device based on the operation, determining a personalized voice command based on at least one of the control operation and the user preference information, and the personalized voice The method may include transmitting an audio signal corresponding to a command to output the at least one external device through the communication circuit or outputting the audio signal through a speaker included in the electronic device.

According to an embodiment, in the method of operating the electronic device, in response to an operation of storing the control operation in association with at least one of the personalized voice command or the user preference information, and the determination of the control operation and the user preference information, , and searching for the stored personalized voice command in association with at least one of the control operation and the user preference information.

Determining an additional control operation based on at least one of the control operation, the user preference information, or device information of a first external device that is a target of the control operation, and the additional control to the personalized voice command The method further includes an operation of storing in association with an operation, and an operation of generating a control signal for controlling the first external device to perform the control operation and the additional control operation and transmitting the control signal to the first external device through the communication circuit. can do.

According to an embodiment of the present disclosure, the method of operating the electronic device includes: storing voice command history information corresponding to each of the at least one user, age information, or preference information input from each of the at least one user in the memory; The method may further include determining the user preference information based on at least one of voice command history information of the user among the at least one user, age information of the user, and preference information input from the user.

The method of operating the electronic device according to an embodiment includes an operation of receiving state information of an external device for each of the at least one external device using the communication circuit, and the at least one information on the state of the external device. The method may further include determining whether there is an external device playing content among external devices and controlling an output size of the audio signal based on a result of the determination.

The method of operating the electronic device according to an embodiment includes an operation of determining a subsequent command used after the general voice command based on the general voice command, and a subsequent personalized command based on the subsequent command and the user preference information. and transmitting a subsequent audio signal corresponding to the determining operation and the subsequent personalization command to be output by the at least one external device through the communication circuit or outputting it through the speaker, wherein the subsequent command is the general voice It may include a command uttered by the user after the command.

According to an embodiment, the method of operating the electronic device includes: receiving a user input for setting at least one personalized voice command corresponding to at least one general voice command stored in the memory; The method may further include storing a command in association with the at least one general voice command.

Claims (20)

In an electronic device,
a memory configured to store at least one voiceprint data corresponding to at least one user;
a communication circuit for transmitting a signal for controlling at least one external device; and
at least one processor electrically coupled to the memory and the communication circuit;
The at least one processor comprises:
identify voiceprint data corresponding to the user from among the at least one voiceprint data based on the received user voice data;
Identifies a general voice command included in the received user voice data,
determine user preference information of the user based on the identified voiceprint data;
determining a control operation for determining an operation to be performed in the at least one external device or the electronic device based on the general voice command identified in the received user voice data;
determine a personalized voice command based on at least one of the control operation and the user preference information;
An electronic device that transmits an audio signal corresponding to the personalized voice command to be output by the at least one external device through the communication circuit or outputs the audio signal through a speaker included in the electronic device. The method according to claim 1, The memory,
storing the control operation in association with at least one of the personalized voice command or the user preference information;
the at least one processor,
In response to determining the control operation and the user preference information, the electronic device retrieves the personalized voice command stored in association with at least one of the control operation and the user preference information. The method according to claim 1,
the at least one processor,
determining an additional control operation based on at least one of the control operation, the user preference information, or device information of a first external device that is a target of the control operation;
An electronic device controlling the memory to store the additional control operation in association with the personalized voice command. 4. The method according to claim 3,
The at least one processor generates a control signal for controlling the first external device to perform the control operation and the additional control operation,
The electronic device transmits the control signal to the first external device through the communication circuit. The method according to claim 1,
The at least one processor stores, in the memory, voice command history information corresponding to each of the at least one user, age information, or preference information input from each of the at least one user, in the memory, the user among the at least one user The electronic device determines the user preference information based on at least one of voice command history information of the user, age information of the user, and preference information input from the user. The method according to claim 3, wherein the at least one processor,
an electronic device that determines the additional control operation based on at least one of the control operation, device information of the first external device, a time at which the user voice data is received, and location information of the electronic device when the user voice data is received . The method according to claim 1, wherein the at least one processor,
An electronic device that receives status information of an external device for each of the at least one external device using the communication circuit. The method according to claim 7, wherein the at least one processor,
determining whether there is an external device playing content among the at least one external device based on the status information of the external device;
An electronic device that controls an output level of the audio signal based on a result of the determination. The method according to claim 7, wherein the at least one processor,
determining whether there is a device playing content among the at least one external device based on the status information of the external device;
Determine a content voice command based on the content being played,
Transmitting an audio signal corresponding to the content voice command to be output by the at least one external device through the communication circuit or outputting it through the speaker,
The content voice command is determined based on information included in the content being reproduced. The method according to claim 1, wherein the at least one processor,
determining a subsequent command to be used after use of the general voice command based on the general voice command;
determine a subsequent personalization command based on the subsequent command and the user preference information;
Transmitting a subsequent audio signal corresponding to the subsequent personalization command to be output by the at least one external device through the communication circuit or outputting it through the speaker;
The subsequent command includes a command uttered by the user after the general voice command. The method according to claim 1, wherein the at least one processor,
Determining whether the control operation includes a restricted operation requiring an authentication operation for a user of the electronic device;
As a result of the determination, in response to that the control operation includes the restricted operation, the electronic device controls the speaker to output a notification regarding the limitation of the operation corresponding to the general voice command. The method according to claim 1, wherein the at least one processor,
receiving a user input for setting at least one personalized voice command corresponding to at least one general voice command stored in the memory;
An electronic device controlling the memory to store the at least one personalized voice command in association with the at least one general voice command. The method according to claim 1, wherein the at least one processor,
using the communication circuit to transmit at least one personalized command list corresponding to the general voice command to an external device including a display among the at least one external device;
An electronic device that transmits a list output signal so that the external device including the display outputs the at least one personalized command list. A method of operating an electronic device comprising: a memory configured to store at least one voiceprint data corresponding to at least one user; and a communication circuit for transmitting a signal for controlling at least one external device, the method comprising:
receiving user voice data;
identifying voiceprint data corresponding to the user from among the at least one voiceprint data;
identifying a general voice command included in the received user voice data;
determining user preference information of the user based on the identified voiceprint data, and determining a control operation based on an operation to be performed by the at least one external device or the electronic device based on the general voice command;
determining a personalized voice command based on at least one of the control operation and the user preference information; and
and transmitting an audio signal corresponding to the personalized voice command through the communication circuit so that the at least one external device outputs the audio signal or outputting the audio signal through a speaker included in the electronic device. . 15. The method of claim 14,
storing the control operation in association with at least one of the personalized voice command and the user preference information; and
In response to the determination of the control operation and the user preference information, the operation method of the electronic device further comprising the operation of searching for the personalized voice command stored in association with at least one of the control operation and the user preference information. 15. The method of claim 14,
determining an additional control operation based on at least one of the control operation, the user preference information, or device information of a first external device that is a target of the control operation;
storing the additional control operation in association with the personalized voice command; and
and generating a control signal for controlling the first external device to perform the control operation and the additional control operation and transmitting the generated control signal to the first external device through the communication circuit. 15. The method of claim 14,
storing voice command history information, age information, or preference information input from each of the at least one user corresponding to each of the at least one user in the memory;
and determining the user preference information based on at least one of voice command history information of the user among the at least one user, age information of the user, and preference information input from the user. . 15. The method of claim 14,
receiving status information of an external device for each of the at least one external device using the communication circuit;
determining whether there is an external device playing content among the at least one external device based on the status information of the external device; and
and controlling an output level of the audio signal based on a result of the determination. 15. The method of claim 14,
determining a subsequent command to be used after use of the general voice command based on the general voice command;
determining a subsequent personalization command based on the subsequent command and the user preference information; and
Transmitting a subsequent audio signal corresponding to the subsequent personalization command to be output by the at least one external device through the communication circuit or outputting it through the speaker,
The subsequent command includes a command uttered by the user after the general voice command. 15. The method of claim 14,
receiving a user input for setting at least one personalized voice command corresponding to at least one general voice command stored in the memory;
and storing the at least one personalized voice command in association with the at least one general voice command.

Images