KR20210015234A - Electronic apparatus, and method of controlling to execute function according to voice command thereof - Google Patents

Abstract

An electronic device of the present disclosure comprises: a memory including a plurality of databases classified according to a predicate; a communication circuit receiving voice data corresponding to a voice command from an external device; and a processor operatively coupled with the memory and the communication circuit. The processor identifies a type of predicate included in the voice data, and based on information included in a database corresponding to the identified predicate, determines at least one device for performing the voice command a function to be performed by the at least one device according to the voice command, and controls the function to be performed according to the voice command in the at least one device.

Description

Electronic device, and a method of controlling to execute a function according to its voice command TECHNICAL FIELD [ELECTRONIC APPARATUS, AND METHOD OF CONTROLLING TO EXECUTE FUNCTION ACCORDING TO VOICE COMMAND THEREOF}

The present disclosure relates to an electronic device and a method of controlling at least one device such that the electronic device executes a function according to a voice command uttered by a user.

The use of electronic devices that recognize voice commands uttered by users is gradually increasing. Electronic devices that recognize voice commands are being actively used not only in PCs and mobile communication devices, but also in home appliances such as digital TVs, air conditioners and home theaters.

In order for the electronic device to accurately recognize the voice command, the user needs to clearly utter the voice command. For example, if the electronic device that the user wants to execute, the action to be executed by the electronic device, and the execution target of the electronic device are clearly described in the user's voice command, the electronic device better recognizes the user's voice command and Can perform the function according to.

Meanwhile, as a process for recognizing a voice command, a large amount of resources may be required to pre-process the voice command as voice data and to analyze the pre-processed voice data. Accordingly, a method of recognizing voice by transmitting voice data to a server or distributing a part of the voice recognition process with other devices to recognize voice is considered.

The present disclosure relates to a method for executing a function of an electronic device according to a voice command, and an object of the present disclosure is to determine a function to be performed by the electronic device according to the voice command by recognizing a voice command uttered by a user well. In addition, an object of the present disclosure is to more quickly and accurately recognize a voice command uttered by a user by using resources of each of electronic devices connected to communication.

An electronic device according to the present disclosure includes a memory including a plurality of databases classified according to a predicate, a communication circuit for receiving voice data corresponding to a voice command from an external device, and a processor operably connected to the memory and the communication circuit Includes. The processor identifies a type of a predicate included in the voice data, and based on information included in a database corresponding to the identified predicate, at least one device to execute the voice command and the at least one device according to the voice command A function to be performed by one device is determined, and a function according to the voice command is executed in the at least one device.

The method of controlling an electronic device according to the present disclosure includes an operation of receiving voice data corresponding to a voice command, an operation of identifying a type of a predicate included in the voice data, and information included in a database corresponding to the predicate. Determining at least one device to execute the voice command and a function to be performed by the at least one device according to the voice command; And controlling the at least one device to execute a function according to the voice command.

According to the present disclosure, a user can execute a function of an electronic device with a voice command that looks like a conversation in a general colloquial language without uttering an accurate voice command.

For example, when a user wants to execute the function of the robot cleaner, the user can execute the function of the robot cleaner by omitting the subject and concisely saying “clean,” rather than uttering “Clean the robot cleaner”. Alternatively, the user may execute the function of the robot cleaner by uttering "It is messy" using an adjective.

As another example, when the user wants to execute the function of the air conditioner, the user can execute the function of the air conditioner by igniting "it is cold" using an adjective rather than igniting "Raise the temperature of the air conditioner room".

According to the present disclosure, even if a user omits part of a predicate and utters a voice command, the electronic device may estimate the omitted predicate from information included in a database to provide accurate voice recognition. In particular, when there are multiple subjects to perform a function, the electronic device may control a specific device that reflects a user's intention to perform a function according to a voice command.

1 is a diagram illustrating a speech recognition system according to an embodiment of the present disclosure.
2 is a block diagram of an electronic device according to an embodiment of the present disclosure.
3 and 4 are diagrams illustrating predicate databases according to an embodiment of the present disclosure.
5 is a diagram illustrating a macro database according to an embodiment of the present disclosure.
6 is a flowchart of an electronic device according to an embodiment of the present disclosure.
7 is a flowchart of an electronic device according to another exemplary embodiment of the present disclosure.
8 is a flowchart of a speech recognition system according to an embodiment of the present disclosure.
9 is a block diagram of an electronic device in a network environment according to an embodiment of the present disclosure.

Various embodiments of the present disclosure and terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various changes, equivalents, or substitutes for the corresponding embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or a plurality of the items unless clearly indicated otherwise in a related context.

In the present disclosure, “A or B”, “at least one of A and B”, “at least one of A or B”, “A, B or C”, “at least one of A, B and C”, “A and Each of the phrases such as "B" and "at least one of A, B, or C" may include any one of the items listed together in the corresponding one of the phrases, or all possible combinations thereof. Terms such as "first", "second", or "first" or "second" may be used simply to distinguish the component from other corresponding components, and the components may be referred to in other aspects (eg, importance or Order) is not limited. Some (eg, a first) component is referred to as “coupled” or “connected” with or without the terms “functionally” or “communicatively” to another (eg, second) component. When mentioned, it means that any of the above components can be connected to the other components directly (eg by wire), wirelessly, or via a third component.

In the present disclosure, a predicate (or a verb) is a word describing a motion, a state, and a property, and may include a verb and an adjective. Verbs are words that represent the movement or process of people or things, and adjectives can be words that represent the state or nature of people or things. In addition, in the present disclosure, a morpheme is a unit of speech having a meaning, and may further include a noun that may be an object or a subject, including verbs and adjectives that may be the aforementioned predicates.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 shows a speech recognition system according to an embodiment of the present disclosure.

In FIG. 1, the speech recognition system 1 includes a speech acquisition device (or a dot device) 100, a speech analysis device (or edge devices) 200, 201, 202, and a speech execution device (or a voice device). An instruction execution device, a Thing device) 300 may be included.

The voice acquisition device 100, the voice analysis devices 200, and the voice execution device 300 may be various types of electronic devices. Various types of electronic devices may include portable electronic devices such as smart phones, computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances, but are not limited to the above-described devices.

Meanwhile, the names of devices such as the voice acquisition device 100, the voice analysis device 200, and the voice execution device 300 are terms defined from one function among various functions performed by each device for convenience of description. , According to embodiments, the voice acquisition device 100 may perform a voice analysis function and/or a voice execution function, the voice analysis device 200 may perform a voice preprocessing and/or a voice execution function, and The execution device 300 may perform a voice preprocessing function and/or a voice execution function. In an embodiment, the speech analysis apparatus 200 may have more abundant resources than the speech acquisition apparatus 100 and/or the speech execution apparatus 300 for speech analysis. For example, the speech analysis apparatus 200 may include a rule-based model and/or an artificial neural network-based model required for speech recognition. Neural networks used in the creation, update, or inference process of artificial neural network models are CNN (Convolutional Neural Network), DNN (Deep Neural Network), RNN (Recurrent Neural Network), RBM (Restricted Boltzmann Machine), DBN (Deep Belief Network). , BRDNN (Bidirectional Recurrent Deep Neural Network) or Deep Q-Networks, etc. may be provided, but are not limited to the above-described example. Such an artificial neural network model may be stored in the memory of the speech analysis apparatus 200 or in an external device capable of communicating with the speech analysis apparatus 200 in the form of software, or integrated into an on-device artificial intelligence chip and the speech analysis apparatus 200 It could be part of the processor.

The voice acquisition device 100 may receive a voice command uttered by a user and convert it into an electrical voice signal. To this end, the voice acquisition device 100 may include a microphone. The converted voice signal has a PCM (pulse code modulation) state or a compressed form of an audio waveform. The voice acquisition apparatus 100 may further include an A/D conversion unit for digitally converting a user's voice signal. On the other hand, when the audio acquisition device 100 includes a digital microphone, the audio acquisition device 100 may not include a separate A/D conversion unit. In various embodiments, the voice acquisition device 100 may be a microphone transmitter, a remote controller, a mobile phone, or the like, but is not limited to the above-described example.

The voice acquisition device 100 may perform a voice preprocessing function. For example, the voice acquisition device 100 divides the digitally converted voice data into syntactic units (eg, words, phrases, morphemes, etc.) using a natural language understanding module, and determines which grammatical element the divided units have. I can. In various embodiments, the voice preprocessing function of the voice acquisition device 100 may be performed by at least one of the plurality of voice analysis devices 200, 201, and 202.

When the voice data is classified according to the grammatical element, the voice acquisition device 100 may transmit the preprocessed voice data having the grammatical element to at least one of the plurality of voice analysis devices 200, 201, and 202. In this case, the voice acquisition device 100 may select at least one voice analysis device to transmit the pre-processed voice data from among the plurality of voice analysis devices 200, 201, and 202 based on a specific condition. For example, the voice acquisition device 100 has the closest distance, the best communication state, or established a communication channel among a plurality of voice analysis devices 200, 201, 202, or analyzes the voice by a user or a manufacturer. The voice analysis device set as the target can be prioritized. In the present disclosure, it is assumed that one voice analysis device 200 is selected from among a plurality of voice analysis devices 200, 201, and 202 for convenience of description.

Having received the preprocessed voice data having a grammatical element, the voice analysis apparatus 200 may analyze the obtained voice data to determine a service corresponding to the voice command uttered by the user.

In various embodiments, the speech analysis apparatus 200 performs a pre-processing function on speech data obtained from the speech acquisition apparatus 100, analyzes preprocessed speech data having grammatical elements, and responds to speech commands uttered by the user. It is also possible to determine the corresponding service.

In various embodiments, the voice analysis apparatus 200 may determine whether to analyze the voice data by considering its own resource state. For example, the voice analysis apparatus 200 may determine whether its own resource state is appropriate for analyzing the voice data, taking into account whether the voice data can be analyzed and the analysis time of the voice data. When its own resource state is not suitable for analyzing the voice data, the voice analysis device 200 transmits the voice data obtained from the voice acquisition device 100 to at least one of the other voice analysis devices 201 and 202 , At least one voice analysis device receiving the voice data may be controlled to analyze the voice data.

When the speech analysis apparatus 200 analyzes the speech data, the speech analysis apparatus 200 identifies the type of a predicate included in the speech data and includes it in a database (DB) corresponding to the identified predicate. Information can be obtained.

The speech analysis apparatus 200 may include a memory including at least one database required for speech analysis. At least one database may be classified according to, for example, a predicate. For example, the at least one database may include at least one of an adjective database, a verb database, and a macro database. Specific examples of the adjective database, verb database, and macro database will be described in detail later.

According to an embodiment, each of the plurality of speech analysis devices 200, 201, 202 may include all of an adjective database, a verb database, and a macro database, and each of the plurality of speech analysis devices 200, 201, 202 They may be distributed to each other to include at least one of an adjective database, a verb database, and a macro database, respectively. For example, the first voice analysis apparatus 200 may include an adjective database, the second voice analysis apparatus 201 may include a verb database, and the third voice analysis apparatus 202 may include a macro database.

In various embodiments, when information corresponding to a predicate included in the voice data does not exist in the database, the voice analysis apparatus 200 may transmit the predicate included in the voice data to the external server 400. The external server 400 may search for information corresponding to a predicate received from a database for each predicate stored in the external server 400. The external server 400 may transmit information corresponding to the searched predicate to the speech analysis apparatus 200. The external server 400 may include, for example, a cloud server. In general, a cloud server can be used as a storage method using a software environment to register data and use the registered data regardless of fixed hardware like a floating cloud in an Internet environment. There can be several types of cloud services. For example, it can be classified into SaaS (Software as a Service), PaaS (Platform as a Service), HaaS (Hardware as a Service), IaaS (Infrastructure as a Service), etc., depending on the service characteristics. It is conceptually the same as there is.

The speech analysis apparatus 200 is based on information retrieved from the database of the speech analysis apparatus 200 or information received from an external server, at least one device to execute a voice command and a function to be performed by at least one device according to the voice command Can be determined. For example, the speech analysis device 200 may determine the speech analysis device 200 itself as a device to execute a voice command or determine the voice execution device 300 based on information on a device corresponding to a predicate included in the database. You can decide which device to execute the voice command.

In various embodiments, when information corresponding to the identified predicate does not exist in the database, the voice analysis device 200 transfers the voice data to at least one of the other voice analysis devices 201 and 201 to deliver the voice data. It is possible to control the received at least one voice analysis device to analyze the voice data. Alternatively, the speech analysis apparatus 200 may update the database when information corresponding to the identified predicate does not exist in the database. For example, the speech analysis apparatus 200 may request and obtain information included in the database from other speech analysis apparatuses 201 and 202, and update the database of the speech analysis apparatus 200. That is, the voice analysis apparatus 200 may synchronize its own database with databases of other voice analysis apparatuses 201 and 202.

When at least one device to execute the voice command and the function to be performed by the at least one device are determined according to the voice command, the voice analysis device 200 controls the function according to the voice command to be executed in at least one device to execute the voice command. can do. In an embodiment, when the voice analysis device 200 itself is determined as a device to execute a voice command, a function according to the voice command may be executed based on information about an action corresponding to a predicate included in a database. In another embodiment, when a device to execute a voice command is identified as the voice execution device 300, the voice analysis device 200 may transmit information about an action acquired from a database to the voice execution device 300.

When the function according to the voice command is executed in the voice analysis device 200 or the voice execution device 300, the voice analysis device 200 or the voice execution device 300 indicates that the function according to the voice command has been executed or is being executed. Voice command feedback can be provided to the user. For example, the voice analysis device 200 or the voice execution device 300 may transmit information about voice command feedback to the voice acquisition device 100. Based on the information on the voice command feedback, the voice acquisition device 100 may provide the voice command feedback to the user. Voice command feedback may be, for example, visual feedback, auditory feedback, or tactile feedback.

The user may provide user feedback in response to the voice command feedback. The user may provide user feedback as a voice command, or may provide user feedback through a screen provided by the voice acquisition device 100 or the voice analysis device 200. The speech analysis apparatus 200 may update a plurality of databases classified according to a predicate based on information about the provided user feedback.

2 is a block diagram of an electronic device according to an embodiment of the present disclosure.

The configuration of the electronic device 500 of FIG. 2 may correspond to the configuration of the voice acquisition device 100, the voice analysis device 200, or the voice execution device 300 of FIG. 1, but in the present disclosure, the voice analysis device ( 200).

The electronic device 500 may include a communication circuit 510, a processor 520, and a memory 530. The electronic device 500 may include at least some of the configuration and/or functions of the electronic device 900 of FIG. 9 to be described later.

The communication circuit 510 may include a wired communication module and/or a wireless communication module for supporting various types of communication. The communication circuit 510 may include at least some of the configuration and/or functions of the communication module 990 of FIG. 9 to be described later. The communication circuit 510 may receive voice data from an external device (eg, the voice acquisition device 100 of FIG. 1) under the control of the processor 520 and transmit the voice data to the processor 520. Further, the communication circuit 510 may transmit information to an external server or another electronic device (eg, the voice analysis devices 201 and 202 of FIG. 1) under the control of the processor 520. For example, the communication circuit 510 may transmit voice data and information on a database to an external server or other electronic device.

The memory 530 may include a volatile memory and/or a non-volatile memory. The memory 530 may include at least some of the configuration and/or functions of the memory 530 of FIG. 9 to be described later. The memory 530 may be operatively and/or electrically connected to the processor 520 and may store various instructions that may be executed by the processor 520. Such instructions may include arithmetic and logical operations that can be recognized by the processor 520, data movement, and control instructions such as input/output. Also, the memory 530 may store a database. The database is a database classified for each predicate and may include a verb database and/or an adjective database. Also, the database may include a macro database.

The processor 520 is a configuration capable of performing an operation or data processing related to control and/or communication of each component of the electronic device 500, and among the configuration and/or functions of the processor 520 of FIG. 9 to be described later It may include at least some. The processor 520 may be operatively and/or electrically connected to internal components of the electronic device 500 such as the communication circuit 510 and the memory 530.

There will be no limit to the operation and data processing functions that the processor 520 can implement in the electronic device 500, but in the present disclosure, the processor 520 loads instructions stored in the memory 530 to describe embodiments to be described later. Can be done.

According to various embodiments, when voice data corresponding to a voice command is received from an external device (for example, the voice acquisition device 100 of FIG. 1 ), the processor 520 determines the type of a predicate included in the received voice data. Can be identified. In this case, when the voice command is preset as a macro, the processor 520 may acquire processed voice data corresponding to the preset macro and identify a type of a predicate included in the processed voice data.

The processor 520 may select a database corresponding to the identified predicate from among a plurality of databases stored in the memory 530.

The processor 520 may determine at least one device to execute a voice command and a function to be performed by the at least one device according to the voice command based on information included in the database corresponding to the identified predicate stored in the memory 530. have. In this case, if there are a plurality of devices to execute the voice command, the processor 520 may request user confirmation from among the plurality of devices to actually execute a function according to the voice command. For example, the processor 520 controls the communication circuit 510 to transmit information on a plurality of devices and a function to be performed by the plurality of devices to an external device (eg, the voice acquisition device 100 of FIG. 1 ), and the The communication circuit 510 may be controlled to receive information on a device selected by a user from among a plurality of devices from an external device.

In various embodiments, when information corresponding to a predicate does not exist in a plurality of databases, the processor 520 transmits the predicate to an external server (eg, the external server 400 of FIG. 1) and transmits the predicate from the external server. The communication circuit 510 may be controlled to receive information corresponding to.

When at least one device and function is determined, the processor 520 may control the at least one device to execute a function according to a voice command. For example, the processor 520 may execute a function according to a voice command as at least one device is determined as the electronic device 500 based on information included in a database stored in the memory 530. As another example, as the processor 520 determines that at least one device is a third device (eg, the voice execution device 300 of FIG. 1) based on information included in the database stored in the memory 530, the voice The communication circuit 510 may be controlled so that information on a function according to a command is transmitted to the third device.

In various embodiments, when information about user feedback is obtained for a function according to a voice command, the processor 520 may update a plurality of databases classified according to a predicate based on the information about the user feedback.

3 and 4 are diagrams illustrating predicate databases according to an embodiment of the present disclosure.

The predicate database of FIGS. 3 and 4 may be included in the memory 530 of FIG. 2. Alternatively, the predicate database may be stored in a memory of an external device capable of communicating with the electronic device 500. In the predicate database, for example, there may be at least one of an adjective database and a verb database. When the electronic device 500 receives preprocessed voice data having a grammatical element, the electronic device 500 may identify a type of a predicate as a grammatical element included in the voice data. For example, the electronic device 500 may identify whether a predicate included in the voice data includes a verb or an adjective. When the type of the predicate is identified, the electronic device 500 may select a database corresponding to the identified predicate from among a plurality of databases included in the memory 450. The electronic device 500 may obtain information on a device for executing a user's voice command and information on an action from the selected database.

As an example of such a database, the adjective database of FIG. 3 includes a user requirement field 302 corresponding to the adjective, a supportable edge device field 303 corresponding to the adjective, and the adjective field 301 It may include at least one of a supportable Thing device field 304 corresponding to an adjective or an action field 305 corresponding to the adjective.

For example, in 311 of FIG. 3, when the adjective is'cold', a user requirement corresponding to'cold' may be'raise indoor temperature'. In addition, there is no edge device corresponding to'cold', and the Thing device corresponding to'cold' may be'air conditioner'. And, the action for the'air conditioner' may be to increase the room temperature by turning off or weakening the'air conditioner'.

As another example, in 312 of FIG. 3, when the adjective is'noisy', the user requirement corresponding to'noisy' is'the sound is too loud'. It could be'lower the sound.' In addition, edge devices corresponding to'Noisy' may include'TV' and'sound equipment', and Thing devices corresponding to'Noisy' include'air conditioners','dehumidifiers','air purifiers','washing machines', A'dryer' may be included. And, the action for'TV' and'sound device' is to lower the sound, and the action for'air conditioner','dehumidifier','air purifier','washing machine' and'dryer' is to lower the sound. The setting of the operation mode may be adjusted, or the setting of the operation mode may be collectively adjusted by grouping.

Although the action of FIG. 3 is described in natural language to aid understanding of the invention, the action may include a command required to perform a function. For example, the action 311 of FIG. 3 may include commands such as turning off the'air conditioner', lowering the temperature of the'air conditioner', and maintaining the'air conditioner' at a constant temperature.

As another example of the predicate database, the verb database of FIG. 4 includes a user requirement field 402 corresponding to the verb and a subject device (or subject) field corresponding to the verb based on the verb field 401 ( 402) or an action target (or object) field 403 corresponding to the verb. Here, the subject device is a device that executes a function according to a voice command, and may include, for example, an edge device (or a voice analysis device) 200 and a Thing device (or a voice execution device) 300. In addition, the action target may include a target of a function performed by the electronic device 500 in response to the verb.

For example, in 401 of FIG. 4, when the verb is'clean up', the main device that executes the function corresponding to'clean up' is'robot cleaner', and the action target corresponding to the verb is'floor'dust' Can be.

As another example, in 402 of FIG. 4, when the verb is'Please be quiet', the main devices that execute a function corresponding to'Please be quiet' are'air conditioner','TV','washing machine','robot cleaner', ' It could be an air purifier. In addition, the action object corresponding to the verb may be'sound'.

In various embodiments, information included in the predicate database may be generated, added, or updated based on a user's voice command. For example, the electronic device 500 may identify a grammatical element of voice data included in a user's voice command. For example, the electronic device 500 may identify whether at least one of a verb, an adjective, or an object is included in the voice data. When a grammatical element of voice data is identified, the electronic device 500 may record the identified subject, verb, adjective, or object in a verb database or adjective database.

For example, the electronic device 500 records the identified adjective in the adjective field 301 of the adjective database of FIG. 3, and assigns a subject to the supportable Edge device field 303 or the supportable Thing device field 304 of the adjective database. And record the verb and object in the action field 305. As another example, the electronic device 500 records the identified verb in the verb field 401 of the verb database of FIG. 4, records the subject in the subject device field 402 of the verb database, and records the subject in the action target field 403. Objects can be recorded.

In various embodiments, the electronic device 500 may share a predicate database with other devices. For example, information included in the predicate database of the first voice analysis device 200 may be transferred to the predicate database of the second voice analysis device 201 and the predicate database of the third voice analysis device 202. Conversely, information included in the predicate database of the second voice analysis apparatus 201 and the predicate database of the third voice analysis apparatus 202 may be transferred to the predicate database of the first voice analysis apparatus 200.

The electronic device 500 may periodically share the predicate database with other devices, or may share the predicate database when an event occurs. For example, when an update request event from the user occurs or the database of the electronic device 500 is updated according to a preset user feedback, the electronic device 500 may share the predicate database with other devices.

5 is a diagram illustrating a macro database according to an embodiment of the present disclosure.

The macro database of FIG. 5 may include a preset macro name field 501 and a processed voice data field 502 corresponding to the macro name.

For example, in 501 of FIG. 5, when the macro name is “I will go out and come”, the processed voice data corresponding to the macro name may be “Turn off all electronic products”. As another example, in 502 of FIG. 5, if the macro name is'Hi Bixby', the processed voice data corresponding to the macro name is'Tell me today's news and weather. Turn the air purifier back for 2 hours and set the air conditioner temperature to 27 degrees.

The setting of the processed voice data corresponding to the macro name may be set by the user, or is written as default information by the manufacturer of the electronic device 500, the provider of the application installed in the electronic device 500, or the provider of the operating system. Can be set. As an example, the electronic device 500 may set a macro name in consideration of a sentence format uttered by the user. For example, the user can set the macro name in the form of a conditional sentence. For example, the user may say, "Bixby! In the future, if I say'I'll go out', turn off all electronic products'", in the form of a conditional sentence. In this case, the electronic device 500 may determine the processed voice data to be performed according to the macro name and the macro name from the voice command uttered by the user.

For example, as shown in 511 of FIG. 5, the electronic device 500 sets'I'll go out', which is voice data corresponding to a voice command, as a macro name, and'turn off power to all electronic products' as processed voice data corresponding to the macro name. Turn it off' can be set.

In this case, when the user utters'I will go out', the electronic device 500 identifies whether a macro name corresponding to the voice command uttered in the macro database exists, and when there is a macro name corresponding to the voice command, 'Turn off all electronic products', which is the processed voice data corresponding to the macro name, can be obtained. The electronic device 500 identifies the type of the predicate of the acquired processed voice data, and based on information included in the database corresponding to the predicate, at least one device to execute the entire voice command, and the at least one device You can decide what functions to do. In addition, the electronic device 500 may control the at least one device to execute a function according to all voice commands.

6 is a flowchart of an electronic device according to an embodiment of the present disclosure.

First, in operation 601, the electronic device 500 may obtain voice data corresponding to a voice command uttered by a user from an external device (eg, the dot device 100).

In operation 603, when the voice command uttered by the user by analyzing the voice data is preset as a macro, the electronic device 500 that obtains the voice data searches for processed voice data corresponding to the preset macro from the macro database, Processed voice data corresponding to the voice command may be obtained. If the voice command uttered by the user is not previously set as a macro, operation 603 may be omitted.

In operation 605, the acquired electronic device 500 that has obtained the voice data may identify a type of a predicate included in the voice data.

In operation 607, the electronic device 500 may determine whether a verb is included in the voice data.

When the verb is included, in operation 609, the electronic device 500 may determine whether the subject is included in the voice data.

When a verb and a subject are included in the voice data, in operation 611, the electronic device 500 transmits a command corresponding to the verb to a device corresponding to the subject (eg, the Edge device 200 or the Thing device 300). , It is possible to control to perform a function according to the voice command.

On the other hand, as a result of the determination in operation 609, if the verb exists but the subject does not exist, in operation 613, the electronic device 500 analyzes the previous voice command of the voice command uttered by the user, and It may be determined whether a device to be referred to (eg, the Edge device 200 or the Thing device 300) exists.

If there is a device (eg, Edge device 200 or Thing device 300) indicated by the user in the previous voice command, the electronic device 500 corresponds to the subject indicated by the user, as in operation 611 described above. By transmitting a command corresponding to the verb to the device to be used, the function according to the voice command can be controlled to be performed.

On the other hand, as a result of the determination in operation 613, if there is no device indicated by the user in the previous voice command, in operation 615, the electronic device 500 is information included in the verb database, and the subject device corresponding to the verb (for example, : Based on the information on the edge device 200 or the Thing device 300 and the action target, a device to execute a voice command and a function to be performed by the device according to the voice command may be determined. At this time, if there is no information corresponding to the verb in the verb database, the electronic device 500 transmits the verb identified to an external server or the voice data obtained in operation 601, and the device and voice to execute a voice command in response thereto. According to an instruction, information on a function to be performed by the device may be received.

Meanwhile, in operation 607, when the verb is not included in the voice data, in operation 617, the electronic device 500 may determine whether the adjective is included.

When the adjective is included, in operation 619, the electronic device 500 is information included in the adjective database, and information on the subject device (eg, Edge device 200 or Thing device 300) corresponding to the adjective and action Based on, a device for executing a voice command and a function to be performed by the device according to the voice command may be determined. At this time, if there is no information corresponding to the adjective in the adjective database, the electronic device 500 transmits the adjective identified to an external server or the voice data obtained in operation 601, and the device and voice to execute a voice command in response thereto. According to an instruction, information on a function to be performed by the device may be received.

In operations 615 and 619, when a device to execute a voice command and a function to be performed by the device are determined according to the voice command, in operation 621, the electronic device 500 may determine whether there are a plurality of devices capable of executing the voice command. .

If it is determined that there are not a plurality of devices capable of executing the voice command, in operation 623, the electronic device 500 may control the device to execute the voice command to execute a function according to the voice command. For example, when a device to execute a function according to a voice command is identified as the electronic device 500 itself, the electronic device 500 may directly perform the function. As another example, when a device to execute a function according to a voice command is identified as a third device, the electronic device 500 may transmit information on the function according to the voice command to the third device.

On the other hand, when there are a plurality of devices capable of executing a voice command, in operation 625, the electronic device 500 may receive a user confirmation for a device that will actually execute a function according to the voice command among a plurality of devices capable of executing the voice command. . For example, the electronic device 500 may provide information on a plurality of devices and functions to be performed by the plurality of devices to an external device (eg, the dot device 100 or the Thing device 300), a device being used by the user, or a user. Can be delivered to adjacent devices. The device receiving the information may provide a user with description information about a plurality of devices and functions to be performed by the plurality of devices. For example, the device may provide the description information through a menu on the screen or may provide the description information through sound. When the user selects a device to execute a voice command with reference to the description information, the device may transmit information on the device selected by the user to the electronic device 500. In various embodiments, the electronic device 500 may directly provide a plurality of devices and description information about functions to be performed by the plurality of devices to a user.

Based on the description information, when the user selects at least one of the plurality of devices, as in operation 623, the electronic device 500 performs a voice function according to a voice command in the at least one device selected by the user. You can control it to run.

When the function according to the voice command is executed in at least one device, in operation 627, the electronic device 500 or an external device (for example, the dot device 100 or the Thing device 300) executes the function according to the voice command. Alternatively, voice command feedback indicating that it is running may be provided to the user. Providing the voice command feedback to the user may include lighting or blinking an LED as visual feedback, displaying a notification message on a screen, or providing voice or sound as an audible feedback.

When the voice command feedback is provided, in operation 629, the electronic device 500 or an external device (eg, the dot device 100 or the Thing device 300) receives the user feedback input by the user according to the voice command feedback information. I can. When the external device receives user feedback, the external device may transmit information about the received user feedback to the electronic device 500.

Based on the information on the user feedback, in operation 631, the electronic device 500 may determine the type of user feedback. For example, the electronic device 500 may determine whether or not a user's feedback exists. When there is user feedback, the electronic device 500 may determine whether the user feedback is a positive feedback or a negative feedback. The positive feedback may be, for example, a voice feedback such as "good job" or "thank you". In addition, the positive feedback may be, for example, a voice feedback such as "no" or "do it again". In various embodiments, when there is no user feedback, the electronic device 500 may regard the user feedback as being a positive feedback type.

In operation 633, the electronic device 500 may update a plurality of databases classified according to a predicate based on user feedback. For example, when the user's feedback is positive feedback, the electronic device 500 may assign a weight to information corresponding to a voice command in the database. On the other hand, when the user's feedback is a negative feedback, the electronic device 500 may attenuate or delete the weight of information corresponding to the voice command in the database, or supplement or modify it by reflecting the user's voice command again.

In operation 635, the electronic device 500 may share information included in a plurality of updated databases with an external device (eg, edge devices 201 and 202). Also, the electronic device 500 may share information included in a plurality of updated databases with the external server 400. In this case, before the electronic device 500 shares information included in the updated plurality of databases with other external devices (eg, edge devices 201 and 202) or an external server 400, the user for information sharing You can also ask for your consent.

In various embodiments, the electronic device 500 may selectively update a plurality of databases based on the type of predicate included in the voice command. For example, when the predicate included in the voice command is identified as a verb, the electronic device 500 updates the verb database, and when the predicate included in the voice command is identified as an adjective, the electronic device 500 updates the adjective database. can do.

7 is a flowchart of an electronic device according to another exemplary embodiment of the present disclosure.

First, in operation 701, the electronic device 100 may obtain voice data corresponding to a voice command uttered by a user from an external device (eg, the dot device 100).

In operation 703, the electronic device 500 that has obtained the voice data may analyze the voice data to determine whether the voice command uttered by the user is a voice command for setting a macro.

If the voice command is not a voice command for setting the macro, in operation 705, the electronic device 500 may control to execute a function according to the voice command. The related process corresponds to the above-described operations 605 to 635, and thus redundant descriptions will be omitted.

When the voice command is a voice command for setting a macro, in operation 707, the electronic device 500 may analyze voice data corresponding to the voice command and determine the processed voice data to be performed according to the macro name and the macro name. . That is, the electronic device 500 may separate the macro name from the voice data and determine processed voice data corresponding to the separated macro name.

When the processed voice data to be performed is determined according to the macro name and the macro name, in operation 709, the electronic device 500 may register the macro name and the processed voice data in the macro database. Additionally, in operation 711, the electronic device 500 may share the registered macro name and processed voice data with an external device (eg, edge devices 201 and 202). Another external device may update its own macro database by using the macro name and processed voice data received from the electronic device 500.

8 is a flowchart of a speech recognition system according to an embodiment of the present disclosure.

In FIG. 8, the first electronic device 801 corresponds to the voice acquisition device 100 of FIG. 1, the second electronic device 803 corresponds to the voice analysis device 200 of FIG. 1, and a third electronic device Reference numeral 805 may correspond to the voice execution device 300 of FIG. 1.

First, in operation 811, the first electronic device 801 may transmit voice data generated by receiving a user's voice command to the second electronic device 803.

In operation 813, the second electronic device 803 may receive the transmitted voice data and identify a type of a predicate included in the voice data. When the type of the predicate is identified, the second electronic device 803 may select a database corresponding to the identified predicate from among a plurality of databases.

In various embodiments, when voice data is received, the second electronic device 803 may identify whether a user's voice command is preset to a macro based on the received voice data. When the user's voice command is preset to a macro, the second electronic device 803 may obtain processed voice data corresponding to the preset macro. In addition, when the type of the predicate included in the processed voice data is identified, the second electronic device 803 may select a database corresponding to the identified predicate from among a plurality of databases.

In operation 815, the second electronic device 803 may determine at least one device to execute a voice command and a function to be performed by at least one device according to the voice command, based on information included in the database corresponding to the predicate. .

In operation 817, the second electronic device 803 may determine at least one device to execute the voice command. In this case, when there are a plurality of devices to execute the voice command, the second electronic device 803 may request user confirmation from among the plurality of devices to actually execute a function according to the voice command.

When at least one device to execute the voice command is the second electronic device 803 itself, in operation 819, the second electronic device 803 may execute a function according to the voice command.

On the other hand, when at least one device to execute the voice command is the third electronic device 805, in operation 821, the second electronic device 803 transmits information on a function according to the voice command to the third electronic device 805. Can be sent to

In operation 823, the third electronic device 805 may execute a function according to a voice command based on information about the received function.

When the function according to the voice command is executed, the second electronic device 803 may obtain information about user feedback on the function according to the voice command. The second electronic device 803 may update a plurality of databases classified according to predicates based on the information on user feedback.

9 is a block diagram of an electronic device 900 in a network environment according to an embodiment of the present disclosure. The electronic device 900 may be an example of the electronic device 500 of FIG. 2 described above.

Referring to FIG. 9, in a network environment, the electronic device 900 communicates with the electronic device 902 through a first network 998 (eg, short-range wireless communication network), or a second network 999 (eg: It is possible to communicate with the electronic device 904 or the server 908 (eg, the external server 400 of FIG. 1) through a long-distance wireless communication network. According to an embodiment, the electronic device 900 may communicate with the electronic device 904 through the server 908. According to an embodiment, the electronic device 900 includes a processor 920, a memory 930, an input device 950, an audio output device 955, a display device 960, an audio module 970, and a sensor module ( 976), an interface 977, a power management module 988, a battery 989, and a communication module 990. In some embodiments, at least one of these components (for example, the display device 960) may be omitted or one or more other components may be added to the electronic device 900. In some embodiments, some of these components may be implemented as one integrated circuit. For example, the sensor module 976 (eg, a fingerprint sensor, an iris sensor, or an illuminance sensor) may be implemented while being embedded in the display device 960 (eg, a display).

The processor 920, for example, executes software (eg, a program 940) to implement at least one other component (eg, a hardware or software component) of the electronic device 900 connected to the processor 920. Control, and perform various data processing or operations. According to an embodiment, as at least part of data processing or operation, the processor 920 may store instructions or data received from other components (eg, the sensor module 976 or the communication module 990) to the volatile memory 932. It loads into, processes the command or data stored in the volatile memory 932, and stores the result data in the non-volatile memory 934. According to an embodiment, the processor 920 is a main processor 921 (eg, a central processing unit or an application processor), and a secondary processor 923 that can be operated independently or together with the main processor 921 (eg, a graphics processing unit, an image signal processor). , A sensor hub processor, or a communication processor). Additionally or alternatively, the coprocessor 923 may be configured to use lower power than the main processor 921 or to be specialized for a designated function. The secondary processor 923 may be implemented separately from the main processor 921 or as a part thereof.

The coprocessor 923 is, for example, on behalf of the main processor 921 while the main processor 921 is in an inactive (eg, sleep) state, or the main processor 921 is active (eg, an application is executed). ) While in the state, together with the main processor 921, at least one of the components of the electronic device 900 (for example, the display device 960, the sensor module 976, or the communication module 990) It is possible to control at least some of the functions or states related to. According to an embodiment, the coprocessor 923 (eg, an image signal processor or a communication processor) may be implemented as a part of the communication module 990 or another functionally related component.

The memory 930 may store various data used by at least one component (eg, the processor 920 or the sensor module 976) of the electronic device 900. The data may include, for example, software (eg, the program 940) and input data or output data for commands related thereto. The memory 930 may include a volatile memory 932 or a nonvolatile memory 934.

The program 940 may be stored as software in the memory 930, and may include, for example, an operating system 942, middleware 944, or an application 946.

The input device 950 may receive a command or data to be used for a component of the electronic device 900 (eg, the processor 920) from outside (eg, a user) of the electronic device 900. The input device 950 may include, for example, a microphone, a mouse, a keyboard, or a digital pen (eg, a stylus pen).

The sound output device 955 may output an sound signal to the outside of the electronic device 900. The sound output device 955 may include, for example, a speaker or a receiver. The speaker can be used for general purposes such as multimedia playback or recording playback, and the receiver can be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part of it.

The display device 960 may visually provide information to the outside of the electronic device 900 (eg, a user). The display device 960 may include, for example, a display, a hologram device, or a projector and a control circuit for controlling the device. According to an embodiment, the display device 960 may include a touch circuitry set to sense a touch, or a sensor circuit (eg, a pressure sensor) set to measure the strength of a force generated by the touch. have.

The audio module 970 may convert sound into an electric signal, or conversely, convert an electric signal into sound. According to an embodiment, the audio module 970 acquires sound through the input device 950, the sound output device 955, or an external electronic device directly or wirelessly connected to the electronic device 900 (for example, Sound may be output through the electronic device 102) (for example, a speaker or headphones).

The sensor module 976 detects an operating state (eg, power or temperature) of the electronic device 900 or an external environmental state (eg, a user state), and generates an electrical signal or data value corresponding to the detected state. can do. According to an embodiment, the sensor module 976 includes, for example, a gesture sensor, a gyro sensor, an atmospheric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an infrared (IR) sensor, a biometric sensor, It may include a temperature sensor, a humidity sensor, or an illuminance sensor.

The interface 977 may support one or more designated protocols that may be used for the electronic device 900 to connect directly or wirelessly with an external electronic device (eg, electronic device 902 ). According to an embodiment, the interface 977 may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

The connection terminal 978 may include a connector through which the electronic device 100 may be physically connected to an external electronic device (eg, the electronic device 902 ). According to an embodiment, the connection terminal 978 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (eg, a headphone connector).

The power management module 988 may manage power supplied to the electronic device 900. According to an embodiment, the power management module 988 may be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

The battery 989 may supply power to at least one component of the electronic device 900. According to an embodiment, the battery 989 may include, for example, a non-rechargeable primary cell, a rechargeable secondary cell, or a fuel cell.

The communication module 990 is a direct (eg, wired) communication channel or a wireless communication channel between the electronic device 900 and an external electronic device (eg, electronic device 902, electronic device 904, or server 908). It is possible to support establishment and communication through the established communication channel. The communication module 990 operates independently of the processor 920 (eg, an application processor) and may include one or more communication processors that support direct (eg, wired) communication or wireless communication. According to an embodiment, the communication module 990 is a wireless communication module 992 (eg, a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (eg : A LAN (local area network) communication module, or a power line communication module) may be included. Among these communication modules, a corresponding communication module is a first network 998 (for example, a short-range communication network such as Bluetooth, WiFi direct or IrDA (infrared data association)) or a second network 999 (for example, a cellular network, the Internet, or It can communicate with external electronic devices through a computer network (for example, a telecommunication network such as a LAN or WAN). These various types of communication modules may be integrated into one component (eg, a single chip), or may be implemented as a plurality of separate components (eg, multiple chips).

At least some of the components are connected to each other through a communication method (e.g., a bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI))) between peripheral devices and signals ( E.g. commands or data) can be exchanged with each other.

According to an embodiment, the command or data may be transmitted or received between the electronic device 900 and the external electronic device 904 through the server 908 connected to the second network 999. Each of the electronic devices 902 and 904 may be a device of the same or different type as the electronic device 900. According to an embodiment, all or some of the operations executed in the electronic device 900 may be executed in one or more of the external electronic devices 902, 904, or 908. For example, when the electronic device 900 needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device 900 may perform the function or service by itself. In addition or in addition, one or more external electronic devices may be requested to perform the function or at least part of the service. One or more external electronic devices that have received the request may execute at least a part of the requested function or service, or an additional function or service related to the request, and transmit the execution result to the electronic device 900. The electronic device 900 may process the result as it is or additionally and provide it as at least a part of a response to the request. To this end, for example, cloud computing, distributed computing, or client-server computing technology may be used.

The term “module” used in the present disclosure may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic blocks, parts, or circuits. The module may be an integrally configured component or a minimum unit of the component or a part thereof that performs one or more functions. For example, according to an embodiment, the module may be implemented in the form of an application-specific integrated circuit (ASIC).

Various embodiments of the present disclosure include one or more instructions stored in a storage medium (eg, memories 930 and 530) that can be read by a machine (eg, electronic devices 900 and 500). It may be implemented as software (for example, the program 940). For example, the processor (eg, the processors 920 and 520) of the device (eg, the electronic device 900) may call and execute at least one command among one or more commands stored from the storage medium. This makes it possible for the device to be operated to perform at least one function according to the at least one command invoked. The one or more instructions may include code generated by a compiler or code executable by an interpreter. A storage medium that can be read by a device may be provided in the form of a non-transitory storage medium. Here,'non-transient' only means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic wave), and this term refers to the case where data is semi-permanently stored in the storage medium. It does not distinguish between temporary storage cases.

According to an embodiment, in the storage medium that can be read by the device, when the electronic devices 900 and 500 are executed, the processor (eg, the processors 920 and 520) identifies the type of predicate included in the voice data. And, based on information included in the database corresponding to the identified predicate, at least one device to execute the voice command and a function to be performed by the at least one device according to the voice command are determined, and the at least one The device may include at least one instruction set to control execution of a function according to the voice command.

According to an embodiment, a method according to various embodiments disclosed in the present disclosure may be provided by being included in a computer program product. Computer program products can be traded between sellers and buyers as commodities. Computer program products are distributed in the form of a device-readable storage medium (e.g. compact disc read only memory (CD-ROM)), or through an application store (e.g. Play StoreTM) or two user devices ( It can be distributed (e.g., downloaded or uploaded) directly between, e.g. smartphones) In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a storage medium that can be read by a device such as a server of a manufacturer, a server of an application store, or a memory of a relay server.

According to an embodiment, each component (eg, a module or program) of the above-described components may include a singular number or a plurality of entities. According to an embodiment, one or more components or operations among the above-described corresponding components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (eg, a module or a program) may be integrated into one component. In this case, the integrated component may perform one or more functions of each component of the plurality of components in the same or similar to that performed by the corresponding component among the plurality of components prior to the integration. . According to one embodiment, operations performed by a module, program, or other component are sequentially, parallel, repeatedly, or heuristically executed, or one or more of the above operations are executed in a different order or omitted. Or one or more other actions may be added.

500: electronic device 510: communication circuit
520: processor 530: memory

Claims (20)

In the electronic device,
A memory including a plurality of databases classified according to predicates;
A communication circuit for receiving voice data corresponding to a voice command from an external device; And
A processor operatively connected with the memory and the communication circuit,
The processor,
Identify the type of predicate included in the voice data,
Determine at least one device to execute the voice command and a function to be performed by the at least one device according to the voice command, based on information included in the database corresponding to the identified predicate,
Controlling to execute a function according to the voice command in the at least one device,
Electronic device.
The method of claim 1,
The processor,
When controlling to execute the function according to the voice command,
Based on information included in the database, as the at least one device is determined to be the electronic device itself, executing a function according to the voice command,
Electronic device.
The method of claim 1,
The processor,
When controlling to execute the function according to the voice command,
Based on the information included in the database, as the at least one device is determined to be a third device, controlling the communication circuit to transmit information on a function according to the voice command to the third device,
Electronic device.
The method of claim 1,
The processor,
Selecting a database corresponding to the identified predicate from among the plurality of databases,
Electronic device.
The method of claim 1,
When there are multiple devices to execute the voice command,
The processor,
Requesting a user confirmation for a device that will actually execute a function according to the voice command among the plurality of devices,
Electronic device.
The method of claim 5,
The processor,
Controlling the communication circuit so that information on the plurality of devices and functions to be performed by the plurality of devices is transmitted to the external device,
Controlling the communication circuit to receive information on a device selected by a user from among the plurality of devices from the external device,
Electronic device.
The method of claim 1,
The processor,
When information regarding user feedback is obtained,
Updating a plurality of databases classified according to the predicate based on the information on the user feedback,
Electronic device.
The method of claim 1,
The processor,
When information corresponding to the predicate does not exist in the plurality of databases, controlling the communication circuit to transmit the predicate to an external server,
Controlling the communication circuit to receive information corresponding to the predicate from the external server,
Electronic device.
The method of claim 1,
The processor,
Identifying whether the voice command is preset as a macro;
When the voice command is preset as a macro, obtaining processed voice data corresponding to the preset macro,
Electronic device.
The method of claim 1,
The plurality of databases,
Containing verb database and adjective database,
Electronic device.
The method of claim 1,
When multiple databases include a verb database,
The information contained in the verb database,
Including at least one of a user requirement corresponding to the verb, a subject device corresponding to the verb, or an action object corresponding to the verb,
Electronic device.
The method of claim 1,
When multiple databases include an adjective database,
The information contained in the adjective database,
Including at least one of a user requirement corresponding to the adjective, a supportable speech analysis device corresponding to the adjective, a supportable speech execution device corresponding to the adjective, or an action corresponding to the adjective,
Electronic device.
In the control method of an electronic device,
Receiving voice data corresponding to a voice command;
Identifying the type of predicate included in the voice data;
Determining at least one device to execute the voice command and a function to be performed by the at least one device according to the voice command, based on information included in a database corresponding to the predicate; And
Including controlling to execute a function according to the voice command in the at least one device,
Control method.
The method of claim 13,
The operation of controlling to execute the function according to the voice command,
Based on information included in the database, as the at least one device is identified as the electronic device itself, executing a function according to the voice command,
Control method.
The method of claim 13,
The operation of controlling to execute the function according to the voice command,
Based on information included in the database, as the at least one device is identified as a third device, transmitting information on a function according to the voice command to the third device,
Control method.
The method of claim 13,
The above method,
Further comprising an operation of selecting a database corresponding to the identified predicate from among a plurality of databases,
Control method.
The method of claim 13,
The above method,
When there are multiple devices to execute the voice command,
Further comprising an operation of requesting user confirmation from a device that will actually execute a function according to the voice command among the plurality of devices,
Control method.
The method of claim 13,
The above method,
When information regarding user feedback is obtained,
Based on the information on the user feedback, further comprising the operation of updating a plurality of databases classified according to the predicate,
Control method.
The method of claim 13,
The above method,
Identifying whether the voice command is preset as a macro;
If the voice command is preset to a macro, further comprising an operation of obtaining processed voice data corresponding to the preset macro,
Control method.
The method of claim 13,
The plurality of databases,
Contains verb database and adjective database,
The information contained in the verb database,
Including at least one of a user requirement corresponding to the verb, a subject device corresponding to the verb, or an action object corresponding to the verb,
The information contained in the adjective database,
Including at least one of a user requirement corresponding to an adjective, a supportable speech analysis device corresponding to the adjective, a supportable speech execution device corresponding to the adjective, or an action corresponding to the adjective,
Control method.

Images