KR20220069777A - Electronic apparatus, control method thereof and electronic system - Google Patents

Abstract

Disclosed is an electronic device. The electronic device of the present invention comprises: a microphone; a memory in which a first neural network model and a second neural network model are stored; a communication interface; and a processor which is connected to the microphone, the memory and the communication interface to control the electronic device. The processor, when user's speech is received using the microphone, inputs the user's speech to the first neural network model to acquire a calculation result, inputs the calculation result to the second neural network to identify at least one device corresponding to the user's speech, and controls the communication interface to transmit the calculation result to the at least one device, wherein the first neural network model is configured to include only an additionally trained partial layer after only the partial layer of a third neural network model trained to identify text from the speech is additionally trained, and the second neural network is trained to identify devices corresponding to the speech. According to the present invention, the electronic device can identify a device corresponding to user's speech and enable the identified device to perform an operation corresponding to the user's speech.

Description

Electronic device, its control method and electronic system { ELECTRONIC APPARATUS, CONTROL METHOD THEREOF AND ELECTRONIC SYSTEM }

The present disclosure relates to an electronic device and a control method thereof, and more particularly, to an electronic device for controlling at least one device among a plurality of devices based on a user's voice, a control method thereof, and an electronic system.

With the development of electronic technology, various types of devices have been developed and distributed, and devices with various communication functions are widely used in most general homes. Furthermore, a communication function is provided even in a device that does not have a conventional communication function, thereby creating a multi device experience (MDE) environment.

In particular, recently, a technique for recognizing a user's voice using a neural network model and controlling a device in an MDE environment based on the recognition result has been developed.

However, when a new device is added in the MDE environment, there is a limit to using the existing neural network model as it is. For example, when a new device is added in the MDE environment, the existing neural network model does not have information about the new device, so there is a possibility of malfunction. Alternatively, re-learning of the neural network model for a new device is required, providing inconvenience to the user.

To solve this, voice recognition may be performed using a server. In this case, the above problem can be solved by updating the neural network model stored in the server periodically or according to a specific event. However, when using the server, problems such as personal information leakage may occur.

Accordingly, a method has been developed that does not use a server and recognizes a user's voice using a lightweight model of a device included in the MDE environment, while flexibly maintaining voice recognition performance without causing user inconvenience even when a new device is added. need to be

The present disclosure is in accordance with the above-mentioned necessity, and an object of the present disclosure is to provide an electronic device that maintains voice recognition performance and minimizes delay even when a device configuration is changed in a multi device experience (MDE) environment, a control method thereof, and To provide an electronic system.

According to an embodiment of the present disclosure for achieving the above object, an electronic device is connected to a microphone, a memory in which the first neural network model and the second neural network model are stored, a communication interface, and the microphone, the memory and the communication interface. a processor for controlling the electronic device, wherein when a user voice is received through the microphone, the processor receives the user voice into the first neural network model to obtain an operation result, and uses the operation result to the second neural network model to identify at least one device corresponding to the user's voice by inputting it to, and control the communication interface to transmit the operation result to the at least one device, and the first neural network model is trained to identify text from voice. After only some layers of the third neural network model are additionally trained, the second neural network model may be a model trained to identify a device corresponding to a voice.

In addition, the processor inputs the user voice into the first neural network model in units of a preset time interval to obtain the operation result in units of the predetermined time, and returns the operation result obtained in units of the predetermined time to the first neural network model. 2 input into a neural network model to identify the at least one device in the preset time unit, and transmit the operation result obtained in the preset time unit to the at least one device identified in the preset time unit Control the communication interface.

In addition, the memory further stores information on a plurality of devices and information on a plurality of projection layers, and the processor is a second processor that can be processed by the at least one device based on the information on the plurality of devices. Identifies dimension information, and when the first dimension and the second dimension of the operation result are different from each other, the operation result is based on a projection layer corresponding to the first dimension and the second dimension among the plurality of projection layers may be changed to the second dimension, and the communication interface may be controlled to transmit the changed second dimension calculation result to the at least one device.

In addition, the memory further stores information on a plurality of devices and the remaining layers of the third neural network model, and the processor does not include a voice recognition function in the at least one device based on the information on the plurality of devices. If it is identified as not, the communication interface may be controlled to obtain a text corresponding to the user's voice by inputting the operation result to the remaining layer, and transmit the obtained text to the at least one device.

Then, the processor inputs the calculation result to the second neural network model to obtain a score for each of a plurality of devices, and transmits the calculation result to a device having a score equal to or greater than a threshold value among the obtained scores. can control

In addition, the memory further stores information on a plurality of projection layers and the remaining layers of the third neural network model, and the processor transmits the operation result to the at least one device and then the at least one device Upon receiving a first response from When a response is received, the operation result may be processed as one of the plurality of projection layers or may be input to the remaining layers.

And, when the second response is information on a dimension processable by the at least one device, the processor is configured to: a projection ray corresponding to a dimension of the operation result among the plurality of projection layers and a dimension processable by the at least one device Change the dimension of the operation result based on , control the communication interface to transmit the operation result with the changed dimension to the at least one device, and if the second response is information indicating that the operation information cannot be processed , input the operation result to the remaining layer to obtain a text corresponding to the user's voice, and control the communication interface to transmit the obtained text to the at least one device.

In addition, the first neural network model fixes weight values of the remaining layers of the third neural network model, and is based on a plurality of sample user voices corresponding to the electronic device and a plurality of sample texts corresponding to the plurality of sample user voices. Thus, after the some layers are additionally learned, the model may be configured to include only the additionally learned partial layers.

And, the at least one device inputs the operation result to a fourth neural network model stored in the at least one device to obtain a text corresponding to the user's voice, and performs an operation corresponding to the obtained text; The fourth neural network model fixes the weight values of the partial layers, and the third neural network model is based on a plurality of sample user voices corresponding to the at least one device and a plurality of sample texts corresponding to the plurality of sample user voices. It may be a model configured to include only the additionally trained remaining layers after the remaining layers of .

According to an embodiment of the present disclosure for achieving the above object, when a user's voice is received, the electronic system inputs the user's voice into a first neural network model to obtain an operation result, and uses the operation result to a second neural network model. An electronic device that identifies at least one device corresponding to the user's voice by inputting it to the electronic device and transmits the calculation result to the at least one device, and a text corresponding to the user's voice by inputting the calculation result into a fourth neural network model and at least one device that performs an operation corresponding to the obtained text, wherein the first neural network model is further trained after only some layers of a third neural network model trained to identify text from speech are further trained. It is a model configured to include only some additionally trained layers, the second neural network model is a model trained to identify a device corresponding to a voice, and the fourth neural network model is after only the remaining layers of the third neural network model are additionally trained It may be a model configured to include only the additionally learned remaining layers.

In addition, when the first dimension of the operation result and the second dimension of the input of the fourth neural network model are different from each other, the at least one device identifies whether a projection layer corresponding to the first dimension exists, and the projection layer When is present, a first signal is transmitted to the electronic device, and when the projection layer does not exist, a second signal including information on the second dimension can be transmitted to the electronic device.

In addition, when the first signal is received, the electronic device maintains an operation of transmitting the operation result to the at least one device, and when the second signal is received, projects the operation result into the second dimension and The projected operation result may be transmitted to the at least one device, or the operation result may be decoded and the decoded operation result may be transmitted to the at least one device.

Meanwhile, according to an embodiment of the present disclosure, a method of controlling an electronic device includes, when a user voice is received, inputting the user voice into a first neural network model to obtain an operation result, and inputting the operation result into a second neural network model to identify at least one device corresponding to the user's voice and transmitting the operation result to the at least one device, wherein the first neural network model is a third neural network trained to identify text from voice After only some layers of the model are additionally trained, the model is configured to include only the additionally trained partial layers, and the second neural network model may be a model trained to identify a device corresponding to a voice.

In addition, the acquiring step inputs the user voice in a preset time interval unit to the first neural network model to obtain the calculation result in the preset time unit unit, and the identifying step is obtained in the preset time unit unit inputting the operation result to be obtained into the second neural network model to identify the at least one device in the preset time unit, and the transmitting includes the operation result obtained in the preset time unit in the preset time unit may be transmitted to the at least one device identified as .

and identifying information on a second dimension that can be processed by the at least one device based on information about a plurality of devices. When the first dimension and the second dimension of the operation result are different from each other, a plurality of projection layers The method further includes changing the calculation result into the second dimension based on a projection layer corresponding to the first dimension and the second dimension, wherein the transmitting includes the changed second dimension calculation result in the second dimension. It can be transmitted to at least one device.

In addition, when it is identified that the at least one device does not have a voice recognition function based on the information on the plurality of devices, the operation result is input to the remaining layers of the third neural network model to correspond to the user voice. The method may further include obtaining a text, wherein the transmitting may transmit the obtained text to the at least one device.

In the identifying step, the calculation result is inputted to the second neural network model to obtain a score for each of a plurality of devices, and the transmitting step includes the calculation result having a score equal to or greater than a threshold value among the obtained scores. can be transferred to the device.

In addition, when a first response is received from the at least one device after transmitting the calculation result to the at least one device, a subsequent calculation result is transmitted to the at least one device, and the calculation result is transmitted to the at least one device When the second response is received from the at least one device after transmitting to the device of

And, in the step of inputting, if the second response is information on a dimension processable by the at least one device, corresponding to a dimension of the operation result among the plurality of projection layers and a dimension processable by the at least one device If the dimension of the calculation result is changed based on a projection ray, the dimension-changed calculation result is transmitted to the at least one device, and the second response is information indicating that the calculation information cannot be processed, the calculation result may be input to the remaining layer to obtain a text corresponding to the user's voice, and transmit the obtained text to the at least one device.

In addition, the first neural network model fixes weight values of the remaining layers of the third neural network model, and is based on a plurality of sample user voices corresponding to the electronic device and a plurality of sample texts corresponding to the plurality of sample user voices. Thus, after the some layers are additionally learned, the model may be configured to include only the additionally learned partial layers.

Then, by the at least one device, inputting the operation result into a fourth neural network model stored in the at least one device to obtain a text corresponding to the user's voice and performing an operation corresponding to the obtained text The method further comprising: fixing the weight values of the partial layers in the fourth neural network model, a plurality of sample user voices corresponding to the at least one device, and a plurality of sample texts corresponding to the plurality of sample user voices After the remaining layers of the third neural network model are additionally trained based on , the model may be configured to include only the additionally trained remaining layers.

According to various embodiments of the present disclosure as described above, the electronic device may identify a device corresponding to the user's voice, and allow the identified device to perform an operation corresponding to the user's voice.

In addition, in the electronic device, voice recognition performance is improved as only some layers for voice recognition reinforcement in the speech recognition neural network model use the additionally trained neural network model, and the operation result of the neural network model is provided to a corresponding device, In the speech recognition neural network model, only the remaining layers for considering the device characteristics use the additionally trained neural network model, so that voice recognition optimized for the device can be performed.

1 is a diagram illustrating an electronic system according to an embodiment of the present disclosure.
2 is a block diagram illustrating a configuration of an electronic device according to an embodiment of the present disclosure.
3 is a diagram for explaining additional learning of a neural network model according to an embodiment of the present disclosure.
4 is a diagram for explaining an operation of identifying a device according to an embodiment of the present disclosure.
5 is a diagram for describing a projection layer according to an embodiment of the present disclosure.
6 is a diagram for explaining an operation of an electronic device according to a device type according to an embodiment of the present disclosure.
7 is a flowchart illustrating an operation of an electronic device according to a response of a device according to an embodiment of the present disclosure.
8 is a flowchart illustrating an operation of a device according to an embodiment of the present disclosure.
9 is a flowchart for describing a method of controlling an electronic device according to an embodiment of the present disclosure.

Hereinafter, the present disclosure will be described in detail with reference to the accompanying drawings.

Terms used in the embodiments of the present disclosure are selected as currently widely used general terms as possible while considering the functions in the present disclosure, which may vary depending on the intention or precedent of a person skilled in the art, the emergence of new technology, etc. . In addition, in a specific case, there is a term arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the description of the corresponding disclosure. Therefore, the terms used in the present disclosure should be defined based on the meaning of the term and the contents of the present disclosure, rather than the simple name of the term.

In this specification, expressions such as “have,” “may have,” “include,” or “may include” indicate the presence of a corresponding characteristic (eg, a numerical value, function, operation, or component such as a part). and does not exclude the presence of additional features.

The expression "at least one of A and/or B" is to be understood as indicating either "A" or "B" or "A and B".

As used herein, expressions such as "first," "second," "first," or "second," can modify various elements, regardless of order and/or importance, and refer to one element. It is used only to distinguish it from other components, and does not limit the components.

The singular expression includes the plural expression unless the context clearly dictates otherwise. In the present application, terms such as "comprises" or "consisting of" are intended to designate that the features, numbers, steps, operations, components, parts, or combinations thereof described in the specification exist, and are intended to indicate that one or more other It should be understood that this does not preclude the possibility of addition or presence of features or numbers, steps, operations, components, parts, or combinations thereof.

In this specification, the term user may refer to a person who uses an electronic device or a device (eg, an artificial intelligence electronic device) using the electronic device.

Hereinafter, various embodiments of the present disclosure will be described in more detail with reference to the accompanying drawings.

1 is a diagram illustrating an electronic system 1000 according to an embodiment of the present disclosure. As shown in FIG. 1 , the electronic system 1000 includes an electronic device 100 and a plurality of devices 200 - 1 to 200 - 3 .

The electronic device 100 is a device that receives a user's voice, and may be a smart phone, a tablet PC, a wearable device, or the like. In FIG. 1 , the electronic device 100 is illustrated as a smart phone for convenience of explanation, but the present invention is not limited thereto. Any device may be used as long as it is a device capable of controlling at least one device among them.

The electronic device 100 may store a neural network model in which some layers have been retrained in the voice recognition neural network model, and may process a user's voice using the stored neural network model. Here, the re-learning of some layers may be re-learning to enhance speech recognition performance through various speech samples after fixing the weight values of the remaining layers of the speech recognition neural network model.

For example, when the electronic device 100 is a smartphone as shown in FIG. 1 , various voice samples may be voices uttered in front of the smartphone. That is, the voice sample used for learning may be a text corresponding to the voice uttered in front of the smartphone.

The electronic device 100 may obtain an operation result by inputting the user's voice into some re-learned layers, and transmit the obtained operation result to at least one of the plurality of devices 200 - 1 to 200 - 3 .

In this case, the electronic device 100 may identify at least one device corresponding to the user's voice using the neural network model trained to identify the device corresponding to the voice, and transmit the obtained operation result to the identified device.

The plurality of devices 200-1 to 200-3 are devices that receive a calculation result from the electronic device 100 and perform a corresponding operation. In FIG. 1 , for convenience of explanation, the plurality of devices 200-1 to 200 -3) is shown as an air conditioner, a refrigerator, and a TV, respectively. However, the present invention is not limited thereto, and any device may be used as long as the plurality of devices 200 - 1 to 200 - 3 can receive an operation result from the electronic device 100 and perform a corresponding operation.

Each of the plurality of devices 200-1 to 200-3 may store a neural network model in which the remaining layers have been retrained in the speech recognition neural network model, and may perform an operation corresponding to the operation result by using the stored neural network model. Here, the re-learning of the remaining layers may be re-learning for performing speech recognition optimized for a device through a speech sample for each device after fixing the weight values of some layers of the speech recognition neural network model.

For example, in the case of a TV, the remaining layers may be re-learned using a voice sample and corresponding text used to control the TV such as volume up, volume down, channel up, channel down, and the like. Alternatively, in the case of an air conditioner, the remaining layers may be re-learned using a voice sample and corresponding text used to control the air conditioner, such as raising the temperature, lowering the temperature, or a sleep mode.

Each of the plurality of devices 200 - 1 to 200 - 3 may obtain a text corresponding to the user's voice by inputting the operation result to the remaining re-learned layer, and may perform an operation corresponding to the obtained text.

As described above, the electronic device 100 and the plurality of devices 200-1 to 200-3 each store a neural network model obtained by re-learning a specific layer of the voice recognition neural network model, and perform an operation corresponding to the user's voice. have.

In FIG. 1 , the plurality of devices 200 - 1 to 200 - 3 are illustrated as three, but the present invention is not limited thereto. For example, the plurality of devices 200 - 1 to 200 - 3 may be added or removed at will. Also, only one device instead of the plurality of devices 200 - 1 to 200 - 3 may be connected to the electronic device 100 .

In FIG. 1 , the electronic device 100 and the plurality of devices 200 - 1 to 200 - 3 are illustrated as being directly connected, but they may be connected through a device such as an access point.

2 is a block diagram illustrating a configuration of an electronic device 100 according to an embodiment of the present disclosure. As shown in FIG. 2 , the electronic device 100 includes a microphone 110 , a memory 120 , a communication interface 130 , and a processor 140 .

The microphone 110 is configured to receive a user's voice and convert it into an audio signal. The microphone 110 is electrically connected to the processor 140 , and may receive a user's voice under the control of the processor 140 . Here, the user's voice may include a voice generated by at least one of the electronic device 100 and other electronic devices surrounding the electronic device 100 and noise around the electronic device 100 .

For example, the microphone 110 may be formed integrally with an upper side, a front direction, a side direction, or the like of the electronic device 100 . Alternatively, the microphone 110 may be provided in a remote control separate from the electronic device 100 . In this case, the remote control may receive sound through the microphone 110 and provide the received sound to the electronic device 100 .

The microphone 110 includes a microphone for collecting analog sound, an amplifier circuit for amplifying the collected sound, an A/D conversion circuit for sampling the amplified sound and converting it into a digital signal, and a method for removing noise components from the converted digital signal. It may include various configurations such as filter circuits and the like.

The microphone 110 may include a plurality of sub-microphones. For example, the microphone 110 may include one sub-microphone at the front, rear, left, and right sides of the electronic device 100 . However, the present invention is not limited thereto, and the electronic device 100 may include only one microphone 110 .

Meanwhile, the microphone 110 may be implemented in the form of a sound sensor.

The memory 120 may refer to hardware that stores information such as data in an electrical or magnetic form so that the processor 140 can access it. To this end, the memory 120 may be implemented with at least one hardware selected from non-volatile memory, volatile memory, flash memory, hard disk drive (HDD) or solid state drive (SSD), RAM, ROM, etc. .

At least one instruction or module required for the operation of the electronic device 100 or the processor 140 may be stored in the memory 120 . Here, the instruction is a unit of code for instructing the operation of the electronic device 100 or the processor 140 , and may be written in machine language, which is a language that a computer can understand. A module may be a set of instructions that perform a specific task of a unit of work.

The memory 120 may store data that is information in units of bits or bytes that can represent characters, numbers, images, and the like. For example, the first neural network model and the second neural network model may be stored in the memory 120 . Here, the first neural network model is a model configured to include only some additionally trained layers after only some layers of the third neural network model trained to identify text from speech are additionally trained, and the second neural network model is a device corresponding to the voice. It can be a model that has been trained to identify.

The memory 120 may store a user voice processing module, information on a plurality of projection layers, information on a plurality of devices, and the remaining layers of the third neural network model.

The memory 120 is accessed by the processor 140 , and reading/writing/modification/deletion/update of instructions, modules, or data may be performed by the processor 140 .

The communication interface 130 is configured to communicate with various types of external devices according to various types of communication methods. For example, the electronic device 100 may communicate with the plurality of devices 200 - 1 to 200 - 3 through the communication interface 130 .

The communication interface 130 may include a Wi-Fi module, a Bluetooth module, an infrared communication module, and a wireless communication module. Here, each communication module may be implemented in the form of at least one hardware chip.

The Wi-Fi module and the Bluetooth module perform communication using a WiFi method and a Bluetooth method, respectively. In the case of using a Wi-Fi module or a Bluetooth module, various types of connection information such as an SSID and a session key are first transmitted and received, and then various types of information can be transmitted/received after communication connection using this. The infrared communication module communicates according to the infrared data association (IrDA) technology, which wirelessly transmits data in a short distance using infrared that is between visible light and millimeter waves.

In addition to the above-described communication methods, the wireless communication module includes Zigbee, 3rd Generation (3G), 3rd Generation Partnership Project (3GPP), Long Term Evolution (LTE), LTE Advanced (LTE-A), 4th Generation (4G), 5G It may include at least one communication chip that performs communication according to various wireless communication standards such as (5th Generation).

Alternatively, the communication interface 130 may include a wired communication interface such as HDMI, DP, Thunderbolt, USB, RGB, D-SUB, DVI, or the like.

In addition, the communication interface 130 may include at least one of a local area network (LAN) module, an Ethernet module, or a wired communication module for performing communication using a pair cable, a coaxial cable, or an optical fiber cable.

The processor 140 controls the overall operation of the electronic device 100 . Specifically, the processor 140 may be connected to each component of the electronic device 100 to control the overall operation of the electronic device 100 . For example, the processor 140 may be connected to components such as the microphone 110 , the memory 120 , and the communication interface 130 to control the operation of the electronic device 100 .

According to an embodiment, the processor 140 may be implemented as a digital signal processor (DSP), a microprocessor, or a time controller (TCON). However, the present invention is not limited thereto, and the central processing unit ( central processing unit (CPU)), micro controller unit (MCU), micro processing unit (MPU), controller, application processor (AP), or communication processor (CP), ARM processor In addition, the processor 140 may be implemented as a SoC (System on Chip) or LSI (large scale integration) in which a processing algorithm is embedded, or an FPGA ( Field programmable gate array) may be implemented.

When the user's voice is received through the microphone 110 , the processor 140 inputs the user's voice into the first neural network model to obtain an operation result, and inputs the operation result to the second neural network model to obtain at least one user voice corresponding to the user's voice. The device can be identified. Here, the first neural network model is a model configured to include only some additionally trained layers after only some layers of the third neural network model trained to identify text from speech are additionally trained, and the second neural network model is a device corresponding to the voice. A model trained to identify, and the third neural network model may be a neural network model trained to perform speech recognition. In particular, the first neural network model fixes the weight values of the remaining layers of the third neural network model, and based on the plurality of sample user voices corresponding to the electronic device 100 and the plurality of sample texts corresponding to the plurality of sample user voices, It may be a model configured to include only some additionally trained layers after some layers are additionally trained. Through this additional learning, the first neural network model may output a voice recognition result that further reflects the characteristics of the surrounding environment of the electronic device 100 as an operation result.

The processor 140 may control the communication interface 130 to transmit the operation result to at least one device. For example, the processor 140 inputs the calculation result into the second neural network model to obtain a score for each of a plurality of devices, and transmits the calculation result to a device having a score greater than or equal to a threshold value among the obtained scores. 140) can be controlled.

The processor 140 inputs the user's voice into the first neural network model in units of a preset time interval to obtain an operation result in units of a preset time, and inputs the operation results obtained in units of a preset time to the second neural network model to At least one device is identified in a preset time unit, and the communication interface 130 may be controlled to transmit an operation result obtained in a preset time unit to the at least one identified device in a preset time unit. For example, the processor 140 inputs the user's voice to the first neural network model in units of 25 ms to obtain an operation result in units of 25 ms, and the second neural network model inputs the operation results obtained in units of 25 ms to at least one in units of 25 ms The device may be identified, and the communication interface 130 may be controlled to transmit an operation result obtained in units of 25 ms to at least one device identified in units of 25 ms.

Meanwhile, the memory 120 further stores information on a plurality of devices and information on a plurality of projection layers, and the processor 140 is capable of being processed by at least one device based on the information on the plurality of devices. Identifies information on the second dimension, and when the first dimension and the second dimension of the calculation result are different, the calculation result is calculated as the second dimension based on the projection layer corresponding to the first dimension and the second dimension among the plurality of projection layers , and the communication interface 130 may be controlled to transmit the changed second-dimensional operation result to at least one device.

For example, the calculation result may be 1536 dimensions, but when the device to which the calculation result is transmitted has a 512 dimension, the processor 140 determines the dimension of the calculation result based on the 1536 × 512 projection layer stored in the memory 120 . It can be converted to 512 dimensions, and the result of the dimensionality-converted operation can be transmitted to the corresponding device.

Here, the information on the plurality of devices may be information received from the device when the electronic device 100 and the device are first connected. However, the present invention is not limited thereto, and information on a plurality of devices may not be stored in the memory 120 . In this case, the processor 140 may request and receive information on a dimension that can be processed by a device to transmit the operation result, and determine whether to use the projection layer based on the received information.

Meanwhile, the memory 120 further stores information on a plurality of devices and the remaining layers of the third neural network model, and the processor 140 includes a voice recognition function in at least one device based on the information on the plurality of devices. If it is identified as not, the communication interface 130 may be controlled to obtain a text corresponding to the user's voice by inputting the operation result to the remaining layers, and transmit the obtained text to at least one device. That is, when the device does not have the capability to process the calculation result, the processor 140 may obtain a text corresponding to the user's voice by performing the remaining processing on the calculation result, and may provide the obtained text to the device. In this case, the device may only perform an operation corresponding to the text.

Alternatively, the memory 120 further stores information on a plurality of projection layers and the remaining layers of the third neural network model, and the processor 140 transmits the calculation result to at least one device and then at least one device When the first response is received from the , the communication interface 130 may be controlled to transmit an operation result obtained thereafter to at least one device. For example, when an ok response is received from the at least one device after transmitting the calculation result to the at least one device, the processor 140 may transmit the subsequently obtained calculation result to the at least one device.

Alternatively, when a second response is received from the at least one device after transmitting the calculation result to the at least one device, the processor 140 may process the calculation result as one of a plurality of projection layers or input the calculation result to the remaining layers.

Specifically, if the second response is information on a dimension processable by at least one device, the processor 140 may store the dimension of the calculation result among the plurality of projection layers and the projection layer corresponding to the dimension processable by the at least one device. The dimension of the calculation result is changed based on the control, and the communication interface 130 is controlled to transmit the calculation result with the changed dimension to at least one device. The communication interface 130 may be controlled to obtain a text corresponding to the user's voice by inputting it into the layer and transmit the obtained text to at least one device.

Meanwhile, the at least one device may obtain a text corresponding to the user's voice by inputting the operation result to the fourth neural network model stored in the at least one device, and may perform an operation corresponding to the obtained text. Here, the fourth neural network model fixes the weight values of some layers, and the remainder of the third neural network model is based on a plurality of sample user voices corresponding to at least one device and a plurality of sample texts corresponding to a plurality of sample user voices. After the layer is additionally trained, it may be a model configured to include only the remaining additionally trained layers. Through this additional learning, the fourth neural network model may output the voice recognition result in consideration of the device's control command as text.

Meanwhile, functions related to artificial intelligence according to the present disclosure are operated through the processor 140 and the memory 120 .

The processor 140 may include one or a plurality of processors. In this case, the one or more processors may be a general-purpose processor such as a CPU, an AP, a digital signal processor (DSP), or the like, a graphics-only processor such as a GPU, a VPU (Vision Processing Unit), or an artificial intelligence-only processor such as an NPU.

One or a plurality of processors control to process input data according to a predefined operation rule or artificial intelligence model stored in the memory. Alternatively, when one or more processors are artificial intelligence-only processors, the artificial intelligence-only processor may be designed with a hardware structure specialized for processing a specific artificial intelligence model. The predefined action rule or artificial intelligence model is characterized in that it is created through learning.

Here, being made through learning means that a basic artificial intelligence model is learned using a plurality of learning data by a learning algorithm, so that a predefined action rule or artificial intelligence model set to perform a desired characteristic (or purpose) is created means burden. Such learning may be performed in the device itself on which artificial intelligence according to the present disclosure is performed, or may be performed through a separate server and/or system. Examples of the learning algorithm include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

The artificial intelligence model may be composed of a plurality of neural network layers. Each of the plurality of neural network layers has a plurality of weight values, and a neural network operation is performed through an operation between the operation result of a previous layer and the plurality of weights. The plurality of weights of the plurality of neural network layers may be optimized by the learning result of the artificial intelligence model. For example, a plurality of weights may be updated so that a loss value or a cost value obtained from the artificial intelligence model during the learning process is reduced or minimized.

The artificial neural network may include a deep neural network (DNN), for example, a Convolutional Neural Network (CNN), a Deep Neural Network (DNN), a Recurrent Neural Network (RNN), a Restricted Boltzmann Machine (RBM), There may be a Deep Belief Network (DBN), a Bidirectional Recurrent Deep Neural Network (BRDNN), or a Deep Q-Networks, but is not limited thereto.

As described above, voice recognition performance can be improved through additional training of the neural network model, and even when a device is added, voice recognition can be performed without an additional learning operation.

Hereinafter, the operation of the electronic device 100 will be described in more detail with reference to FIGS. 3 to 8 . 3 to 8 , individual embodiments will be described for convenience of description. However, the individual embodiments of FIGS. 3 to 8 may be embodied in any combination.

3 is a diagram for explaining additional learning of a neural network model according to an embodiment of the present disclosure.

The upper part of FIG. 3 shows an example of the third neural network model 310 trained to perform speech recognition.

The lower left of FIG. 3 shows an example of the first neural network model 320 configured to include only some additionally trained layers after only some layers near the input end of the third neural network model 310 are additionally learned. The first neural network model 320 fixes the weight values of the remaining layers near the output end of the third neural network model 310 , and corresponds to a plurality of sample user voices corresponding to the electronic device 100 and a plurality of sample user voices corresponding to the plurality of sample user voices. It may be a model configured to include only some additionally trained layers after some layers are additionally learned based on a plurality of sample texts.

For example, when the electronic device 100 is a TV, some layers of the third neural network model 310 are additionally trained through a sample user voice including a sound of the TV itself and a user voice and a corresponding text, and then additionally The first neural network model 320 may be configured with some layers of the learned third neural network model 310 . Through this additional learning, voice recognition performance can be maintained even when a user's voice including the sound of the TV itself is input. This effect can be equally applied to a smart phone, a refrigerator, a washing machine, etc. as well as a TV.

3 shows an example of the fourth neural network model 330 configured to include only the additionally trained remaining layers after only the remaining layers near the output end of the third neural network model 310 are additionally learned. The fourth neural network model 330 fixes the weight values of some layers near the input end of the third neural network model 310 , and a plurality of sample user voices corresponding to the device and a plurality of sample texts corresponding to the plurality of sample user voices It may be a model configured to include only the additionally trained remaining layers after the remaining layers are additionally learned based on .

For example, when the device is a TV, the remaining layers may be re-learned using a voice sample and corresponding text used to control the TV such as volume up, volume down, channel up, channel down, and the like. Alternatively, when the device is an air conditioner, the remaining layers may be re-learned using a voice sample and corresponding text used to control the air conditioner, such as raising the temperature, lowering the temperature, or a sleep mode. Through this additional learning, the possibility that the user's voice is output as text corresponding to each device is improved.

4 is a diagram for explaining an operation of identifying a device according to an embodiment of the present disclosure.

Processor 140 As shown in FIG. 4 , the user's voice is input to the first neural network model in units of 25 ms to obtain calculation results in units of 25 ms, and the operation results obtained in units of 25 ms are input by the second neural network model to the model for 25 ms At least one device may be identified in units of units, and the communication interface 130 may be controlled to transmit an operation result obtained in units of 25 ms to at least one device identified in units of 25 ms.

That is, the processor 140 inputs the user voice of section 410 into the first neural network model to obtain an operation result corresponding to section 410, and inputs the operation result corresponding to section 410 into the second neural network model to correspond to section 410 A first device, a second device, and a third device to be used may be identified. Here, the processor 140 may obtain a score for each of the plurality of devices, and identify the first device, the second device, and the third device having a score equal to or greater than a threshold value. The processor 140 may transmit the operation result corresponding to the 410 section to the first device, the second device, and the third device.

Then, the processor 140 inputs the user voice of section 420 into the first neural network model to obtain a calculation result corresponding to section 420, and inputs the calculation result corresponding to section 420 into the second neural network model to correspond to section 420 A first device and a third device to be used may be identified. The processor 140 may transmit an operation result corresponding to section 420 to the first device and the third device.

Then, the processor 140 inputs the user voice of the 430 section into the first neural network model to obtain an operation result corresponding to the 430 section, and inputs the operation result corresponding to the 430 section into the second neural network model to correspond to the 430 section A third device to be used can be identified. The processor 140 may transmit an operation result corresponding to section 430 to the third device.

In the above manner, the processor 140 may identify a device to which the operation result is to be transmitted. That is, the device to which the calculation result is transmitted may be a device corresponding to the user's voice. In this operation, since each section is very short, the final transmission device may be determined before text is derived from the device. In FIG. 4 , the first device receives the operation result for 50 ms, but the text cannot be derived because the additional operation result is not received. This point is also the same in the second device. Finally, only the third device may continuously receive the operation result to obtain the text and perform an operation corresponding to the text.

On the other hand, the third device processes the received operation result in 25 ms in real time, so that the text acquisition speed may be improved compared to the prior art. For example, when the conventional apparatus does not provide a calculation result in a preset time period and identifies a device corresponding to the user's voice based on the entire user voice time period, the user's voice is transmitted to the corresponding device only after processing the entire user's voice. will send In this case, the delay time may be a time for processing the entire user's voice. On the other hand, according to the present disclosure, since the device receives and processes the operation result every 25 ms, the delay time may be about 25 ms rather than the time for processing the entire user voice. Accordingly, the speech recognition speed can be improved by transmitting the operation result in units of sections.

Meanwhile, although 25 ms is illustrated as a preset time interval in FIG. 4 , the present invention is not limited thereto, and various time intervals may be set according to hardware specifications, a time for processing an operation result of each time interval, and the like.

5 is a diagram for describing a projection layer according to an embodiment of the present disclosure.

As shown in FIG. 5 , when the calculation result, which is the output of the first neural network model, is 1536 dimensions, and the processingable dimension of the device is 512 dimensions, the processor 140 calculates the calculation result through a 1536 × 512 projection layer. It can be changed to a dimension, and the changed operation result can be provided to the device.

However, the present invention is not limited thereto, and when it is identified that the device stores the 1536×512 projection layer, the processor 140 may provide a 1536-dimensional calculation result to the device, and the device may change the dimension of the calculation result.

Meanwhile, although FIG. 5 shows a projection layer of 1536×512, various types of projection layers may exist.

6 is a diagram for explaining an operation of the electronic device 100 according to a device type according to an embodiment of the present disclosure.

The processor 140 of the electronic device 100 may obtain an operation result (Encoder state) by inputting the user's voice into the first neural network model (Encoder). Then, the processor 140 may identify at least one device corresponding to the user's voice by inputting the operation result into the second neural network model (Domain Classifier). Here, the first neural network model is a model configured to include only some additionally trained layers after only some layers of the third neural network model trained to identify text from speech are additionally trained, and the second neural network model is a device corresponding to the voice. It can be a model that has been trained to identify.

If the first device 200 - 1 is a device corresponding to the user's voice or a device that cannot process the calculation result, the processor 140 inputs the calculation result to the remaining layers of the third neural network model to provide text corresponding to the user's voice. may be obtained (Decoder), and the text may be provided to the first device 200 - 1 .

The second device 200 - 2 and the third device 200 - 3 are devices capable of processing the calculation result, and the processor 140 outputs the calculation result to the second device 200 based on the output of the second neural network model. -2) or the third device 200-3. In FIG. 6 , the third device 200 - 3 is a device corresponding to the user's voice, and the processor 140 may transmit the operation result to the third device 200 - 3 .

The third device 200 - 3 may obtain a text corresponding to the user's voice by inputting the operation result into the fourth neural network model, and may perform an operation corresponding to the obtained text. Here, the fourth neural network model fixes the weight values of some layers, and the remainder of the third neural network model is based on a plurality of sample user voices corresponding to at least one device and a plurality of sample texts corresponding to a plurality of sample user voices. After the layer is additionally trained, it may be a model configured to include only the remaining additionally trained layers.

7 is a flowchart illustrating an operation of the electronic device 100 according to a response of a device according to an embodiment of the present disclosure.

First, the processor 140 identifies whether the device probability (score) exceeds a threshold value (S710). The processor 140 may transmit the operation result when the device probability exceeds the threshold value (S720), and may not perform an additional operation if the device probability does not exceed the threshold value.

The processor 140 may transmit the operation result and receive a response from the device (S730).

When the first response is received, the processor 140 may maintain the operation result transmission operation (S740). For example, when an ok response is received, the processor 140 may maintain the operation result transmission operation.

Alternatively, when the second response is received, the processor 140 may decode the operation result and transmit the decoding result (S750). For example, when a response indicating that the device cannot process the calculation result is received, the processor 140 may obtain text from the calculation result and transmit the obtained text to the device.

Alternatively, when the third response is received, the processor 140 may identify whether a projection layer exists (S760). For example, the processor 140 may receive information from the device that the processing result is capable of being processed, but the dimension that can be processed is different from the dimension of the operation result. In this case, if the projection layer does not exist, the processor 140 may decode the operation result and transmit the decoding result (S750). Alternatively, if the projection layer exists, the processor 140 may change the dimension by projecting the operation result (S770), and transmit the projected operation result to the device (S780).

8 is a flowchart illustrating an operation of a device according to an embodiment of the present disclosure.

First, the device may receive an operation result from the electronic device 100 ( S810 ). Then, the device may identify whether a decoder exists (S820). That is, if the device can process the operation result, it identifies whether a processable dimension and a dimension of the operation result are the same (S830), and if the operation result cannot be processed, transmits no to the electronic device 100 It can be (S870).

If the dimensions are the same, the device may transmit ok to the electronic device 100 (S850), and if they are not the same, the device may identify whether a projection layer exists (S840).

If the projection layer exists, the device may transmit ok to the electronic device 100 (S850), and if the projection layer does not exist, it may transmit dimension information that the device can process (S860).

If the dimensions are the same, the device uses the calculation result received from the electronic device 100 as it is, and if the dimensions are not the same and a projection layer exists, the dimension of the calculation result received from the electronic device 100 is changed based on the projection layer can be used by

9 is a flowchart for describing a method of controlling an electronic device according to an embodiment of the present disclosure.

First, when the user's voice is received, the user's voice is input to the first neural network model to obtain a calculation result (S910). Then, at least one device corresponding to the user's voice is identified by inputting the calculation result into the second neural network model ( S920 ). Then, the calculation result is transmitted to at least one device (S930). Here, the first neural network model is a model configured to include only some additionally trained layers after only some layers of the third neural network model trained to identify text from speech are additionally trained, and the second neural network model is a device corresponding to the voice. It can be a model that has been trained to identify.

Here, the obtaining step (S910) is to input the user's voice into the first neural network model in a predetermined time interval unit to obtain a calculation result in a predetermined time unit unit, and the identifying step (S920) is to obtain a predetermined time unit In the step of inputting the calculation result into the second neural network model to identify at least one device in a preset time unit, and transmitting (S930), the operation result obtained in the preset time unit is input to the at least one device identified in the preset time unit. can be transferred to the device.

In addition, the step of identifying information on a second dimension that can be processed by at least one device based on the information on the plurality of devices; The method further includes the step of changing the calculation result to the second dimension based on the dimension and the projection layer corresponding to the second dimension, and transmitting (S930) may transmit the changed second-dimensional calculation result to at least one device. have.

On the other hand, when it is identified that at least one device does not have a speech recognition function based on the information on the plurality of devices, inputting the calculation result into the remaining layers of the third neural network model to obtain text corresponding to the user's voice It further includes, and the transmitting ( S930 ) may transmit the obtained text to at least one device.

In addition, the step of identifying (S920) obtains a score for each of the plurality of devices by inputting the calculation result into the second neural network model, and the step of transmitting (S930) includes the calculation result having a score greater than or equal to a threshold value among the obtained scores can be sent to the device.

On the other hand, when a first response is received from the at least one device after transmitting the calculation result to the at least one device, after transmitting the obtained calculation result to the at least one device, and after transmitting the calculation result to the at least one device When the second response is received from the at least one device, the method may further include processing the calculation result as one of a plurality of projection layers or inputting the operation result into the remaining layers of the third neural network model.

Here, when the second response is information on a dimension processable by at least one device, the inputting step is based on a dimension of a calculation result among a plurality of projection layers and a projection ray corresponding to a dimension processable by at least one device. If the dimension of the computation result is changed, the dimension-changed computation result is transmitted to at least one device, and the second response is information indicating that the computation information cannot be processed, the computation result is input to the remaining layers and text corresponding to the user's voice may be obtained, and the obtained text may be transmitted to at least one device.

Meanwhile, in the first neural network model, the weight values of the remaining layers of the third neural network model are fixed, and some layers are formed based on a plurality of sample user voices corresponding to the electronic device and a plurality of sample texts corresponding to the plurality of sample user voices. After additional training, it may be a model configured to include only some additionally trained layers.

The method further includes, by at least one device, inputting a calculation result into a fourth neural network model stored in the at least one device to obtain text corresponding to the user's voice and performing an operation corresponding to the obtained text. wherein the fourth neural network model fixes the weight values of some layers, and based on a plurality of sample user voices corresponding to at least one device and a plurality of sample texts corresponding to a plurality of sample user voices, After the remaining layers are additionally trained, it may be a model configured to include only the additionally trained remaining layers.

According to various embodiments of the present disclosure as described above, the electronic device may identify a device corresponding to the user's voice, and allow the identified device to perform an operation corresponding to the user's voice.

In addition, in the electronic device, voice recognition performance is improved as only some layers for voice recognition reinforcement in the speech recognition neural network model use the additionally trained neural network model, and the operation result of the neural network model is provided to a corresponding device, In the speech recognition neural network model, only the remaining layers for considering the device characteristics use the additionally trained neural network model, so that voice recognition optimized for the device can be performed.

Meanwhile, in the above description, for convenience of description, it has been described that a plurality of devices receive an operation result from the electronic device and perform an operation corresponding to the received operation result. That is, although it has been described above that each of the plurality of devices processes the calculation result using the fourth neural network model, the present invention is not limited thereto. For example, each of the plurality of devices may store a first neural network model and a second neural network model, receive a user's voice, perform an operation together with the electronic device, and then transmit the operation result to the electronic device or at least one device.

Meanwhile, according to a temporary example of the present disclosure, the various embodiments described above may be implemented as software including instructions stored in a machine-readable storage media readable by a machine (eg, a computer). can A device is a device capable of calling a stored command from a storage medium and operating according to the called command, and may include an electronic device (eg, the electronic device A) according to the disclosed embodiments. When the instruction is executed by the processor, the processor may perform a function corresponding to the instruction by using other components directly or under the control of the processor. Instructions may include code generated or executed by a compiler or interpreter. The device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' means that the storage medium does not include a signal and is tangible, and does not distinguish that data is semi-permanently or temporarily stored in the storage medium.

In addition, according to an embodiment of the present disclosure, the method according to the various embodiments described above may be provided by being included in a computer program product. Computer program products may be traded between sellers and buyers as commodities. The computer program product may be distributed in the form of a machine-readable storage medium (eg, compact disc read only memory (CD-ROM)) or online through an application store (eg, Play Store™). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a storage medium such as a memory of a server of a manufacturer, a server of an application store, or a relay server.

In addition, according to an embodiment of the present disclosure, the various embodiments described above are stored in a recording medium readable by a computer or a similar device using software, hardware, or a combination thereof. can be implemented in In some cases, the embodiments described herein may be implemented by the processor itself. According to the software implementation, embodiments such as the procedures and functions described in this specification may be implemented as separate software modules. Each of the software modules may perform one or more functions and operations described herein.

Meanwhile, computer instructions for performing the processing operation of the device according to the above-described various embodiments may be stored in a non-transitory computer-readable medium. When the computer instructions stored in the non-transitory computer-readable medium are executed by the processor of the specific device, the specific device performs the processing operation in the device according to the various embodiments described above. The non-transitory computer-readable medium refers to a medium that stores data semi-permanently, not a medium that stores data for a short moment, such as a register, cache, memory, etc., and can be read by a device. Specific examples of the non-transitory computer-readable medium may include a CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM, and the like.

In addition, each of the components (eg, a module or a program) according to the above-described various embodiments may be composed of a single or a plurality of entities, and some sub-components of the aforementioned sub-components may be omitted, or other sub-components may be omitted. Components may be further included in various embodiments. Alternatively or additionally, some components (eg, a module or a program) may be integrated into a single entity to perform the same or similar functions performed by each corresponding component prior to integration. According to various embodiments, operations performed by a module, program, or other component are executed sequentially, parallel, iteratively, or heuristically, or at least some operations are executed in a different order, are omitted, or other operations are added. can be

In the above, preferred embodiments of the present disclosure have been illustrated and described, but the present disclosure is not limited to the specific embodiments described above, and it is common in the technical field pertaining to the present disclosure without departing from the gist of the present disclosure as claimed in the claims. Various modifications may be made by those having the knowledge of

1000: electronic system 100: electronic device
110: microphone 120: memory
130: communication interface 140: processor
200-1 ~ 200-3: Multiple devices

Claims (20)

In an electronic device,
MIC;
a memory in which the first neural network model and the second neural network model are stored;
communication interface; and
a processor connected to the microphone, the memory, and the communication interface to control the electronic device;
The processor is
When a user voice is received through the microphone, the user voice is input to the first neural network model to obtain an operation result, and the operation result is input to the second neural network model to select at least one device corresponding to the user voice identify,
controlling the communication interface to transmit the operation result to the at least one device;
The first neural network model is
After only some layers of the third neural network model trained to identify text from speech are additionally trained, it is a model configured to include only some of the additionally trained layers,
The second neural network model is
An electronic device, which is a model trained to identify a device corresponding to a voice. According to claim 1,
The processor is
inputting the user voice into the first neural network model in units of a preset time interval to obtain the operation result in units of the preset time,
inputting the operation result obtained in the preset time unit into the second neural network model to identify the at least one device in the preset time unit;
and controlling the communication interface to transmit the operation result obtained in the preset time unit to the at least one device identified in the preset time unit. According to claim 1,
The memory further stores information on a plurality of devices and information on a plurality of projection layers,
The processor is
Identifies information on a second dimension that can be processed by the at least one device based on the information on the plurality of devices,
When the first dimension and the second dimension of the operation result are different from each other, the operation result is changed to the second dimension based on a projection layer corresponding to the first dimension and the second dimension among the plurality of projection layers, and ,
and controlling the communication interface to transmit the changed second-dimensional calculation result to the at least one device. According to claim 1,
The memory further stores information on a plurality of devices and the remaining layers of the third neural network model,
The processor is
When it is identified that the at least one device does not have a voice recognition function based on the information on the plurality of devices, the operation result is input to the remaining layer to obtain a text corresponding to the user's voice,
and controlling the communication interface to transmit the obtained text to the at least one device. According to claim 1,
The processor is
Input the calculation result to the second neural network model to obtain a score for each of a plurality of devices,
and controlling the communication interface to transmit the operation result to a device having a score equal to or greater than a threshold among the obtained scores. According to claim 1,
The memory further stores information on a plurality of projection layers and the remaining layers of the third neural network model,
The processor is
When a first response is received from the at least one device after transmitting the calculation result to the at least one device, controlling the communication interface to transmit the calculation result obtained thereafter to the at least one device,
When a second response is received from the at least one device after transmitting the calculation result to the at least one device, the calculation result is processed as one of the plurality of projection layers or input to the remaining layers. 7. The method of claim 6,
The processor is
If the second response is information on a dimension processable by the at least one device, the second response is based on a dimension of the operation result among the plurality of projection layers and a projection ray corresponding to a dimension processable by the at least one device. changing the dimension of the calculation result, and controlling the communication interface to transmit the calculation result of which the dimension is changed to the at least one device,
If the second response is information indicating that the operation information cannot be processed, input the operation result to the remaining layer to obtain a text corresponding to the user's voice, and transmit the obtained text to the at least one device An electronic device for controlling the communication interface. According to claim 1,
The first neural network model is
The weight values of the remaining layers of the third neural network model are fixed, and the partial layers are additionally learned based on a plurality of sample user voices corresponding to the electronic device and a plurality of sample texts corresponding to the plurality of sample user voices. Afterwards, the electronic device is a model configured to include only some of the additionally learned layers. According to claim 1,
the at least one device,
inputting the calculation result into a fourth neural network model stored in the at least one device to obtain a text corresponding to the user's voice;
performing an operation corresponding to the obtained text,
The fourth neural network model is
The weight values of the partial layers are fixed, and the remaining layers of the third neural network model are added based on a plurality of sample user voices corresponding to the at least one device and a plurality of sample texts corresponding to the plurality of sample user voices. After being trained, the electronic device is a model configured to include only the additionally trained remaining layers. In an electronic system,
When the user's voice is received, the user's voice is input to a first neural network model to obtain an operation result, and the operation result is input to a second neural network model to identify at least one device corresponding to the user's voice, and the operation result an electronic device that transmits to the at least one device; and
At least one device for obtaining a text corresponding to the user's voice by inputting the operation result into a fourth neural network model and performing an operation corresponding to the obtained text;
The first neural network model is
After only some layers of the third neural network model trained to identify text from speech are additionally trained, it is a model configured to include only some of the additionally trained layers,
The second neural network model is
A model trained to identify a device corresponding to a voice,
The fourth neural network model is
The electronic system of claim 1, wherein after only the remaining layers of the third neural network model are additionally trained, the model is configured to include only the additionally trained remaining layers. 11. The method of claim 10,
the at least one device,
If the first dimension of the operation result is different from the second dimension of the input of the fourth neural network model, it is identified whether a projection layer corresponding to the first dimension exists,
When the projection layer exists, a first signal is transmitted to the electronic device,
When the projection layer does not exist, a second signal including information on the second dimension is transmitted to the electronic device. 12. The method of claim 11,
The electronic device is
When the first signal is received, the operation of transmitting the operation result to the at least one device is maintained,
When the second signal is received, the operation result is projected into the second dimension and the projected operation result is transmitted to the at least one device, or the operation result is decoded and the decoded operation result is transmitted to the at least one device. An electronic system that transmits to a device. A method for controlling an electronic device, comprising:
when the user's voice is received, inputting the user's voice into a first neural network model to obtain an operation result;
inputting the calculation result into a second neural network model to identify at least one device corresponding to the user's voice; and
Including; transmitting the operation result to the at least one device;
The first neural network model is
After only some layers of the third neural network model trained to identify text from speech are additionally trained, it is a model configured to include only some of the additionally trained layers,
The second neural network model is
A control method, which is a model trained to identify a device corresponding to a voice. 14. The method of claim 13,
The obtaining step is
inputting the user voice into the first neural network model in units of a preset time interval to obtain the operation result in units of the preset time,
The identifying step is
inputting the operation result obtained in the preset time unit into the second neural network model to identify the at least one device in the preset time unit;
The transmitting step is
The control method of transmitting the operation result obtained in the preset time unit to the at least one device identified in the preset time unit. 14. The method of claim 13,
identifying information on a second dimension that can be processed by the at least one device based on information about a plurality of devices;
When the first dimension and the second dimension of the operation result are different from each other, changing the operation result to the second dimension based on a projection layer corresponding to the first dimension and the second dimension among a plurality of projection layers further including;
The transmitting step is
and transmitting the changed second-dimensional calculation result to the at least one device. 14. The method of claim 13,
When it is identified that the at least one device is not equipped with a speech recognition function based on information on a plurality of devices, the operation result is input to the remaining layers of the third neural network model to obtain text corresponding to the user's voice further comprising;
The transmitting step is
and transmitting the obtained text to the at least one device. 14. The method of claim 13,
The identifying step is
Input the calculation result to the second neural network model to obtain a score for each of a plurality of devices,
The transmitting step is
and transmitting the calculation result to a device having a score equal to or greater than a threshold among the obtained scores. 14. The method of claim 13,
After transmitting the operation result to the at least one device, when a first response is received from the at least one device, a subsequent calculation result is transmitted to the at least one device, and the calculation result is transmitted to the at least one device When a second response is received from the at least one device after transmitting to , processing the calculation result as one of a plurality of projection layers or inputting the operation result into the remaining layers of the third neural network model. 19. The method of claim 18,
The input step is
If the second response is information on a dimension processable by the at least one device, the second response is based on a dimension of the operation result among the plurality of projection layers and a projection ray corresponding to a dimension processable by the at least one device. If the dimension of the calculation result is changed, the calculation result of which the dimension has been changed is transmitted to the at least one device, and the second response is information indicating that the calculation information cannot be processed, the calculation result is input to the remaining layer, A control method of acquiring a text corresponding to the user's voice and transmitting the acquired text to the at least one device. 14. The method of claim 13,
The first neural network model is
The weight values of the remaining layers of the third neural network model are fixed, and the partial layers are additionally learned based on a plurality of sample user voices corresponding to the electronic device and a plurality of sample texts corresponding to the plurality of sample user voices. Then, the control method, which is a model configured to include only some of the additionally learned layers.

Images