KR20210053150A - The hub device, multi device system comprising the hub device and a plurality of devices and method operating the same - Google Patents

Abstract

The present invention relates to a hub device, a multidevice system including the hub device and a plurality of devices and an operating method thereof. An embodiment of the present invention provides a hub device and a multidevice system including the hub device. The hub device stores, in an internal memory thereof, at least a portion of a voice assistant model for receiving a user voice input and automatically determining a device to perform an operation according to the intention of a user based on the received voice input and a voice assistant model for providing information necessary to perform a service according to the determined device.

Description

A multi-device system including a hub device, a hub device, and a plurality of devices, and its operation method {THE HUB DEVICE, MULTI DEVICE SYSTEM COMPRISING THE HUB DEVICE AND A PLURALITY OF DEVICES AND METHOD OPERATING THE SAME}

The present disclosure provides an artificial intelligence model for determining an operation performing device to perform an operation according to a user's intention included in a voice input received from a user in a multi-device environment including a hub device and a plurality of devices, and a device performing the determined operation. The present invention relates to a hub device including an artificial intelligence model for controlling an operation to be performed, a multi-device system, and an operation method thereof.

With the development of multimedia technology and network technology, users can receive various services using devices. In particular, with the development of voice recognition technology, a user can input a voice (eg, speech) to a device and receive a response message according to the voice input through a service providing agent.

However, in a multi-device system such as a home network environment including a plurality of devices, when a user wants to receive a service through a device other than a client device that interacts through voice input, the device to provide the service There is a hassle of having to choose directly. In particular, since the types of services that can be provided by each of a plurality of devices are different, there is a demand for a technology capable of effectively providing a service by grasping an intention included in a user's voice input.

When grasping the intention included in the user's voice input, artificial intelligence (AI) technology may be used, and rule-based natural language understanding technology (NLU) may be used. When a user's voice input is received through a hub device, the hub device cannot directly select a device to provide a service through voice input and must control the device using a separate voice assistant service providing server. From the user's point of view, there is a problem in that network usage costs are consumed, and response speed is slowed down because the voice assistant service providing server is used. In order to solve the above problem, the hub device may consider an on device model method in which a voice assistant model that controls a device to provide a service based on a voice input is stored in an internal memory. However, in implementing the on-device model method, it may not be possible to store the voice assistant model depending on the internal memory capacity of the hub device, the remaining capacity of the RAM (Random Access Memory), or the processing speed.

The present disclosure relates to a multi-device system including a hub device, a hub device, and a plurality of devices, and a method of operating the same, and more specifically, to receive a user's voice input, and according to the user's intention based on the received voice input. A hub device, a multi-device system, which stores at least some of a voice assistant model that automatically determines a device to perform an operation and a voice assistant model that provides necessary information to perform a service according to the determined device in the internal memory of the hub device, And an operation method thereof.

In order to solve the above-described technical problem, an embodiment of the present disclosure includes the steps of receiving information on a new device connected to a hub device, and in response to receiving the information on the new device, the voice assistant server , Requesting an update of a device determination model stored in the hub device, receiving and storing the updated device determination model from the voice assistant server, a function determination model corresponding to the new device in the voice assistant server Requesting information of, receiving information on a function determination model corresponding to the new device from the voice assistant server, and determining whether to store the function determination model in the hub device based on the received information And storing the function determination model corresponding to the new device when it is determined to store the function determination model in the hub device. do.

For example, when it is determined not to store the function determination model in the hub device, the method may further include receiving access information for a function determination model corresponding to the new device.

For example, the access information may include identification information, location information, IP address information (IP address), MAC address (MAC address) of the voice assistant server in which a function determination model corresponding to the new device is stored, and in the voice assistant server. It may include at least one of API information (Application Programming Interface) for accessing the function determination model, a language used for the function determination model, and identification information of the new device.

For example, determining whether to store the function determination model in the hub device may be determined based on information on a resource state of the hub device and a function determination model corresponding to the new device. .

For example, the hub device may be determined by the voice assistant server based on resource status information for each of a plurality of devices previously registered according to a user account.

For example, the method includes selecting at least one hub device candidate from among a plurality of devices previously registered in the IoT server according to a user account logged in to the hub device, at least one hub Determining any one of the at least one hub device candidate based on the usage history information and performance information of each of the device candidates, and changing the hub device by replacing the hub device with the determined device. have.

For example, the selecting of the at least one hub device candidate may include the at least one of the plurality of devices based on at least one of whether the plurality of devices are always powered on, computing power, and power consumption. Hub device candidates can be selected.

For example, the determining of any one of the at least one hub device candidate may include analyzing a usage history log database stored in the hub device, thereby Acquiring information on the frequency of use, by analyzing a performance history log database stored in the hub device, obtaining information on the utterance processing time of each user of the hub device and at least one hub device candidate And determining a device to replace the hub device, based on frequency information and speech processing time information of each of the hub device and the at least one hub device candidate.

For example, selecting at least one hub device candidate from among the plurality of devices may be performed after receiving information on a new device connected to the hub device.

For example, the method includes receiving a user input for selecting any one device among a plurality of devices pre-registered in the IoT server according to a user account logged in to the hub device, and based on the user input. It may further include determining the selected device as the hub device.

In order to solve the above technical problem, an embodiment of the present disclosure is a communication interface for performing data communication with at least one of a plurality of devices, a voice assistant server, and an IoT server, and a program including one or more instructions. And a processor that executes one or more instructions of a program stored in the memory, and the processor receives information on a new device connected to the hub device, and receives information on the new device. , Requesting the voice assistant server to update a device determination model previously stored in the memory, controlling the communication interface to receive the updated device determination model from the voice assistant server, and receiving the updated device determination model The communication to store information on a function determination model corresponding to the new device from the voice assistant server, and to request information on a function determination model corresponding to the new device from the voice assistant server. When it is determined whether to control the interface and to store the function determination model in the hub device based on the received information, and when it is determined to store the function determination model in the hub device, a function corresponding to the new device A hub device is provided for storing a judgment model in the memory.

For example, when it is determined not to store the function determination model in the hub device, the processor may control the communication interface to receive access information for a function determination model corresponding to the new device. .

For example, the access information may include identification information, location information, IP address information (IP address), MAC address (MAC address) of the voice assistant server in which a function determination model corresponding to the new device is stored, and in the voice assistant server. It may include at least one of API information (Application Programming Interface) for accessing the function determination model, a language used for the function determination model, and identification information of the new device.

For example, the processor may determine whether to store the function determination model in the memory based on the resource state of the hub device and information on a function determination model corresponding to the new device.

For example, the hub device may be determined by the voice assistant server based on resource status information for each of a plurality of devices previously registered according to a user account.

For example, the processor selects at least one hub device candidate from among a plurality of devices pre-registered in the IoT server according to a user account logged in to the hub device, and at least one hub device The hub device may be changed by determining any one of at least one hub device candidate based on usage history information and performance information of each of the candidates, and replacing the hub device with the determined device.

For example, the processor may select at least one hub device candidate from among a plurality of devices based on at least one of whether power is always supplied to each of the plurality of devices, computing power, and power consumption.

For example, the hub device may include a usage history log database that stores information on a frequency of use of each of the hub device and at least one hub device candidate, and an utterance processing time of each user of the hub device and at least one hub device candidate. It may further include a performance history log database for storing information about the.

For example, the processor obtains information on the frequency of use of each of the hub device and at least one hub device candidate by analyzing the usage history log database, and analyzes the performance history log database, so that the hub device and at least one hub device Information about the speech processing time of each of the candidate users may be obtained, and a device to replace the hub device may be determined based on the frequency information and the speech processing time information of the hub device and each of the at least one hub device candidate.

For example, the hub device further includes a voice input unit for receiving a user's voice input for selecting any one of a plurality of devices pre-registered in the IoT server according to a user account logged in to the hub device, the The processor converts the user's voice input received from the voice input unit into text by performing ASR (Automatic Speech Recognition), and interprets the converted text using the NLU model (Natural Language Understanding model) to input the user's voice. According to this, the selected device may be determined as a hub device.

In order to solve the above technical problem, according to an embodiment of the present disclosure, as a hub device receives information on a new device, requesting an update of a device determination model stored in the hub device from a voice assistant server, When the voice assistant server receives an update request for the device determination model, performing an update on the device determination model stored in the hub device, the hub device receiving the updated device determination model from the voice assistant server. Receiving and storing, the hub device requesting information of a function determination model corresponding to the new device from the voice assistant server, the voice assistant server obtaining information of a function determination model corresponding to the new device Transmitting to the hub device, and the hub device receiving information on a function determination model corresponding to the new device from the voice assistant server, and storing the function determination model in the hub device based on the received information It provides a method of storing a voice assistant model in which a voice assistant server and a hub device control a device, comprising the step of determining whether to do so.

For example, the method may further include storing a function determination model corresponding to the new device when it is determined that the hub device stores the function determination model in the hub device.

For example, the method, when it is determined that the hub device does not store the function determination model in the hub device, requests and receives access information for the function determination model corresponding to the new device to the voice assistant server It may further include the step of.

For example, the access information may include identification information, location information, IP address information (IP address), MAC address (MAC address) of the voice assistant server in which a function determination model corresponding to the new device is stored, and in the voice assistant server. It may include at least one of API information (Application Programming Interface) for accessing the function determination model, a language used for the function determination model, and identification information of the new device.

For example, the hub device may be determined by the voice assistant server based on resource status information for each of a plurality of devices previously registered according to a user account.

In order to solve the above technical problem, an embodiment of the present disclosure provides a system including a hub device and a voice assistant server. The hub device includes a communication interface for performing data communication with the voice assistant server, a memory for storing a program including one or more instructions, and a processor for executing one or more instructions of a program stored in the memory, , The processor receives information on the new device, controls the communication interface to request an update of a device determination model previously stored in the memory from the voice assistant server, and determines the updated device from the voice assistant server Receiving a model, storing the received updated device determination model in the memory, requesting information on a function determination model corresponding to the new device from the voice assistant server, and responding to the new device from the voice assistant server The communication interface is controlled to receive information on a function determination model to be performed, and based on the received information, it is determined whether to store the function determination model in the hub device, and the voice assistant server Upon receiving an update request for a device determination model, an update to a device determination model previously stored in the hub device may be performed, and information on a function determination model corresponding to the new device may be obtained and transmitted to the hub device.

For example, when it is determined to store the function determination model in the hub device, the processor may store a function determination model corresponding to the new device in the memory.

For example, when it is determined not to store the function determination model in the hub device, the processor requests access information for a function determination model corresponding to the new device to the voice assistant server, and the access information The communication interface can be controlled to receive from a voice assistant server.

For example, the access information may include identification information, location information, IP address information (IP address), MAC address (MAC address) of the voice assistant server in which a function determination model corresponding to the new device is stored, and in the voice assistant server. It may include at least one of API information (Application Programming Interface) for accessing the function determination model, a language used for the function determination model, and identification information of the new device.

For example, the hub device may be determined by the voice assistant server based on resource status information for each of a plurality of devices previously registered according to a user account.

The present disclosure may be easily understood by a combination of the following detailed description and accompanying drawings, and reference numerals denote structural elements.
1 is a block diagram illustrating some components of a multi-device system including a hub device, a voice assistant server, an IoT server, and a plurality of devices according to an embodiment of the present disclosure.
2 is a block diagram illustrating components of a hub device according to an embodiment of the present disclosure.
3 is a block diagram illustrating components of a voice assistant server according to an embodiment of the present disclosure.
4 is a block diagram illustrating components of an IoT server according to an embodiment of the present disclosure.
5 is a block diagram illustrating some components of a plurality of devices according to an embodiment of the present disclosure.
6 is a flowchart illustrating a method of receiving and storing at least a part of a voice assistant model from a voice assistant server by a hub device of the present disclosure.
7 is a flowchart illustrating a method of determining whether to store a function determination model based on information on a function determination model received from a voice assistant server by the hub device of the present disclosure.
8 is a flowchart illustrating a method of controlling an operation of a new device based on a user's voice input by the hub device of the present disclosure.
9 is a flowchart illustrating a method of operating each of a hub device, a voice assistant server, an IoT server, and a new device according to the present disclosure.
10 is a flowchart illustrating a method of operating a hub device and a new device according to the present disclosure.
11 is a flowchart illustrating a method of operating each of a hub device, a voice assistant server, an IoT server, and a new device of the present disclosure.
12A is a conceptual diagram illustrating an operation of a hub device and a listener device according to the present disclosure.
12B is a conceptual diagram illustrating an operation of a hub device and a second device according to the present disclosure.
13 is a block diagram illustrating components of a hub device according to an embodiment of the present disclosure.
14 is a flowchart illustrating a method of operating a hub device according to an embodiment of the present disclosure.
15 is a flowchart illustrating a method of operating a hub device according to an embodiment of the present disclosure.
16 is a flowchart illustrating a method of operating a hub device according to an embodiment of the present disclosure.
17 is a diagram illustrating a multi-device system environment including a hub device, a voice assistant server, and a plurality of devices.
18A and 18B are diagrams illustrating an embodiment of a voice assistant model executable in a hub device and a voice assistant server of the present disclosure.

The terms used in the embodiments of the present specification have selected general terms that are currently widely used as possible while considering the functions of the present disclosure, but this may vary according to the intention or precedent of a technician working in the field, the emergence of new technologies, etc. . In addition, in certain cases, there are terms arbitrarily selected by the applicant, and in this case, the meaning of the terms will be described in detail in the description of the corresponding embodiment. Therefore, the terms used in the present specification should be defined based on the meaning of the term and the contents of the present disclosure, not the name of a simple term.

Singular expressions may include plural expressions unless the context clearly indicates otherwise. Terms used herein, including technical or scientific terms, may have the same meaning as commonly understood by one of ordinary skill in the technical field described herein.

When a part of the specification is said to "include" a certain component, it means that other components may be further included rather than excluding other components unless specifically stated to the contrary. In addition, terms such as "... unit" and "... module" described in the specification mean a unit that processes at least one function or operation, which is implemented as hardware or software, or as a combination of hardware and software. Can be.

The expression "configured to (configured to)" used in this specification is, for example, "suitable for", "having the capacity to" depending on the situation. It can be used interchangeably with ", "designed to", "adapted to", "made to", or "capable of". The term "configured to (or set)" may not necessarily mean only "specifically designed to" in hardware. Instead, in some situations, the expression "a system configured to" may mean that the system "can" along with other devices or parts. For example, the phrase “a processor configured (or configured) to perform A, B, and C” means a dedicated processor (eg, an embedded processor) for performing the operation, or by executing one or more software programs stored in memory, It may mean a generic-purpose processor (eg, a CPU or an application processor) capable of performing corresponding operations.

In this specification, the'first natural language understanding model' is a model that is trained to analyze text converted from a voice input and determine a device for performing an operation based on the analysis result. The first natural language understanding model may determine an intent by interpreting the text, and may be used to determine a device for performing an operation based on the intent.

In this specification, the'second natural language understanding model' is a model learned to analyze text related to a specific device. The second natural language understanding model may be a model that is trained to obtain motion information about an action to be performed by a corresponding device by analyzing at least a portion of the text. The storage capacity of the second natural language understanding model may be larger than that of the first natural language understanding model.

In the present specification, the'voice assistant model' may include a device determination model and a function determination model. The'device determination model' is a model that is trained to determine an operation performing device among a plurality of devices previously registered according to a user account based on a text analysis result analyzed by the first natural language understanding model. The device determination model may include a first natural language understanding model. The'function determination model' is a model used by the device to obtain detailed operations for performing an operation according to the determined function and operation information regarding a relationship between the detailed operations. The function determination model may include a second natural language understanding model and an action plan management model.

In the present specification, the'action plan management model' may be a model trained to manage operation information related to the detailed operation of the device in order to generate detailed operations to be performed by the operation performing device and an execution order of the detailed operations. The action plan management model may manage detailed operations of a device for each device type and operation information regarding a relationship between the detailed operations.

In this specification, the'intent' is information indicating the intention of the user determined by interpreting the text. The intent is information indicating the intention of the user to speak, and may be information indicating the operation of the device performing the operation requested by the user. The intent may be determined by interpreting the text using a Natural Language Understanding (NLU) model. For example, if the text converted from the user's voice input is "Play Movie Avengers on TV", the intent may be "Play content". For another example, when the text converted from the user's voice input is "Turn down the air conditioner temperature to 18°C", the intent may be "temperature control".

The intent may include not only information indicating the user's utterance intention (hereinafter, intention information), but also a numerical value corresponding to information indicating the user's intention. The numerical value may indicate a probability that the text is associated with information indicating a specific intent. When a plurality of pieces of information indicating the user's intention are obtained as a result of analyzing the text using the natural language understanding model, intention information having a maximum numerical value corresponding to each intention information may be determined as the intent.

In this specification, the'action' of the device may mean at least one action performed by the device by executing a specific function in the device. The operation may represent at least one action performed by the device by executing an application on the device. The operation may represent, for example, playing a video, playing music, writing an email, receiving weather information, displaying news information, executing a game, and taking a picture, etc. performed by the device through execution of an application. However, the operation is not limited to the above-described example.

The operation of the device may be performed based on information on detailed operations output from the action plan management model. The device may perform at least one action by executing a function corresponding to the detailed operation output from the action plan management model. The device may store a command for executing a function corresponding to a detailed operation, and when a detailed operation is determined, the device may execute a specific function by determining a command corresponding to the detailed operation and executing the command.

Also, the device may store a command for executing an application corresponding to a detailed operation. The command for executing the application may include a command for executing the application itself and a command for executing detailed functions constituting the application. When a detailed operation is determined, the device may execute an application by executing a command for executing an application corresponding to the detailed operation, and execute a detailed function by executing a command for executing a detailed function of the application corresponding to the detailed operation.

In the present specification,'action information' may be information related to detailed operations to be performed by the device, a relationship between each detailed operation and other detailed operations, and an execution order of the detailed operations. The relationship between each detailed operation and another detailed operation includes information on another operation that must be performed before executing the operation in order to execute one operation. For example, when the operation to be performed is “music playback”, “power on” may be another detailed operation that must be essentially executed before the “music playback” operation. The operation information may include, for example, functions to be executed by the operation performing device in order to perform a specific operation, an execution order of functions, an input value required to execute the functions, and an output value output as a result of execution of the functions. However, it is not limited thereto.

In the present specification, the'operation performing device' refers to a device determined to perform an operation based on an intent obtained from a text among a plurality of devices. The operation performing device may analyze the text using the first natural language understanding model, and may be determined based on the analyzed result. The operation performing device may perform at least one action by executing a function corresponding to the detailed operation output from the action plan management model. The operation performing device may perform an operation based on the operation information.

The IoT server is a server that acquires, stores, and manages device information for each of a plurality of devices. The IoT server may acquire, determine, or generate a control command capable of controlling a device by using the stored device information. The IoT server may transmit a control command to a device determined to perform an operation based on the operation information. The IoT server may be configured as a hardware device independent from the'server' described herein, but is not limited thereto. The IoT server may be a component of the'voice assistant server' of the present specification, or may be a server designed to be classified by software.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art may easily implement the embodiments. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein.

Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the drawings.

1 is a diagram illustrating some components of a multi-device system including a hub device 1000, a voice assistant server 2000, an IoT server 3000, and a plurality of devices 4000 according to an embodiment of the present disclosure. It is a block diagram.

In FIG. 1, for convenience of description, only essential components for describing operations of the hub device 1000, the voice assistant server 2000, the IoT server 3000, and the plurality of devices 4000 are illustrated. Configurations included in the hub device 1000, the voice assistant server 2000, the IoT server 3000, and the plurality of devices 4000 are not limited as illustrated in FIG. 1.

Reference numerals S1 to S16 shown on the arrows of FIG. 1 denote data movement, transmission, and reception between a plurality of subjects through a network. The numbers attached to the alphabetic letter S in S1 to S16 are for identification, and are not related to the order of movement, transmission, and reception of data.

Referring to FIG. 1, a hub device 1000, a voice assistant server 2000, an IoT server 3000, and a plurality of devices 4000 may be interconnected through wired communication or wireless communication, and may perform communication. . In an embodiment, the hub device 1000 and the plurality of devices 4000 may be directly connected to each other through a communication network, but are not limited thereto. The hub device 1000 and the plurality of devices 4000 may be connected to the voice assistant server 2000, and the hub device 1000 may be connected to the plurality of devices 4000 through the voice assistant server 2000. In addition, the hub device 1000 and the plurality of devices 4000 may be connected to the IoT server 3000. In another embodiment, each of the hub device 1000 and the plurality of devices 4000 may be connected to the voice assistant server 2000 through a communication network, and may be connected to the IoT server 3000 through the voice assistant server 2000.

The hub device 1000, the voice assistant server 2000, the IoT server 3000, and the plurality of devices 4000 include a local area network (LAN), a wide area network (WAN), and a value-added communication network ( Value Added Network (VAN), a mobile radio communication network, a satellite communication network, or a combination thereof. Wireless communication methods include, for example, wireless LAN (Wi-Fi), Bluetooth, Bluetooth low energy, ZigBee, Wi-Fi Direct (WFD), ultra wideband (UWB), infrared communication (IrDA, infrared data). Association), NFC (Near Field Communication), and the like, but are not limited thereto.

The hub device 1000 is a device that receives a user's voice input and controls at least one of the plurality of devices 4000 based on the received voice input. The hub device 1000 may be a listener device that receives a voice input from a user. For example, the hub device 1000 may be a voice assistant speaker, but is not limited thereto.

The hub device 1000 may be determined by the voice assistant server 2000 based on resource status information for each of a plurality of devices previously registered in the IoT server 3000 according to a user account.

At least one of the plurality of devices 4000 may be an operation performing device that performs a specific operation by receiving a control command from the hub device 1000 or the IoT server 3000. The plurality of devices 4000 may be devices that are logged in with the same user account as the user of the hub device 1000 and registered in the IoT server 3000 as a user account of the hub device 1000. However, the present invention is not limited thereto, and the plurality of devices 4000 may include a new device 4300 newly registered in the IoT server 3000 according to a user account. The new device 4300 is a device registered in the IoT server 3000 by newly acquiring the user of the hub device 1000 and logging in with the same user account as the hub device 1000 and other devices 4100 and 4200. Can be

At least one of the plurality of devices 4000 may receive a control command from the IoT server 3000 (S12, S14, S16), or receive at least a portion of the text converted from a voice input from the hub device 1000. Yes (S3, S5). At least one of the plurality of devices 4000 may receive a control command from the IoT server 3000 without receiving at least a part of the text from the hub device 1000 (S12, S14, S16).

The hub device 1000 may include a device determination model 1340 that determines a device to perform an operation based on a user's voice input. The device determination model 1340 may determine a device for performing an operation from among a plurality of devices 4000 registered according to a user account. In one embodiment, the hub device 1000 includes identification information (eg, device ID information) of each of a plurality of devices 4000 from the voice assistant server 2000, a device type of each of the plurality of devices 4000, and a plurality of Device information including at least one of function performance capability, location information, and status information of each of the devices 4000 of may be received (S2). The hub device 1000 may determine a device to perform an operation according to a user's voice input from among the plurality of devices 4000 based on the received device information, using data on the device determination model 1340.

In another embodiment, the hub device 1000 includes identification information (eg, device ID information) of each of the plurality of devices 4000 directly from the plurality of devices, a device type of each of the plurality of devices 4000, and a plurality of devices. (4000) Device information including at least one of each function capability, location information, and state information may be received. The hub device may identify a plurality of devices connected to an access point connected to the hub device, and may obtain device information by requesting device information from the identified plurality of devices. In this case, the plurality of devices may include an SW module configured to transmit device information according to the request of the hub device. Also, the hub device may obtain device information by receiving device information transmitted by a plurality of devices. In this case, the plurality of devices may include an SW module configured to identify a hub device connected to an access point connected to the plurality of devices and an SW module configured to transmit device information to the identified hub device.

The hub device 1000 is. When information on a new device connected to the hub device 1000 is obtained, a request is made to the voice assistant server 2000 to update the device determination model stored in the hub device 1000. In one embodiment, the hub device 1000 analyzes text related to the new device from the user's voice input based on the device information of the new device received from the IoT server 3000, and determines the new device as the device to perform the operation. In order to be able to do so, a query signal for requesting an update on a device determination model previously stored in the memory 1300 (refer to FIG. 2) may be transmitted to the voice assistant server 2000. In this case, the hub device 1000 may transmit at least user account information and identification information (eg, ID information of the hub device) of the hub device 1000 to the voice assistant server 2000 together with a query signal.

In response to the request of the hub device 1000, the hub device 1000 receives and stores the updated device determination model from the voice assistant server 2000. In one embodiment, the hub device 1000 analyzes a user's voice input for a new device, and downloads an updated device determination model from the voice assistant server 2000 so that the new device can be determined as an operation performing device as a result of the analysis. download), and the updated device determination model may be stored in the memory 1300 (refer to FIG. 2) inside the hub device 1000.

In addition, the function determination model corresponding to the operation performing device determined by the hub device 1000 is stored in the memory 1300 of the hub device 1000 (refer to FIG. 2), stored in the operation performing device itself, or It may be stored in the memory 2300 (refer to FIG. 3) of the secretary server 2000. Here, the memory of the hub device 1000 refers to a nonvolatile memory. Non-volatile memory refers to a storage medium in which information can be stored and maintained even when power is not supplied, and information stored again can be used when power is supplied. For example, flash memory ), hard disk, solid state drive (SSD), multimedia card micro type, card-type memory (for example, SD or XD memory, etc.), read only memory (ROM) , A magnetic memory, a magnetic disk, and an optical disk. The function determination model corresponding to each device refers to a model used by the device to obtain detailed operations for performing an operation according to the determined function and operation information regarding a relationship between the detailed operations.

The hub device 1000 may monitor resource states of the hub device 1000 and the plurality of devices 4000 by using a resource tracking module 1350 and obtain resource state information. The resource status information includes information related to the usage status of the processors 1200, 4120, 4220, 4320 and the memories 1300, 4130, 4230, 4310 of the hub device 1000 and the plurality of devices 4000. The resource status information includes, for example, at least one of the remaining capacity of the hub device 1000 and the memory (1300, 4130, 4230, 4310) of the plurality of devices 4000, the average RAM usage capacity, and the average processor occupancy information. can do. The resource tracking module 1350 may provide the acquired resource state information to the function determination model management module 1360.

The hub device 1000 stores the function determination model 1370 in the hub device 1000 and the memories 1300, 4130, 4230, and 4310 of the plurality of devices 4000 using the function determination model management module 1360 It can be determined whether or not to process. When there are multiple function determination models, it is possible to determine whether to store and process the plurality of function determination models 1370 in the hub device 1000 and the memories 1300, 4130, 4230, and 4310 of each of the plurality of devices. have. The function determination model management module 1360 compares the information of the function determination model received from the voice assistant server 2000 (for example, capacity information of the function determination model) and the resource state information received from the resource tracking module 1350 And can be analyzed. The function determination model management module 1360 downloads the function determination model from the voice assistant server 2000 by using the comparison/analysis result, and the memory 1300 of the hub device 1000 and the plurality of devices 4000 and 4130, 4230, 4310).

In one embodiment, when the function determination model management module 1360 determines to download the function determination model from the voice assistant server 2000 and store it in the memory 1300, the hub device 1000 is the voice assistant server 2000. A query requesting that the function determination model be transmitted may be transmitted, and the function determination model may be received from the voice assistant server 2000 and stored in the memory 1300.

In another embodiment, the function determination model management module 1360 downloads the function determination model from the voice assistant server 2000 and stores at least one memory 4130, 4230, and a plurality of devices 4000 other than the hub device 1000. 4310), the hub device 1000 transmits a query requesting to transmit the function determination model to the voice assistant server 2000, and the function determination model from the voice assistant server 2000 May be received and transmitted to at least one of the plurality of devices 4000. The voice assistant server 2000 may transmit the function determination model to at least one of the plurality of devices 4000 instead of the hub device 1000.

In another embodiment, when the function determination model management module 1360 decides not to download the function determination model from the voice assistant server 2000, the hub device 1000 uses the function determination model previously stored in the voice assistant server 2000. Access information related to may be received, and the received access information may be stored in the function determination model database 1362.

The function determination model database 1362 is registered in the same user account as the hub device 1000 or newly registered among a plurality of function determination models 2362, 2364, 2366, 2368 previously stored in the voice assistant server 2000. Capacity information and access information on a function determination model corresponding to the device may be stored. The access information includes identification information (eg, server ID), location information, IP address information (IP address), MAC address (MAC address), and voice assistant server (2000) of the voice assistant server 2000 in which each function determination model is stored. ) May include at least one of API information (Application Programming Interface) for accessing the function determination model, a language used for each function determination model, and identification information of a corresponding device.

In one embodiment, when a new device is registered in a user account, the hub device 1000 uses the function determination model management module 1360 to provide information on a function determination model corresponding to the new device (e.g., capacity Information) is received from the voice assistant server 2000. The hub device 1000 may determine whether to download the function determination model from the voice assistant server 2000 and store it in the memory 1300 (refer to FIG. 2) based on the received information on the function determination model. When it is determined to store the function determination model in the memory 1300, the hub device 1000 may download the function determination model corresponding to the new device from the voice assistant server 2000 and store it in the memory 1300. When it is determined not to store the function determination model, the hub device 1000 may store access information on the function determination model corresponding to the new device in the function determination model database 1362.

The hub device 1000 determines a function of the hub device 1000 itself, and a function determination model 1372 for performing an operation according to the determined function and a function corresponding to at least one of the plurality of devices 4000 The judgment model 1374 may be self-saving. For example, when the hub device 1000 is a voice assistant speaker, the hub device 1000 determines a speaker function used to obtain detailed operations for performing its own function and operation information about a relationship between the detailed operations. The model 1372 can be saved. For another example, the hub device 1000 may store a TV function determination model 1374 used to obtain detailed operations corresponding to a TV and operation information about a relationship between the detailed operations. The TV may be a device previously registered in the IoT server 3000 with the same user account as the user account of the hub device 1000.

The speaker function determination model 1372 and the TV function determination model 1374 may include second natural language understanding models 1372a and 1374a and action plan management models 1372b and 1374b, respectively. The second natural language understanding models 1372a and 1374a and the action plan management models 1372b and 1374b will be described in detail with reference to FIG. 2.

The voice assistant server 2000 may determine an operation performing device to perform an operation intended by the user based on the text received from the hub device 1000. The voice assistant server 2000 may receive information about a user account from the hub device 1000 (S1). When the voice assistant server 2000 receives user account information from the hub device 1000, a query for requesting device information on a plurality of devices 4000 that are previously registered for the received user account information to the IoT server 3000 The (query) may be transmitted (S9), and device information regarding the plurality of devices 4000 may be received from the IoT server 3000 (S10). The device information includes identification information (eg, device ID information) of each of the plurality of devices 4000, a device type of each of the plurality of devices 4000, the capability of each of the plurality of devices 4000, It may include at least one of location information and status information. The voice assistant server 2000 may transmit device information received from the IoT server 3000 to the hub device 1000 (S2).

The voice assistant server 2000 includes a device determination model 2330, a voice assistant model update module 2340, an on device model update module 2350, and a plurality of function determination models 2362, 2364, and 2366, 2368). The voice assistant server 2000 uses the device determination model 2330, a function determination model corresponding to at least a part of the text received from the hub device 1000 among a plurality of function determination models 2362, 2364, 2366, 2368 After selecting and using the selected function determination model, operation information for the operation performing device to perform the operation may be obtained. The voice assistant server 2000 may transmit operation information to the IoT server 3000 (S9).

The voice assistant server 2000 may update the voice assistant model using the voice assistant model update module 2340. When a new device is newly registered in the user account of the IoT server 3000 or a new function is added to an existing device, the voice assistant server 2000 uses the voice assistant model update module 2340 to determine the device determination model 2330. ) And the function determination model 2368 may be updated. For example, when the function determination model 2364 corresponding to the second device 4200 is updated, such as when a function of the second device 4200 is added or changed, the voice assistant model update module 2340 The device determination model 2330 is updated to a new model through learning so that the device determination model 2330 analyzes the updated function to determine the second device 4200 as a device performing an operation related to the updated function. I can. In one embodiment, the voice assistant model update module 2340 includes a first natural language understanding model through learning so that the first natural language understanding model 2332 of the device determination model 2330 interprets text related to the newly updated function. 2332) can be updated with new models.

For example, when a new device 4300 is registered in the user account of the IoT server 3000, the voice assistant model update module 2340 may provide identification information of the new device 4300 and the new device ( A function determination model 2368 corresponding to 4300 may be received, and the received function determination model 2368 may be added to and stored in the existing function determination models 2362, 2364, 2366 in the memory 2300 (refer to FIG. 3). . The voice assistant model update module 2340 is a device determination model through learning so that the device determination model 2330 analyzes the intent from the text and determines a new device 4300 as a device performing an operation related to the intent as a result of the analysis. (2330) can be updated with a new model. In one embodiment, the voice assistant model update module 2340 is a first natural language understanding model through learning so that the first natural language understanding model 2332 of the device determination model 2330 interprets text related to the new device 4300. (2332) can be updated with new models.

The voice assistant server 2000 uses the on-device model update module 2350 to learn a voice assistant model for each user account, and whether to transmit at least some of the updated voice assistant models to the hub device 1000 through learning. Can be determined. In another embodiment, it may be determined to transmit at least a part of the voice assistant model updated through learning to at least one of the plurality of devices 4000 other than the hub device 1000. The on-device model update module 2350 may include an on-device model information database 2352 for storing information on the configuration of the voice assistant model for each user account. The on-device model information database 2352 includes, for example, user account information, identification information for a plurality of devices registered in association with a user account, identification information of a function determination model corresponding to each of the plurality of devices, and a function determination model. At least one of version information, version information of a device determination model, and information on a device type that may be determined as an operation performing device based on the device determination model may be stored.

When it is determined that the voice assistant server 2000 transmits the updated device determination model 2330 to the hub device 1000 using the on-device model update module 2350, the communication interface 2100 (refer to FIG. 3) is used. The updated device determination model 2330 may be transmitted to the hub device 1000 (S2). In one embodiment, when the voice assistant server 2000 receives a signal requesting transmission of the function determination model 2368 corresponding to the new device from the hub device 1000 (S1), the voice assistant server 2000 The function determination model 2368 corresponding to the new device may be transmitted to the hub device 1000 using the on-device model update module 2350 (S2). In another embodiment, the voice assistant server 2000 may transmit a function determination model 2368 corresponding to a new device to at least one of the plurality of devices 4000 using the on-device model update module 2350.

Each of the plurality of function determination models 2362, 2364, 2366, and 2368 stored in the voice assistant server 2000 will be described in detail with reference to FIG. 3.

The IoT server 3000 is connected through a network and may store information on a plurality of devices 4000 that are previously registered in a user account of the hub device 1000. In one embodiment, the IoT server 3000 includes user account information logged in from each of the plurality of devices 4000, identification information (eg, device ID information) of the plurality of devices 4000, and the plurality of devices 4000 At least one of each device type and function performance capability information of each of the plurality of devices 4000 may be received (S11, S13, S15). In an embodiment, the IoT server 3000 may receive power on/off of each of the plurality of devices 4000 or status information regarding an operation being executed from the plurality of devices 4000 ( S11, S13, S15). The IoT server 3000 may store device information and device state information received from the plurality of devices 4000.

The IoT server 3000 may transmit device information and status information received from each of the plurality of devices 4000 to the voice assistant server 2000 (S10).

The IoT server 3000 may generate a control command that can be read and executed by the operation performing device based on the operation information received from the voice assistant server 2000. The IoT server 3000 may transmit a control command to a device determined as an operation performing device among the plurality of devices 4000 (S12, S14, S16).

Components of the IoT server 3000 will be described in detail with reference to FIG. 4.

In the embodiment illustrated in FIG. 1, the plurality of devices 4000 may include a first device 4100, a second device 4200, and a new device 4300. In FIG. 1, the first device 4100 is an air conditioner, the second device 4200 is a TV, and the new device 4300 is an air purifier, but the present invention is not limited thereto. The plurality of devices 4000 include not only air conditioners, TVs, and air purifiers, but also home appliances such as robot cleaners, washing machines, ovens, microwave ovens, scales, refrigerators, and electronic frames, smartphones, and tablet personal computers. Mobile phone, video phone, e-book reader, desktop personal computer, laptop personal computer, netbook computer, workstation, server , A personal digital assistant (PDA), a portable multimedia player (PMP), an MP3 player, a mobile medical device, a camera, or a wearable device.

At least one device among the plurality of devices 4000 may itself store a function determination model. For example, in the memory 4130 of the first device 4100, detailed operations for the first device 4100 to perform an operation determined from the user's voice input and operation information regarding the relationship between the detailed operations are obtained. In addition, a function determination model 4132 used to generate a control command based on the operation information may be stored.

Among the plurality of devices 4000, the second device 4200 and the new device 4300 do not store a function determination model corresponding to each device. However, it is not limited thereto. In an embodiment, a function determination model corresponding to the new device 4300 may be stored in an internal memory of the new device 4300 by itself.

At least one of the plurality of devices 4000 may transmit information on whether or not a function determination model corresponding to each of the plurality of devices 4000 is stored on its own to the hub device 1000 (S4, S6). , S8).

2 is a block diagram illustrating components of a hub device 1000 according to an embodiment of the present disclosure.

The hub device 1000 is a device that receives a user's voice input and controls at least one of the plurality of devices 4000 based on the received voice input. The hub device 1000 may be a listener device that receives a voice input from a user.

Referring to FIG. 2, the hub device 1000 may be configured to include at least a microphone 1100, a processor 1200, a memory 1300, and a communication interface 1400. The hub device 1000 may receive a voice input (eg, a user's utterance) from a user through the microphone 1100 and obtain a voice signal from the received voice input. In one embodiment, the processor 1200 of the hub device 1000 converts the sound received through the microphone 1100 into an acoustic signal, and removes noise (for example, a non-speech component) from the acoustic signal to The signal can be acquired.

Although not shown in the drawing, the hub device 1000 detects a designated voice input (for example, a wake-up input such as'Hi Bixby','OK Google', etc.) or receives a voice signal acquired from some voice inputs. It may include a speech recognition module having a function of pre-processing.

The processor 1200 may execute one or more instructions of a program stored in the memory 1300. The processor 1200 may be composed of hardware components that perform arithmetic, logic, input/output operations, and signal processing. The processor 1200 is, for example, a central processing unit (Central Processing Unit), a microprocessor (microprocessor), a graphics processor (Graphic Processing Unit), ASICs (Application Specific Integrated Circuits), DSPs (Digital Signal Processors), DSPDs (Digital Signal Processing Devices), Programmable Logic Devices (PLDs), and Field Programmable Gate Arrays (FPGAs), but are not limited thereto.

A program including instructions for controlling the plurality of devices 4000 based on a user's voice input received through the microphone 1100 may be stored in the memory 1300. The memory 1300 may store instructions and program codes that the processor 1200 can read. In the following embodiments, the processor 1200 may be implemented by executing instructions or codes of a program stored in a memory.

The memory 1300 includes data about the ASR module 1310, data about the NLG module 1320, data about the TTS module 1330, data about the device determination model 1340, and the resource tracking module 1350. Data, data related to the function determination model management module 1360, and data corresponding to each of the plurality of function determination models 1370 may be stored.

The memory 1300 may include non-volatile memory. The memory 1300 is, for example, a flash memory type, a hard disk type, a solid state drive (SSD), a multimedia card micro type, and a card type memory ( For example, SD or XD memory), RAM (Random Access Memory) SRAM (Static Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), a magnetic memory, a magnetic disk, and an optical disk.

The processor 1200 converts the speech signal received through the microphone 1100 into text by performing ASR (Automatic Speech Recognition) using data on the ASR module 1310 stored in the memory 1300. I can.

The processor 1200 analyzes the converted text using data on the device determination model 1340 stored in the memory 1300 and determines a device performing an operation among the plurality of devices 4000 based on the analysis result of the text. I can. The device determination model 1340 may include a first natural language understanding model 1342. In one embodiment, the processor 1200 analyzes text by using data for a first natural language understanding (NLU) 1342 included in the device determination model 1340, and analyzes the text according to the analysis result of the text. Based on the plurality of devices 4000, a device performing an operation to perform an operation according to the user's intention may be determined.

The first natural language understanding model 1342 is a model trained to analyze text converted from a voice input and determine a device for performing an operation based on the analysis result. The first natural language understanding model 1342 may determine an intent by analyzing text, and may be used to determine a device for performing an operation based on the intent.

In one embodiment, the processor 1200 parses the text in units of morphemes, words, or phrases by using data on the first natural language understanding model 1342 stored in the memory 1300 ( parse), and can infer the meaning of the word extracted from the parsed text using the linguistic features (eg, grammatical elements) of the morpheme, word, or phrase. The processor 1200 may determine an intent corresponding to the meaning of the inferred word by comparing the meaning of the inferred word with predefined intents provided in the first natural language understanding model 1342.

The processor 1200 may determine a device related to the intent recognized from the text as an operation performing device based on a matching model that determines the relationship between the intent and the device. In one embodiment, the matching model is included in data on the device determination model 1340 stored in the memory 1300, and may be obtained through learning through a rule-based system, but is not limited thereto. Does not.

In one embodiment, the processor 1200 obtains a plurality of numerical values representing the degree of association between the intent and a plurality of devices (see 4000 (see FIG. 1)) by applying a matching model to the intent, and the obtained plurality of numerical values The device having the maximum value among the values may be determined as the device performing the final operation. For example, when the intent is related to the first device 4100 (see FIG. 1) and the second device 4200 (see FIG. 1), respectively, the processor 1200 A first device that obtains a first numerical value indicating the degree and a second numerical value indicating the degree of relationship between the intent and the second device 4200, and having a higher numerical value among the first numerical value and the second numerical value ( 4100) may be determined as an operation performing device.

For example, when the hub device 1000 receives a voice input from the user saying "It's hot, drop the set temperature by 2°C", the processor 1200 uses the data on the ASR module 1310 to input the voice. An intent of'controlling a set temperature' may be obtained by performing ASR for converting the text into text and analyzing the converted text using data related to the first natural language understanding model 1342. The processor 1200 applies a matching model to provide a first numerical value indicating a degree of relation between the intent of'set temperature control' and the first device 4100 as an air conditioner, and the intent of'set temperature control' and a second value that is TV. A second numerical value indicating a degree of relation between the devices 4200 and a third numerical value indicating a degree of relation between the'set temperature control' and the new device 4300 (refer to FIG. 1) as an air purifier may be obtained, respectively. The processor 1200 may determine the first device 4100 as a device for performing an operation related to'setting temperature control' based on a first numerical value that is a maximum value among the obtained numerical values.

In another example, when the hub device 1000 receives a voice input of "Play Movie Avengers~" from the user, the processor 1200 analyzes the text converted from the voice input and plays'content'. You can acquire an intent called. The processor 1200 includes a first numerical value indicating the degree of relation between the intent of'content playback' calculated using the matching model and the first device 4100 as an air conditioner, the intent of'content playback' and a second value of the TV. In the second numerical value information, which is the maximum value among the second numerical values indicating the degree of relation between the devices 4200 and the third numerical values indicating the degree of relation between the intent of'content playback' and the new device 4300, which is an air purifier. Based on this, the second device 4200 may be determined as a device for performing an operation related to'content playback'.

However, it is not limited to the above-described example, and the processor 1200 may determine a predetermined number of devices as an operation performing device by arranging numerical values indicating a degree of association between an intent and a plurality of devices in high order. In an embodiment, the processor 1200 may determine a device having a numerical value indicating a degree of relevance equal to or greater than a predetermined threshold as a device for performing an operation related to the intent. In this case, a plurality of devices may be determined as an operation performing device.

The processor 1200 may learn a matching model between an intent and an operation performing device using, for example, a rule-based system, but is not limited thereto. The artificial intelligence model used by the processor 1200 is, for example, a neural network-based system (e.g., a convolution neural network; CNN, a recurrent neural network; RNN), and SVM (Support Vector Machine), linear regression, logistic regression, Naive Bayes, random forest, decision tree, or k-nearest neighbor algorithm. A combination of one or another artificial intelligence model may be used, and the artificial intelligence models used by the processor 1200 may be stored in the device determination model 1340.

The device determination model 1340 stored in the memory 1300 of the hub device 1000 may determine a device performing an operation from among the plurality of devices 4000 registered according to the user account of the hub device 1000. The hub device 1000 may receive device information regarding each of the plurality of devices 4000 from the voice assistant server 2000 by using the communication interface 1400. The device information includes, for example, identification information (e.g., device ID information) of each of the plurality of devices 4000, a device type of each of the plurality of devices 4000, and a function performance capability of each of the plurality of devices 4000 ( capability), location information, and status information. The processor 1200 may determine a device to perform an operation according to an intent among the plurality of devices 4000 based on device information, using data on the device determination model 1340 stored in the memory 1300.

In one embodiment, the processor 1200 uses the device determination model 1340 to determine the relationship between each of the plurality of devices 4000 previously registered in the same user account as the user account of the hub device 1000 and the intent. The indicated numerical value may be analyzed, and a device having a maximum value among numerical values indicating a degree of relationship between each of the plurality of devices 4000 and the intent may be determined as the operation performing device.

The device determination model 1340 is configured to determine the device performing the operation by logging in with the same user account as the user of the hub device 1000 and using only the registered devices 4000 as device candidates. There is a technical effect that the amount of computation performed by the processor 1200 is reduced compared to the processor 2200 (refer to FIG. 3) of the voice assistant server 2000 in order to determine the relevance of In addition, due to the reduction in the amount of computation, the processing time required to determine the device for performing the operation is reduced, and thus the response speed is improved.

In one embodiment, the processor 1200 obtains the name of the device from the text using the first natural language understanding model 1342, and uses the data on the device determination model 1340 stored in the memory 1300, the device The device for performing the operation may be determined based on the name of. In one embodiment, the processor 1200 uses the first natural language understanding model 1342 to extract a common name associated with the device and a word or phrase relating to the installation location of the device from the text, and the extracted common name and installation of the device. The device for performing the operation may be determined based on the location. For example, if the text converted from the voice input is "Play Movie Avengers on TV~", the processor 1200 parses the text into words or phrases using the first natural language understanding model 1342, and The name of the device'TV' may be recognized by comparing the word or phrase with the previously stored word or phrase. The processor 1200 is logged in with the same account as the user account of the hub device 1000, and the second device 4200, which is a TV, among the plurality of devices 4000 connected to the hub device 1000, is used as an operation performing device. You can decide.

The NLG module (Natural Language Generator) 1320 may be used to provide a response message during a conversation between the hub device 1000 and a user. For example, processor 1200 uses data about NLG module 1320 to "play a movie on TV." Alternatively, a response message such as "I will lower the set temperature of the air conditioner by 2°C" can be generated. The response message generated by the NLG module 1320 may be a message composed of text.

The NLG module 1320 queries for determining a device for performing a specific operation when there are a plurality of devices for performing an operation determined by the processor 1200 or for a plurality of devices whose degree of relevance to an intent is similarly calculated. You can store the data used to generate the message. In an embodiment, the processor 1200 may generate a query message for selecting a device for performing an operation from among a plurality of device candidates by using data on the NLG module 1320. The query message may be a message for inducing a response from a user regarding which one of the plurality of device candidates is to be determined as an operation performing device.

The TTS module 1330 may convert a text response message or an inquiry message generated by the NLG module 1320 into an audio signal. In an embodiment, the processor 1200 may convert a text response message or an inquiry message into an audio signal by using data on the TTS module 1330 and control it to be output through a speaker. In an embodiment, the hub device 1000 may receive a notification message indicating an execution result of the operation from the device performing the operation. In this case, the notification message is in the form of text, and the processor 1200 may convert the notification message in the text form into an audio signal by using data on the TTS module 1330. The processor 1200 may output a notification message converted into an audio signal through a speaker.

The hub device 1000 is. When information on a new device connected to the hub device 1000 is obtained, a request is made to the voice assistant server 2000 to update the device determination model stored in the hub device 1000. In one embodiment, the hub device 1000 is based on device information of a new device received from the IoT server 3000 (eg, identification information of a new device, type information of a new device, etc.), from a user's voice input. The voice assistant server 2000 sends a query signal for requesting an update on the device determination model previously stored in the memory 1300 (refer to FIG. 2) so that the text related to the device can be analyzed and a new device can be determined as an operation performing device. ). In this case, the hub device 1000 may transmit at least user account information and identification information (eg, ID information of the hub device) of the hub device 1000 to the voice assistant server 2000 together with a query signal.

In another embodiment, the hub device 1000 may receive device information (eg, identification information of a new device, type information of a new device, etc.) of a new device from the new device. In this case, the hub device 1000 may transmit a query signal requesting a related update to the device determination model previously stored in the hub device 1000 to the voice assistant server based on the device information of the new device received from the new device. .

In response to the request of the hub device 1000, the hub device 1000 receives and stores the updated device determination model from the voice assistant server 2000. In one embodiment, the hub device 1000 analyzes a user's voice input for a new device, and downloads an updated device determination model from the voice assistant server 2000 so that a new device can be determined as an operation performing device as a result of the analysis. (download), and the updated device determination model may be stored in the memory 1300 (refer to FIG. 2) inside the hub device 1000. The operation of requesting an update of the device determination model and receiving and storing the updated device determination model may be performed by a separate software module. Also, this operation may be performed by the function determination model management module 1360.

The resource tracking module 1350 includes components of the hub device 1000 and the plurality of devices 4000, for example, processors 1200, 4120, 4220, 4320, and memories 1300, 4130, 4230, and 4310) is a module that monitors the resource status and obtains resource status information. The resource status information is information indicating the usage status of the processors 1200, 4120, 4220, 4320 and the memories 1300, 4130, 4230, and 4310 of the hub device 1000 and the plurality of devices 4000. The resource status information includes, for example, at least one of the remaining capacity of the hub device 1000 and the memory (1300, 4130, 4230, 4310) of the plurality of devices 4000, the average RAM usage capacity, and the average processor occupancy information. can do. In one embodiment, the hub device 1000 uses the resource tracking module 1350, the average processing speed of the processors 1200, 4120, 4220, 4320, the remaining capacity of the memories 1300, 4130, 4230, 4310, And at least one of the average remaining capacity of the RAM may be monitored in real time or periodically. The resource tracking module 1350 may provide resource state information acquired through monitoring to the function determination model management module 1360.

The function determination model management module 1360 may determine whether to process the function determination model 1370 by storing it in the hub device 1000 and the memories 1300, 4130, 4230, and 4310 of the plurality of devices 4000. have. When there are multiple function determination models, it is possible to determine whether to store and process the plurality of function determination models 1370 in the hub device 1000 and the memories 1300, 4130, 4230, and 4310 of each of the plurality of devices. have. The function determination model management module 1360 may determine whether to store the function determination model 1370 in the memory 1300, 4130, 4230, or 4310 of which device among the hub device 1000 and the plurality of devices 4000.

The function determination model management module 1360 includes information of a function determination model (eg, capacity information of a function determination model) received from the voice assistant server 2000 (refer to FIG. 1) and a resource received from the resource tracking module 1350. Status information can be compared and analyzed. The function determination model management module 1360 downloads a function determination model from the voice assistant server 2000 using the comparison/analysis result, and the downloaded function determination model is transferred to the hub device 1000 and a plurality of devices ( 4000) of the memory (1300, 4130, 4230, 4310) can be determined whether to store. In one embodiment, the function determination model management module 1360 stores a function determination model corresponding to the new device 4300 (see FIG. 1) received from the voice assistant server 2000 in the memories 1300, 4130, 4230, and 4310. You can decide whether to save it or not. In one embodiment, the function determination model management module 1360 is based on the resource state information obtained from the resource tracking module 1350, the processor 1200 according to the usage pattern of the hub device 1000 and the plurality of devices 4000 , 4120, 4220, 4320) and the remaining capacity and occupancy of the memories 1300, 4130, 4230, 4310 can be analyzed. For example, when a user frequently plays music or video content through the hub device 1000, decoding occurs continuously, so that the occupancy rate through the operation of the processor 1200 is high, and the RAM is The remaining capacity may be low. In another example, when the user only performs a simple operation such as asking for the weather through one of the plurality of devices 4000, the occupancy of the processors 4120, 4220, 4320 is not high, and the memory 4130, 4230, 4310) may have a large amount of remaining capacity. The function determination model management module 1360 of the hub device 1000 compares the resource state information of the processors 1200, 4120, 4220, 4320 and the memories 1300, 4130, 4230, 4310 with the information of the function determination model, It may be determined whether to store the function determination model in the memory 1300 of the hub device 1000 or the memories 4130, 4230, and 4310 of the plurality of devices 4000. Here, the information on the function determination model includes information on the capacity of the function determination model, information on the occupied RAM occupancy to read the function determination model, and the processor 1200, 4120, 4220, 4320 reads the function determination model and details according to the function. It may include at least one of information on a simulated processing time required to obtain motion information through an order of operations and detailed operations.

In one embodiment, the hub device 1000 uses data on the function determination model management module 1360, and based on the information of the function determination model received from the voice assistant server 2000, the memory 1300, 4130, and Simulation of comparing the remaining capacity of 4230 and 4310 with capacity information of the received function determination model may be performed. In one embodiment, the hub device 1000 processes the average remaining capacity of the RAM connected to the processors 1200, 4120, 4220, 4320 and the function determination model using data on the function determination model management module 1360. It is possible to perform a simulation comparing the amount of RAM occupied in the case. In one embodiment, the hub device 1000 may simulate a processing time in the case of processing a function determination model in consideration of the average processing speed of the processor 1200 by using data on the function determination model management module 1360. have.

In one embodiment, the function determination model management module 1360 determines not to store the function determination model in the memory 1300 when the difference between the remaining capacity of the memory 1300 and the capacity of the function determination model is less than a preset threshold. I can. In an embodiment, when the virtual processing time of the function determination model is longer than a preset threshold time, the function determination model management module 1360 may determine not to store the function determination model in the memory 1300. A detailed method of determining whether or not to store the function determination model in the memory 1300 by the hub device 1000 using data on the function determination model management module 1360 will be described in detail with reference to FIG. 7.

In another embodiment, the function determination model management module 1360 may analyze characteristics of a function determination model corresponding to the new device 4300. The characteristic of the function determination model is analyzed based on detailed information on a function determined to be performed by the device through the function determination model. Detailed information on the function includes the name of the function, a detailed description of the function, the resources required to perform the function (memory amount, processing speed, etc.), the application required to perform the function, and the content subject to the function. It may include types and the like. The function determination model management module 1360 acquires detailed information on a function, and provides information on the already stored user's voice assistant usage pattern, predetermined rules (one or more), and sensitive information previously input by the user. According to the type, the characteristics of the function judgment model are analyzed.

The characteristic of the function determination model includes a minimum time required for a device to perform a function and output the result through a corresponding device or another peripheral device, and the required minimum time may be determined by a predetermined rule. For example, the hub device may store information that the basic minimum time for a function for power on/off of the device is 0.1 ms, and the basic minimum time for a function for searching and playing video content from the device is 1 s. In addition, the hub device may store a rule that calculates that 0.5 ms is added to the basic minimum time when a device to which a user inputs a voice and a device to perform a function are different from each other.

In addition, the characteristic of the function determination model includes whether or not the function is similar to a function frequently used by the user when the user's voice assistant usage pattern is considered. For example, the hub device may store usage pattern information indicating that the hub device plays music once every two days. In this case, using detailed information on the functions included in the function determination model of the new device, the function determination model of the new device includes a function related to music playback, and this function is played by the user every two days. It can be determined that it is consistent with the function to be performed.

In addition, the characteristics of the function determination model include whether or not the user's sensitive information is used to perform the function, and the user's sensitive information is determined by a predetermined rule or is based on the sensitive information previously entered by the user. It can be judged according to the type. For example, when the function determination model of the new device includes a function of acquiring schedule information and registering it in a calendar app, the hub device determines whether the schedule information is sensitive information of the user. The hub device may determine that the schedule information is the user's sensitive information according to the type of sensitive information previously input by the user.

The function determination model management module 1360 may determine whether to download a function determination model corresponding to a new device by analyzing characteristics of the function determination model. For example, as a result of analyzing the characteristics of the function determination model, if some of the functions included in the function determination model are performed and the minimum time required to output the result is less than a certain amount of time, the function is determined by the hub device or a new device. You can decide to download the model.

As another example, as a result of analyzing the characteristics of the function determination model, if it is determined that some of the functions included in the function determination model are similar to the functions frequently used by the user, the function determination model is downloaded to a hub device or a new device. Can be determined. As another example, as a result of analyzing the characteristics of the function determination model, if it is determined that some of the functions included in the function determination model utilize the user's sensitive information, the function determination model is downloaded to a hub device or a new device. You can decide.

In another embodiment, the function determination model management module 1360 may obtain manufacturer or model name information of a new device and determine whether to download a function determination model corresponding to the new device according to the manufacturer or model name information. The hub device may pre-store data on whether to download a function determination model for each manufacturer or model name. When the hub device determines that the manufacturer or model name information of the new device is designated as downloading the function determination model, using the previously stored data, it may determine that the function determination model is to be downloaded. In addition, the hub device may compare the manufacturer information of the hub device with the manufacturer information of the new device and, if it is determined to match, may determine to download a function determination model corresponding to the new device.

In another embodiment, when the function determination model management module 1360 acquires manufacturer or model name information of a new device and downloads a function determination model corresponding to the new device according to the manufacturer or model name information, the data adapter module ( (Not shown) can be downloaded along with the function determination model. The data adapter module (not shown) refers to a SW module that converts output data of the voice assistant server into data in a form that can be processed by the IoT server in order to link the voice assistant server with the IoT server. In an embodiment, the data adapter module may determine similarity by comparing data obtained from the outside by the device with data stored inside the device, and convert data obtained from the outside into data most similar thereto. Data conversion may be performed based on a database composed of pairs of data before conversion and data after conversion. In this case, the data adapter module compares data obtained from the outside with data before conversion to determine similarity, and checks the data most similar to data obtained from outside among the data before conversion. The data adapter module checks the corresponding post-conversion data using the confirmed pre-conversion data, and outputs the confirmed data after conversion. In another embodiment, the data adapter module may convert externally acquired data using a data conversion model instead of a database. Of course, the data adapter module can use a database and a data conversion model together.

The data in a form that can be processed by the IoT server may be data in the same form as data that is received from the IoT server by at least one of the plurality of devices and used to perform a specific function. In addition, the data adapter module may perform an operation of converting the output data of the IoT server into data in a form that can be processed by the voice assistant server. The data in a form that can be processed by the voice assistant server may be data in the same form as information on at least one of a plurality of devices that the hub device receives from the voice assistant server.

For example, the output data of the voice assistant server includes information on detailed operations, and the output data of the IoT server includes device information including at least one of device identification information, device type, function performance capability, location information, and status information. Includes. When the hub device downloads the data adapter module to the hub device, the data adapter module (not shown) converts the output data of the hub device so that the hub device and the new device are interlocked. Specifically, the data adapter module processes the output data of the function determination model in the new device for interworking between the function determination model downloaded to the hub device and a SW module (not shown) for executing detailed operations stored in the new device. Convert it into a possible form. In addition, when the hub device receives device information for a new device received from the new device, the data adapter module converts the received device information into a form that can be processed by the hub device.

The data adapter module may be configured to correspond to the manufacturer or model name of the new device. In this case, the voice assistant server may store data adapter modules for each manufacturer or model name, and may store a plurality of data adapter modules. The hub device may request a data adapter module corresponding to the new device from the voice assistant server based on the manufacturer or model name information of the new device. In addition, even when there is no request from the hub device, the voice assistant server may transmit a data adapter module corresponding to the new device to the hub device based on the manufacturer or model name information of the new device obtained from the IoT server.

In one embodiment, when the function determination model management module 1360 determines to download the function determination model from the voice assistant server 2000 and store it in the memory 1300, the hub device 1000 is the voice assistant server 2000. A query requesting that the function determination model be transmitted may be transmitted, and the function determination model may be received from the voice assistant server 2000 and stored in the memory 1300. In this case, the hub device may request a data adapter module corresponding to the new device from the voice assistant server based on the manufacturer or model name information of the new device. The data adapter module (not shown) converts the output data of the function determination model downloaded and stored in the hub device into data in a form that can be processed by the new device.

In another embodiment, when the function determination model management module 1360 decides not to download the function determination model from the voice assistant server 2000, the hub device 1000 uses the function determination model previously stored in the voice assistant server 2000. Access information related to may be received, and the received access information may be stored in the function determination model database 1362. The access information includes identification information (eg, server ID), location information, IP address information, MAC address, and voice information of the voice assistant server (see 2000, FIG. 1) in which each function determination model is stored. It may include at least one of API information (Application Programming Interface) for accessing the function determination model in the secretary server 2000, a language used for each function determination model, and identification information of a corresponding device.

In one embodiment, when a new device is registered in a user account, the hub device 1000 uses data on the function determination model management module 1360 to provide information on a function determination model corresponding to the new device (for example, For example, capacity information) is received from the voice assistant server 2000, and based on the information on the received function determination model, whether to download the function determination model from the voice assistant server 2000 and store it in the memory 1300 is determined. You can decide. When it is determined to store the function determination model in the memory 1300, the hub device 1000 may download the function determination model corresponding to the new device from the voice assistant server 2000 and store it in the memory 1300. When it is determined not to store the function determination model, the hub device 1000 may store access information on the function determination model corresponding to the new device in the function determination model database 1362.

The hub device 1000 determines its own function and determines a function corresponding to at least one of a function determination model 1372 and a plurality of devices 4000 (refer to FIG. 1) for performing an operation according to the determined function It may be storing the model 1374 itself. The'function determination model corresponding to an operation performing device' refers to a model used by the operation performing device to obtain detailed operations for performing an operation according to the determined function and operation information regarding a relationship between the detailed operations. In one embodiment, the plurality of function determination models 1372 and 1374 stored in the memory 1300 of the hub device 1000 are logged in with the same account as the user account, and are logged in through the hub device 1000 and the network. Each may correspond to a plurality of connected devices.

For example, the first function determination model 1372 is used by the first device 4100 (refer to FIG. 1) to obtain detailed operations for performing an operation according to a function and operation information regarding a relationship between the detailed operations. It can be a model that becomes. In an embodiment, the first function determination model 1372 may be a model used to obtain operation information according to a function of the hub device 1000 itself, but is not limited thereto. Similarly, the second function determination model 1374 is a model used to obtain detailed operations for the second device 4200 (refer to FIG. 1) to perform an operation according to a function and operation information regarding a relationship between the detailed operations. Can be

Each of the first function determination model 1372 and the second function determination model 1374 analyzes at least a part of the text and performs an operation determined based on the analysis result of the at least part of the text. A second natural language understanding model (1372a, 1374a) configured to obtain information may be included. Each of the first function determination model 1372 and the second function determination model 1374 is an action plan that manages operation information related to the detailed operation of the device in order to generate detailed operations to be performed by the device and an execution order of the detailed operations. Management models 1372b and 1374b may be included. The action plan management models 1372b and 1374b may manage detailed operations of a device for each device and operation information regarding a relationship between the detailed operations. The action plan management models 1372b and 1374b may plan detailed operations to be performed by the device and an execution order of the detailed operations based on an analysis result of at least a part of the text.

The processor 1200 uses a function determination model corresponding to the operation performing device stored in the memory 1300, for example, the second natural language understanding model 1374a of the TV function determination model 1374, to be performed by the operation performing device. It is possible to obtain motion information about the motion. The second natural language understanding model 1374a is a model specialized for a specific device (eg, TV), and is associated with a device corresponding to the device performing the operation determined by the first natural language understanding model 1342 and corresponds to an intent corresponding to the text. It may be an artificial intelligence model that has been trained to obtain. In addition, the second natural language understanding model 1374a may be a model trained to determine an operation of a device related to a user's intention by analyzing text. An operation may mean at least one action performed by the device by executing a specific function in the device. The operation may represent at least one action performed by the device by executing an application on the device.

In an embodiment, the processor 1200 may analyze the text by using the second natural language understanding model 1374a of the TV function determination model 1374 corresponding to the determined operation performing device (eg, TV). The processor 1200 parses the text into morphemes, words, or phrases using the second natural language understanding model 1374a, and grasps and grasps the meaning of the parsed morphemes, words, or phrases through grammatical and semantic analysis. An intent and a parameter may be determined by matching the defined meaning to a predefined word. Here, the'parameter' means variable information for determining detailed operations of the device performing the operation related to the intent. For example, if the text transmitted to the function determination model 1374 is "Play Movie Avengers on TV~", the intent is'content play', and the parameter is'Avenger', which is information about the content to be played. It could be a movie.

The processor 1200 may obtain operation information on at least one detailed operation related to an intent and a parameter by using the action plan management model 1374b of the TV function determination model 1374. The action plan management model 1374b may manage detailed operations of a device for each device and information about a relationship between detailed operations. The processor 1200 uses the action plan management model 1374b to plan detailed operations to be performed by the operation performing device (eg, TV) and the execution order of the detailed operations based on intents and parameters. ), and through this, operation information can be obtained. The operation information may be information related to detailed operations to be performed by the device and an execution order of the detailed operations. The operation information may include detailed operations to be performed by the device, a relationship between each detailed operation and other detailed operations, and information related to an execution order. The operation information may include, for example, functions to be executed by the operation performing device in order to perform a specific operation, an execution order of functions, an input value required to execute the functions, and an output value output as a result of execution of the functions. However, it is not limited thereto.

The processor 1200 may generate a control command capable of controlling an operation performing device based on the operation information. The control command means a command that can be read and executed by the operation performing device so that the operation performing device can perform detailed operations included in the operation information. In this case, the processor 1200 must receive a control command conversion module for converting operation information into a control command from the IoT server 3000 through the communication interface 1400. In one embodiment, the processor 1200 controls the communication interface 1400 to request the IoT server 3000 to transmit a control command conversion module for converting operation information on an operation performing device into a control command. A (query) may be transmitted, and a control command conversion module may be received from the IoT server 3000 and stored in the memory 1300. However, the present invention is not limited thereto, and a control command conversion module corresponding to an operation performing device may be previously stored in the memory 1300.

The processor 1200 may control the communication interface 1400 to transmit the generated control command to an operation performing device.

The communication interface 1400 may perform data communication with the voice assistant server 2000, the IoT server 3000, and a plurality of devices 4000. The communication interface 1400 is, for example, wired LAN, wireless LAN, Wi-Fi, Bluetooth, zigbee, Wi-Fi Direct (WFD), infrared communication (IrDA, infrared Data Association), BLE (Bluetooth Low Energy), NFC (Near Field Communication), Wibro (Wireless Broadband Internet, Wibro), WiMAX (World Interoperability for Microwave Access, WiMAX), SWAP (Shared Wireless Access Protocol), Wiigg Data communication with the voice assistant server 2000, the IoT server 3000, and a plurality of devices 4000 may be performed using at least one of a data communication method including (Wireless Gigabit Allicance, WiGig) and RF communication. .

In one embodiment, the processor 1200 is logged in to the IoT server 3000 (see FIG. 1) with the same user account as the user account of the hub device 1000 through the communication interface 1400, and is newly registered instead of the existing device. Device information about the new device 4300 (refer to FIG. 1) may be received. The processor 1200 may receive device information about the new device 4300 from the IoT server 3000 using the communication interface 1400, but is not limited thereto. The IoT server 3000 transmits device information and user account information about the new device 4300 newly registered in the user account to the voice assistant server 2000 (refer to FIG. 1), and the processor 1200 of the hub device 1000 May control the communication interface 1400 to receive device information of the new device 4300 from the voice assistant server 2000.

The processor 1200 requests the voice assistant server 2000 to update the device determination model 1340 stored in the memory 1300 through the communication interface 1400, and the updated device determination model ( 1340). When a new device 4300 is newly registered in the user account of the IoT server 3000, the voice assistant server 2000 updates the device determination model 2330 (see FIG. 1) and the function determination model 2368 (see FIG. 1). can do. In this case, the processor 1200 transmits a query to the voice assistant server 2000 for inquiring about the existence of the updated device determination model 2330 for device determination regarding the new device 4300, and If there is an updated device determination model 2330, the communication interface 1400 may be controlled to transmit a signal requesting transmission to the hub device 1000.

The processor 1200 analyzes the text on the new device 4300 from the voice assistant server 2000 through the communication interface 1400 and determines the new device 4300 as an operation performing device. 2330 may be received and stored in the memory 1300.

The processor 1200 may transmit a query requesting information on a function determination model corresponding to the new device 4300 to the voice assistant server 2000 through the communication interface 1400. The information of the function determination model corresponding to the new device 4300 includes, for example, detailed operations performed by the new device 4300, a relationship between each detailed operation and other detailed operations, and the execution order of the detailed operations. It may include at least one of capacity information and access information of a function determination model for obtaining information related to. The processor 1200 downloads the function determination model from the voice assistant server 2000 based on the determination of whether to store the function determination model corresponding to the new device 4300 by the function determination model management module 1360 or Or, the communication interface 1400 may be controlled to receive only access information for the function determination model.

1 and 2, the hub device 1000 is logged in with the same user account information as the user of the hub device 1000, and a new device 4300 is registered in the user's account, or A function corresponding to the new device 4300 by downloading the updated device determination model (2330, see FIG. 1) from the voice assistant server 2000 and storing it in the memory 1300 according to the case where a new function is added to the existing device. For the determination model or the updated function determination model corresponding to the existing device, whether to download and store in the memory 1300 may be determined based on resource state information inside the hub device 1000. In general, when an operation performing device is determined through the hub device 1000 and operation information on the operation performing device is obtained, processing speed is faster than when the above processes are performed through the voice assistant server 2000, and voice There is an effect of reducing the cost of using the network consumed to access the secretary server 2000. However, a resource problem may occur in storing all of the updated function determination models in the own memory 1300 of the hub device 1000. Accordingly, in an embodiment of the present disclosure, the updated device determination model is downloaded from the voice assistant server 2000 and stored in the memory 1300, but the updated function determination model and the function determination model corresponding to the new device 4300 are used. Regarding, since the resource of the hub device 1000 itself is reviewed and whether or not to be downloaded is determined, a resource problem can be solved and a processing speed for a voice command can be improved.

3 is a block diagram illustrating components of a voice assistant server 2000 according to an embodiment of the present disclosure.

The voice assistant server 2000 receives the text converted from the user's voice input from the hub device 1000, determines an operation performing device based on the received text, and uses a function determination model corresponding to the operation performing device. It is a server that obtains operation information.

Referring to FIG. 3, the voice assistant server 2000 may include at least a communication interface 2100, a processor 2200, and a memory 2300.

The communication interface 2100 of the voice assistant server 2000 performs data communication with the IoT server 3000 (refer to FIG. 1), and thus identification information (e.g., device ID) of each of the plurality of devices 4000 (refer to FIG. 1) Information), device type of each of the plurality of devices 4000, capability of each of the plurality of devices 4000, location information, and device information including at least one of status information from the IoT server 3000 You can receive it. In one embodiment, the voice assistant server 2000 may receive user account information registered with the new device 4300 and device information about the new device 4300 from the IoT server 3000 through the communication interface 2100. have. The voice assistant server 2000 receives user account information from the hub device 1000 through the communication interface 2100, and transmits device information on the plurality of devices 4000 registered according to the received user account information to the hub device ( 1000). In an embodiment, the voice assistant server 2000 may transmit the updated device determination model 2330 to the hub device 1000 through the communication interface 2100. In one embodiment, the voice assistant server 2000 transmits information of the function determination model 2368 corresponding to the new device 4300 to the hub device 1000 through the communication interface 2100, or the function determination model (2368) It may transmit itself to the hub device 1000.

In another embodiment, the function determination model 2368 corresponding to the new device 4300 by the function determination model management module 1360 (refer to FIG. 2) of the hub device 1000 is When it is determined to transmit to any one of the devices 4000, the voice assistant server 2000 signals to transmit the device identification information and the function determination model determined from the hub device 1000 through the communication interface 2100. A query including a may be received, and a function determination model 2368 may be transmitted to the determined device in response to the received query.

The processor 2200 and the memory 2300 of the voice assistant server 2000 have the same or similar functions as the processor 1200 (see FIG. 2) and the memory 1300 (see FIG. 2) of the hub device 1000 (see FIG. 2). You can do it. Accordingly, descriptions of the processor 2200 and the memory 2300 of the voice assistant server 2000 that overlap with those described in the processor 1200 and the memory 1300 of the hub device 1000 will be omitted.

The memory 2300 of the voice assistant server 2000 includes data on the ASR module 2310, data on the NLG module 2320, data on the device determination model 2330, and data on the voice assistant model update module 2340. Data, data on the on-device model update module 2350, and data corresponding to each of the plurality of function determination models 2360 may be stored. The memory 2300 of the voice assistant server 2000 is different from the data on the plurality of function determination models 1370 (refer to FIG. 2) stored in the memory 1300 of the hub device 1000. A plurality of function determination models 2360 corresponding to a plurality of devices related to the account may be stored. In addition, the memory 2300 of the voice assistant server 2000 includes a plurality of function determination models 2360 for more types of devices than the plurality of function determination models 1370 stored in the memory 1300 of the hub device 1000. May be stored. The total capacity of the plurality of function determination models 2360 stored in the memory 2300 of the voice assistant server 2000 is larger than the capacity of the plurality of function determination models 1370 stored in the memory 1300 of the hub device 1000. I can.

When receiving at least a portion of the text from the hub device 1000, the communication interface 2100 of the voice assistant server 2000 delivers at least a portion of the received text to the processor 2200, and the processor 2200 uses a memory ( At least a part of the text may be analyzed using the first natural language understanding model 2332 stored in the 2300. The processor 2200 may determine a device for performing an operation related to at least a portion of the text based on the analysis result using the device determination model 2330 stored in the memory 2300. The processor 2200 selects a function determination model corresponding to the operation performing device from among a plurality of function determination models stored in the memory 2300, and uses the selected function determination model to perform detailed operations for the operation performing device to perform a function, and It is possible to obtain operation information about a relationship between detailed operations.

For example, when it is determined that the operation performing device is the first device 4100 that is an air conditioner, the processor 2200 uses the second natural language understanding model 2362a of the function determination model 2362 corresponding to the air conditioner. Operation information may be obtained by analyzing a part and planning detailed operations to be performed by the device and an execution order of the detailed operations using the action plan management model 2362b.

For example, when the voice assistant server 2000 receives the text'Raise the temperature by 1 degree' from the hub device 1000, the voice assistant server 2000 determines the air conditioner as an operation performing device through the device determination model 2330. Thereafter, the voice assistant server 2000 selects a function determination model 2362 corresponding to the air conditioner from among the plurality of function determination models 2360, and relates to a second natural language understanding model 2362a of the selected function determination model 2362. The text may be analyzed using the data, and operation information for performing the temperature control operation may be obtained based on the text analysis result using the data on the action plan management model 2362b.

For example, when receiving the text'change channel' from the hub device 1000, the voice assistant server 2000 determines the TV as an operation performing device through the device determination model 2330. Thereafter, the voice assistant server 2000 selects a function determination model 2364 corresponding to the TV from among the plurality of function determination models 2360, and relates to a second natural language understanding model 2364a of the selected function determination model 2364. The text may be analyzed using data, and operation information for performing a channel change operation may be obtained based on a result of the text analysis, using data related to the action plan management model 2364b.

The voice assistant server 2000 may update the voice assistant model by using the voice assistant model update module 2340 stored in the memory 2300. When a new device 4300 (refer to FIG. 1) is newly registered in the user account of the IoT server 3000 or a new function is added to an existing device, the voice assistant server 2000 uses the user account through the communication interface 2100. And device information of the new device 4300 registered in the user account may be received from the IoT server 3000. The voice assistant server 2000 may update the device determination model 2330 and the function determination model 2368 corresponding to the new device 4300 using the voice assistant model update module 2340.

For example, when the function determination model 2362 corresponding to the second device 4200 is updated, such as when the function of the second device 4200 (refer to FIG. 1) is added or changed, the voice assistant model is updated. The module 2340 creates a new device determination model 2330 through learning so that the device determination model 2330 analyzes the updated function to determine the second device 4200 as a device performing an operation related to the updated function. You can update the model. In one embodiment, the voice assistant model update module 2340 includes a first natural language understanding model through learning so that the first natural language understanding model 2332 of the device determination model 2330 interprets text related to the newly updated function. 2332) can be updated with new models.

For example, when a new device 4300 is registered in the user account of the IoT server 3000, the voice assistant model update module 2340 may provide identification information of the new device 4300 and the new device ( A function determination model 2368 corresponding to 4300 may be received, and the received function determination model 2368 may be added to and stored in the existing function determination models 2362, 2364, 2366 in the memory 2300. The voice assistant model update module 2340 is a device determination model through learning so that the device determination model 2330 analyzes the intent from the text and determines a new device 4300 as a device performing an operation related to the intent as a result of the analysis. (2330) can be updated with a new model. In one embodiment, the voice assistant model update module 2340 is a first natural language understanding model through learning so that the first natural language understanding model 2332 of the device determination model 2330 interprets text related to the new device 4300. (2332) can be updated with new models.

The voice assistant server 2000 trains a voice assistant model for each user account using the on-device model update module 2350, and uses at least some of the updated voice assistant models through the learning to a hub device 1000 (refer to FIG. 2). You can decide whether to send it to or not. In another embodiment, the voice assistant server 2000 may determine to transmit at least some of the updated voice assistant models to at least one of the plurality of devices 4000 using the on-device model update module 2350. .

The on-device model update module 2350 may include an on-device model information database 2352 for storing information on the configuration of the voice assistant model for each user account. The on-device model information database 2352 includes, for example, user account information, identification information for a plurality of devices registered in association with a user account, identification information of a function determination model corresponding to each of the plurality of devices, and a function determination model. At least one of version information, version information of a device determination model, and information on a device type that may be determined as an operation performing device based on the device determination model may be stored.

In an embodiment, the voice assistant server 2000 may determine to transmit the updated device determination model 2330 to the hub device 1000 (refer to FIG. 2) using the on-device model update module 2350. In an embodiment, the voice assistant server 2000 may transmit the updated device determination model 2330 to the hub device 1000 using the communication interface 2100. In another embodiment, the voice assistant server 2000 may transmit the updated device determination model 2330 to at least one of the plurality of devices 4000 using the communication interface 2100.

In another embodiment, the voice assistant server 2000 queries an update of a device determination model previously stored in the hub device 1000 together with information on a new device connected to the hub device 1000 from the hub device 1000 Receive a signal. In this case, the voice assistant server 2000 may receive at least user account information and identification information (eg, ID information of the hub device) of the hub device 1000 from the hub device 1000 together with a query signal. . The on-device model update module 2350 of the voice assistant server 2000 updates the device determination model previously stored in the hub device by using at least information on a new device and user account information received from the hub device 1000. You can do it.

The update of the device determination model is similar to the update of the device determination model 2330 of the voice assistant server 2000. That is, the on-device model update module 2350 learns so that the device determination model of the hub device 1000 analyzes the intent from the text and determines a new device 4300 as a device performing an operation related to the intent as a result of the analysis. Through this, the device determination model of the hub device 1000 may be updated to a new model. In one embodiment, the on-device model update module 2350 is a first natural language understanding model through learning to interpret the text related to the new device 4300 by the first natural language understanding model of the device determination model of the hub device 1000. Can be updated with a new model. A detailed description of the model update will be described in more detail with reference to FIGS. 18A and 18B.

In an embodiment, the voice assistant server 2000 may receive a signal requesting transmission of a function determination model 2368 corresponding to the new device 4300 from the hub device 1000. When a signal requesting transmission of the function determination model 2368 corresponding to the new device 4300 is received, the voice assistant server 2000 determines the function corresponding to the new device using the on-device model update module 2350 The model 2368 may be transmitted to the hub device 1000.

The voice assistant server 2000 may include a data adapter module (not shown). The data adapter module (not shown) refers to a SW module that converts output data of the voice assistant server into data in a form that can be processed by the IoT server in order to link the voice assistant server with the IoT server. The voice assistant server 2000 may store data adapter modules for each manufacturer or model name, and may store a plurality of data adapter modules. The voice assistant server 2000 may transmit a data adapter module corresponding to the new device to the hub device 1000 at the request of the hub device 1000. In another embodiment, the voice assistant server 2000 transmits a data adapter module corresponding to the new device to the hub device based on the manufacturer or model name information of the new device obtained from the IoT server, even when there is no request from the hub device. I can.

In another embodiment, the voice assistant server 2000 is a query requesting to transmit the function determination model 2368 corresponding to the new device 4300 from the hub device 1000 to any one of the plurality of devices 4000 (query) can be received. In this case, the voice assistant server 2000 may receive identification information (eg, device ID) regarding any one of the plurality of devices 4000 together with the query. The voice assistant server 2000 may transmit a function determination model 2368 corresponding to the new device to a device determined according to the received device information among the plurality of devices 4000 using the on-device model update module 2350. .

4 is a block diagram illustrating components of an IoT server 3000 according to an embodiment of the present disclosure.

The IoT server 3000 is a server that acquires, stores, and manages device information for each of a plurality of devices 4000 (refer to FIG. 1 ). The IoT server 3000 may acquire, determine, or generate a control command capable of controlling a device by using the stored device information. In FIG. 1, the IoT server 3000 is illustrated as being configured as an independent hardware device separate from the voice assistant server 2000, but is not limited thereto. In one embodiment, the IoT server 3000 may be a component of the voice assistant server 2000 (refer to FIG. 1 ), or may be a server designed to be classified by software.

Referring to FIG. 4, the IoT server 3000 may include at least a communication interface 3100, a processor 3200, and a memory 3300.

The IoT server 3000 is connected to the voice assistant server 2000 or an operation performing device through a network through the communication interface 3100 and may receive or transmit data. The IoT server 3000 may transmit data stored in the memory 3300 to the voice assistant server 2000 or an operation performing device through the communication interface 3100 under the control of the processor 3200. In addition, the IoT server 3000 may receive data from the voice assistant server 2000 or an operation performing device through the communication interface 3100 under the control of the processor 3200.

In one embodiment, the communication interface 3100 includes device identification information (eg, device ID information), function performance capability information (capability), location information, and status information from each of a plurality of devices 4000 (see FIG. 1). Device information including at least one may be received. In an embodiment, the communication interface 3100 may receive user account information from each of the plurality of devices 4000. In addition, the communication interface 3100 may receive power on/off from the plurality of devices 4000 or information on an operation being executed. The communication interface 3100 may provide the received device information to the memory 3300.

In an embodiment, the communication interface 3100 may receive user account information logged in to the new device 4300 and device information about the new device 4300 from the new device 4300 (refer to FIG. 1 ).

In one embodiment, the communication interface 3100 transmits the user account information of the new device 4300 and the device information of the new device 4300 registered in the user account under the control of the processor 3200 to the voice assistant server 2000. 1).

The processor 3200 may store device information regarding each of the plurality of devices 4000 received through the communication interface 3100 in the memory 3300. In one embodiment, the memory 3300 may classify device information according to user account information received from a plurality of devices 4000 and store the classified device information in the form of a look-up table (LUT). have.

The processor 3200 may store device information on the new device 4300 received through the communication interface 3100 and user account information logged in through the new device 4300 in the memory 3300. In one embodiment, the processor 3200 includes device information of the new device 4300, for example, identification information of the new device 4300 (eg, device ID information), function performance information (capability), location information, And at least one of the state information may be stored in the user account information corresponding to the new device 4300 in the memory 3300 in the form of a lookup table.

In one embodiment, the communication interface 3100 may receive a query for requesting device information about a plurality of devices 4000 previously registered for user account information and user account information from the voice assistant server 2000. I can. In response to the received query, the processor 3200 acquires device information about a plurality of devices 4000 previously registered for a user account from the memory 3300, and transmits the acquired device information to the voice assistant server 2000. The communication interface 3100 can be controlled to transmit.

In an embodiment, the communication interface 3100 may receive a query requesting device information about a new device 4300 newly registered in user account information from the voice assistant server 2000. In this case, the processor 3200 may control the communication interface 3100 to transmit device information about the new device 4300 newly registered in the user account to the voice assistant server 2000 in response to the received query. .

The processor 3200 may control the communication interface 3100 to transmit a control command to an operation performing device determined to perform an operation based on the operation information received from the voice assistant server 2000.

The IoT server 3000 may receive a result of performing an operation according to a control command through the communication interface 3100 from a device that has performed the operation.

5 is a block diagram illustrating some components of a plurality of devices 4000 according to an embodiment of the present disclosure.

The plurality of devices 4000 may be controlled devices that are controlled by a hub device 1000 (see FIG. 1) or an IoT server 3000 (see FIG. 1). In an embodiment, the plurality of devices 4000 may be an executor device that executes an operation based on a control command received from the hub device 1000 or the IoT server 3000.

Referring to FIG. 5, a plurality of devices 4000 may include a first device 4100, a second device 4200, and a new device 4300. However, the present invention is not limited thereto, and in one embodiment, the plurality of devices 4000 may include only the previously registered first device 4100 and the second device 4200, and may not include the new device 4300. have.

The new device 4300 is newly acquired by a user of the hub device 1000 and logs in with the same user account as the hub device 1000 and other previously registered devices 4100 and 4200. 1).

In the embodiment shown in FIG. 5, it is shown that the first device 4100 is an air conditioner, the second device 4200 is a TV, and the new device 4300 is an air purifier, but this is exemplary. The plurality of devices 4000 of the present disclosure are not limited to the device type illustrated in FIG. 5.

In FIG. 5, each of the first device 4100, the second device 4200, and the new device 4300 is shown to include only a processor, a memory, and a communication interface, but this is for convenience of description. In FIG. 5, components for the plurality of devices 4000 to perform an operation based on a control command are omitted.

Some of the plurality of devices 4000 may store a function determination model. In the embodiment shown in FIG. 5, the first device 4100 includes a communication interface 4110, a processor 4120, and a memory 4130, and a function determination model 4132 is stored in the memory 4130. Can be. The function determination model 4132 stored in the first device 4100 may be a model used to obtain detailed operations for the first device 4100 to execute an operation and operation information regarding a relationship between the detailed operations. have. The function determination model 4132 analyzes at least a part of the text received from the hub device 1000 or the IoT server 3000, and the first device 4100 to perform based on the analysis result of the at least part of the text. A natural language understanding model 4134 configured to obtain motion information regarding motion may be included. The function determination model 4132 includes an action plan management model 4136 that manages operation information related to detailed operations of the device in order to generate detailed operations to be performed by the first device 4100 and an execution order of the detailed operations. can do. The action plan management model 4136 may plan detailed operations to be performed by the first device 4100 and an execution order of the detailed operations based on an analysis result of at least a portion of the text.

The second device 4200 may include a communication interface 4210, a processor 4220, and a memory 4230. The new device 4300 may include a communication interface 4310, a processor 4320, and a memory 4330. Unlike the first device 4100, the second device 4200 and the new device 4300 do not store a function determination model. The second device 4200 and the new device 4300 do not receive at least a portion of the text from the hub device 1000 (see FIG. 1) or the IoT server 3000 (see FIG. 1 ). The second device 4200 and the new device 4300 may receive a control command from the hub device 1000 or the IoT server 3000 and execute an operation based on the received control command.

However, this is for convenience of description and is not limited thereto. In one embodiment, the new device 4300 is a function determination model for acquiring detailed operations for the new device 4300 to execute an operation, a correlation relationship between the detailed operations, and operation information regarding an execution order of the detailed operations Can contain itself.

The plurality of devices 4000 may transmit each user's account information and device information to the IoT server 3000 using the communication interfaces 4110, 4210, and 4310. The device information includes, for example, identification information (e.g., device ID information) of each of the plurality of devices 4000, a device type of each of the plurality of devices 4000, and a function performance capability of each of the plurality of devices 4000 ( capability), location information, and status information.

In an embodiment, at least one device among the plurality of devices 4000 may include an updated function determination model regarding an added, changed, or deleted function. For example, when the'windless function' is added to the first device 4100, the function determination model 4132 stored in the memory 4130 of the first device 4100 may be updated. When the function determination model 4132 of the first device 4100 is updated, the first device 4100 transmits information on the updated function determination model 4132 using the communication interface 4110 to the IoT server 3000. Can be transferred to.

The new device 4300 may transmit user account information and device information of a user logged in to the new device 4300 to the IoT server 3000 through the communication interface 4310. In an embodiment, the new device 4300 may transmit information on whether or not it stores a function determination model to the IoT server 3000 by using the communication interface 4310.

In one embodiment, the plurality of devices 4000 use the communication interfaces 4110, 4210, and 4310 to determine whether the user's account information and whether each of the plurality of devices 4000 stores their own device determination models and functions. Information on whether or not the determination model is stored may be transmitted to the hub device 1000 (refer to FIG. 1 ).

6 is a flowchart illustrating a method of receiving and storing at least a part of a voice assistant model from the voice assistant server 2000 by the hub device 1000 of the present disclosure.

In step S610, the hub device 1000 receives information on a new device connected to the hub device 1000. In one embodiment, the hub device 1000 is logged in with the same user account as the user of the hub device 1000 and receives device information about a new device newly registered in the IoT server 3000 (refer to FIG. 1) through the user account. You can receive it. The new device may be a device registered in the IoT server 3000 by newly acquiring a user of the hub device 1000 through purchase or transfer of ownership, and logging in with the same user account as the hub device 1000. The device information of the new device received by the hub device 1000 includes, for example, identification information of the new device (eg, device id information), the device type of the new device, and capability information of the new device. And at least one of location information. In an embodiment, the hub device 1000 may receive power on/off of a new device or status information regarding an operation currently being executed.

The hub device 1000 may receive device information of a new device from the IoT server 3000. However, the present invention is not limited thereto, and the hub device 1000 may receive device information of a new device from the voice assistant server 2000 (refer to FIG. 1 ).

In step S620, the hub device 1000 requests the voice assistant server 2000 to update the device determination model stored in the hub device 1000. In an embodiment, based on the device information of the new device received in step S610, the hub device 1000 analyzes text related to the new device from the user's voice input, and determines the new device as the device for performing the operation, A query signal requesting an update on a device determination model previously stored in the memory 1300 (refer to FIG. 2) may be transmitted to the voice assistant server 2000. In this case, the hub device 1000 may transmit user account information of the new device and identification information of the hub device 1000 (eg, ID information of the hub device) to the voice assistant server 2000 together with a query signal. .

In step S630, the hub device 1000 receives and stores the updated device determination model from the voice assistant server 2000. In one embodiment, the hub device 1000 analyzes a user's voice input for a new device, and as a result of the analysis, downloads an updated device determination model from the voice assistant server 2000 in order to determine the new device as an operation performing device ( download), and the updated device determination model may be stored in the memory 1300 (refer to FIG. 2) inside the hub device 1000.

In step S640, the hub device 1000 requests information on the function determination model corresponding to the new device from the voice assistant server 2000, and receives the information on the function determination model corresponding to the new device from the voice assistant server 2000. Receive. In an embodiment, when the hub device 1000 requests information on a function determination model corresponding to a new device, identification information (eg, device ID) and user account information of the hub device 1000 and the new device May be transmitted to the voice assistant server 2000.

The function determination model corresponding to the new device is related to the detailed operations to be performed by the new device, the relationship between each detailed operation and other detailed operations, and the execution order of the detailed operations when the new device is determined as the operation performing device. It is a model for acquiring information. The information on the function determination model includes, for example, capacity information of a file constituting a function determination model corresponding to a new device, information on the occupied RAM occupancy to read the function determination model, and the processor 1200 of the hub device 1000, 2) reads a function determination model and includes at least one of information on a simulated processing time required to obtain operation information through detailed operations according to the function and the order of detailed operations. I can.

In operation S650, the hub device 1000 determines whether to store the function determination model corresponding to the new device in the hub device 1000, based on the received information on the function determination model corresponding to the new device. In one embodiment, the hub device 1000 monitors the resource status of the processor 1200 (see FIG. 2) and the memory 1300 (see FIG. 2), and provides resource status information and information of a function determination model corresponding to a new device. By comparing and analyzing, it is possible to determine whether to store a function determination model corresponding to the new device in the internal memory 1300. The resource status information is information indicating the usage status of the processors 1200, 4120, 4220, 4320 and the memories 1300, 4130, 4230, 4310 of the hub device 1000 and the plurality of devices 4000, for example, The hub device 1000 and the plurality of devices 4000 may include at least one of the remaining capacity of the internal memories 1300, 4130, 4230, and 4310, the average RAM usage capacity, and the average processor occupancy information.

In one embodiment, the hub device 1000 monitors at least one of the average processing speed of the processor 1200, the remaining capacity of the memory 1300, and the average remaining capacity of the RAM in real time or periodically, and monitored result information By comparing the information of the function determination model corresponding to the new device, it is possible to determine whether to download the function determination model corresponding to the new device from the voice assistant server 2000 and store it in the memory 1300.

In another embodiment, the hub device 1000 may monitor resource state information of each of the plurality of devices 4000 (refer to FIG. 5) and obtain resource state information of each of the plurality of devices 4000. The resource status information of the plurality of devices 4000 includes the usage status of the processors 4120, 4220, 4320 (see FIG. 5) and the memory (1300, 4130, 4230, 4310, see FIG. 5) of the plurality of devices 4000. As related information, it may include at least one of the remaining capacity of the memories 4130, 4230, and 4310, the average RAM usage capacity, and the average processor occupancy information. The hub device 1000 may determine whether to store a function determination model corresponding to the new device in at least one of the plurality of devices 4000 based on the resource state information. The hub device 1000 compares and analyzes the information of the function determination model and the resource state information of each of the plurality of devices 4000, and determines a function determination model corresponding to the new device based on the comparison/analysis result. ) Of the memory (4130, 4230, 4310) can be determined whether to store.

A specific embodiment of determining whether or not the hub device 1000 to store a function determination model corresponding to a new device in step S650 will be described in detail with reference to FIG. 7.

When it is determined to store the function determination model corresponding to the new device (step S660), the hub device 1000 receives the function determination model corresponding to the new device from the voice assistant server 2000, and the internal memory 1300, 2). By storing the function determination model corresponding to the new device in the memory 1300 inside the hub device 1000, the hub device 1000 does not access the voice assistant server 2000 when the new device is determined to be an operation performing device. Even if not, information about detailed operations for a new device, a correlation relationship between detailed operations, and an execution order of the detailed operations may be obtained. Accordingly, it is possible to prevent consumption of time and network usage costs for accessing the voice assistant server 2000, and improve processing speed. In addition, when a network is not available, for example, when a wireless communication network cannot be used or a Wifi communication cannot be used, the hub device 1000 may acquire operation information about a new device by itself.

In step S660, it is described that the function determination model corresponding to the new device is stored in the memory 1300 of the hub device 1000, but is not limited thereto. In step S650, the hub device 1000 determines to store a function determination model corresponding to the new device in at least one of the plurality of devices 4000 based on the resource state information of each of the plurality of devices 4000. In this case, in step S660, the hub device 1000 may receive the function determination model from the voice assistant server 2000 and transmit it to the determined device.

In another embodiment, the hub device 1000 may transmit device information (eg, device ID) of a device determined to store a function determination model corresponding to a new device to the voice assistant server 2000. In this case, the voice assistant server 2000 may transmit a function determination model corresponding to the new device to the determined device based on the device information.

When it is determined not to store the function determination model corresponding to the new device (step S670), the hub device 1000 receives access information for the function determination model corresponding to the new device from the voice assistant server 2000. . The access information includes, for example, identification information (eg, server id), location information, IP address information (IP address), MAC address (MAC) of the voice assistant server 2000 in which the function determination model corresponding to the new device is stored. address), API information for accessing the function determination model in the voice assistant server 2000 (Application Programming Interface), a language used for the function determination model, and at least one of identification information of a corresponding device. The hub device 1000 may store the received access information in the function determination model information database 1362 (refer to FIG. 2) in the memory 1300. By storing the access information in the memory 1300, the hub device 1000 connects to the voice assistant server 2000 using the access information even if the function determination model corresponding to the new device is not stored by itself, and the voice assistant server ( 2000), detailed operations for the new device, a correlation relationship between the detailed operations, and information on the execution order of the detailed operations may be obtained by using the function determination model corresponding to the new device.

7 is a flowchart illustrating a method of determining whether to store a function determination model based on information on a function determination model received from the voice assistant server 2000 by the hub device 1000 of the present disclosure. 7 is a flowchart illustrating step S650 of FIG. 6. Steps S710, S712, and S714 illustrated in FIG. 7 may be performed after step S640 illustrated in FIG. 6 is performed.

In step S710, the hub device 1000 identifies the remaining capacity of the internal flash memory. The flash memory may be a storage medium that stores an updated device determination model and at least one function determination model in the hub device 1000. In one embodiment, the hub device 1000 uses a hard disk type, a solid state drive (SSD), or a card type external memory (eg, SD or XD memory), not a flash memory type. It can be included as a storage medium.

In step S712, the hub device 1000 checks the average remaining amount of RAM memory. The average remaining RAM memory capacity is available, excluding the amount of RAM occupied by at least one application running in the background to receive push notifications or at least one application executed by the hub device 1000 to provide a service. Shows the average of RAM capacity.

In step S714, the hub device 1000 checks the average processing speed. The hub device 1000 may check the average processing speed of the processor 1200 (refer to FIG. 2 ). The average processing speed is a function of the processor 1200 executing at least one application, converting a user's voice input into text by performing ASR, or determining a device for performing an operation related to text using a device determination model. It represents the average speed of processing the operation to perform.

Step S710, step S712, and step S714 are steps of monitoring the resource state of the hub device 1000 and obtaining resource state information, in which the processor 1200 is stored in the memory 1300 (refer to FIG. 2). 1350) can be used. Step S710, step S712, and step S714 may be performed in real time or periodically.

Step S710, step S712, and step S714 may be performed simultaneously, but are not limited thereto.

Step S710, step S712, and step S714 are not all performed, and at least one step may be performed. If any one of steps S710, S712, and S714 is not performed, steps performed after the steps that are not performed may also not be performed. For example, when step S710 is not performed, a subsequent step S720 performed after step S710 may not be performed.

Step S710, step S712, and step S714 are shown as checking the internal flash memory remaining capacity of the hub device 1000, the average RAM remaining capacity of the hub device 1000, and the average processing speed of the hub device 1000, respectively. However, it is not limited thereto. In one embodiment, steps S710, S712, and S714 each determine the remaining flash memory capacity, the average RAM remaining capacity, and the average processing speed of a plurality of devices (4000, see FIG. 5), not the hub device 1000. Can be monitored.

In step S720, the hub device 1000 compares the remaining capacity of the flash memory with information on the received function determination model. The information on the function determination model received in step S640 includes information on the capacity of a file constituting the function determination model, information on the occupied RAM occupied to read the function determination model, and the processor 1200 of the hub device 1000 (see FIG. 2 ). ) May include at least one of information on a simulated processing time required to read the function determination model and obtain operation information through detailed operations according to a function and a sequence of detailed operations. The hub device 1000 may compare the remaining capacity of the flash memory and capacity information of a file constituting the function determination model. In an embodiment, the hub device 1000 may compare the remaining capacity of the flash memory of each of the plurality of devices 4000 with the capacity information of the function determination model.

In step S722, the hub device 1000 simulates the operation of the function determination model in consideration of the remaining amount of RAM memory. In one embodiment, the hub device 1000 compares the average RAM memory remaining capacity checked in step S712 with the RAM occupancy information among the information of the function determination model received in step S640, so that the processor 1200 is included in the function determination model. It is possible to simulate the amount of RAM occupied by reading and executing instructions composed of a programming language. In an embodiment, the hub device 1000 may simulate an operation of a function determination model in consideration of the remaining amount of RAM of each of the plurality of devices 4000.

In step S724, the hub device 1000 simulates the operation of the function determination model in consideration of the average processing speed. In one embodiment, the hub device 1000 reads the function determination model by comparing the average processing speed checked in step S714 with the information on the virtual processing time among the information of the function determination model received in step S640. The processing speed for acquiring motion information may be simulated through detailed operations according to a function and a sequence of detailed operations. In an embodiment, the hub device 1000 may simulate an operation of a function determination model in consideration of an average processing speed of each of the processors 4120, 4220, and 4320 of each of the plurality of devices 4000.

In step S730, the hub device 1000 checks whether the difference between the remaining capacity of the memory and the capacity of the function determination model is greater than a predetermined threshold of the remaining capacity of the memory. In an embodiment, the hub device 1000 may calculate a difference between the remaining capacity of the flash memory and capacity information of a file constituting the function determination model. The capacity information of the file constituting the function determination model may be obtained from the information of the function determination model received in step S640. In an embodiment, the hub device 1000 may compare the calculated difference value with a preset threshold value of the remaining memory capacity, and determine whether the calculated difference value as a result of the comparison is greater than a preset threshold value.

When it is confirmed that the difference value between the remaining capacity of the flash memory and the capacity of the function determination model is larger than the preset threshold of the remaining memory capacity (step S740), the hub device 1000 performs a simulated processing time of the function determination model. ) Is less than a preset processing time threshold. In an embodiment, the hub device 1000 may obtain information on the virtual processing time of the function determination model from the information on the function determination model received in operation S640. The hub device 1000 may compare the virtual processing time of the acquired function determination model with a preset processing time threshold, and as a result of the comparison, determine whether the virtual processing time of the function determination model is shorter than a preset threshold.

When it is determined that the difference between the remaining capacity of the flash memory and the capacity of the function determination model is smaller than the preset threshold of the remaining memory capacity (step S670), the hub device 1000 is transferred from the voice assistant server 2000 to the new device. Receives access information for a corresponding function determination model.

When it is determined that the virtual processing time of the function determination model is shorter than the preset processing time threshold (step S660), the hub device 1000 receives the function determination model corresponding to the new device from the voice assistant server 2000 and internally Can be stored in memory.

When it is determined that the virtual processing time of the function determination model is longer than the preset processing time threshold (step S670), the hub device 1000 accesses the function determination model corresponding to the new device from the voice assistant server 2000 (step S670). access) information is received.

8 is a flowchart illustrating a method of controlling an operation of a new device based on a user's voice input by the hub device 1000 of the present disclosure. 8 includes an embodiment of a series of steps performed after step S660 shown in FIGS. 6 and 7. In step S810 shown in FIG. 8, the hub device 1000 receives a function determination model corresponding to the new device from the voice assistant server 2000 (refer to FIG. 1), and the function determination model corresponding to the received new device is stored in an internal memory. It shows the steps after saving to.

In step S810, the hub device 1000 requests a control command conversion module to convert operation information for a new device into a control command to the IoT server 3000 (refer to FIG. 1). The operation information for the new device is information obtained by analyzing the user's voice input using a function determination model corresponding to the new device. It may be information related to an association relationship and an execution order of detailed operations. The control command conversion module, so that the new device can perform detailed operations included in the operation information acquired by the hub device 1000, the operation information can be read by the new device, and can be executed as a command or a set of commands. It may be a module that converts. In an embodiment, the hub device 1000 may transmit identification information of a new device (eg, an id of a new device) and a query for requesting a control command conversion module to the IoT server 3000.

In step S820, the hub device 1000 receives a control command conversion module for a new device from the IoT server 3000. In an embodiment, the hub device 1000 may store the received control command conversion module in the memory 1300 (refer to FIG. 2 ).

In step S830, the hub device 1000 receives a user's voice input. In an embodiment, the hub device 1000 may receive a voice input (eg, a user's utterance) from a user through a microphone 1100 (see FIG. 2) and obtain a voice signal from the received voice input. . In one embodiment, the processor 1200 (refer to FIG. 2) of the hub device 1000 converts sound received through the microphone 1100 into an acoustic signal, and noise (eg, a non-speech component) from the acoustic signal It is possible to obtain a voice signal by removing.

In step S840, the hub device 1000 converts the received voice input into text by performing Automatic Speech Recognition (ASR). In one embodiment, the processor 1200 of the hub device 1000 converts a speech signal into a computer-readable text using a predefined model such as an acoustic model (AM) or a language model (LM). ASR to convert can be performed. When the hub device 1000 receives an acoustic signal from which noise has not been removed, the processor 1200 may obtain a voice signal by removing noise from the received acoustic signal, and perform ASR on the voice signal.

In step S850, the hub device 1000 analyzes the text using the updated device determination model, and determines a new device as a device for performing an operation related to the analyzed text. In one embodiment, the processor 1200 of the hub device 1000 analyzes the text using the updated first natural language understanding model included in the updated device determination model, and based on the analysis result of the text. Accordingly, a new device may be determined as an operation performing device to perform an operation according to the user's intention among the plurality of devices. Here, the plurality of devices are devices logged in with the same user account as the hub device 1000 and connected to the hub device 1000 through a network. The plurality of devices may be devices registered with the same user account as the hub device 1000 in the IoT server.

The first natural language understanding model is a model trained to analyze text converted from a speech input and determine a device for performing an operation based on the analysis result. The first natural language understanding model may determine an intent by interpreting the text, and may be used to determine a device for performing an operation based on the intent. The hub device 1000 parses the text in units of morphemes, words, or phrases using the first natural language understanding model, and performs linguistic characteristics of the morphemes, words, or phrases (e.g., grammatical Element) can be used to infer the meaning of the word extracted from the parsed text. The processor 1200 may determine an intent corresponding to the meaning of the inferred word by comparing the meaning of the inferred word with predefined intents provided in the first natural language understanding model.

The hub device 1000 may determine a device related to the intent recognized from the text as an operation performing device based on a matching model that determines the relationship between the intent and the device. In one embodiment, the matching model may be obtained through learning through a rule-based system, but is not limited thereto.

In one embodiment, the hub device 1000 obtains a plurality of numerical values representing the degree of relevance between the intent and the plurality of devices by applying a matching model to the intent, and has a maximum value among the obtained plurality of numerical values. The device may be determined as the device performing the final operation. For example, when the intent is related to the first device and the second device, respectively, the hub device 1000 includes a first numerical value indicating a degree of association between the intent and the first device, and between the intent and the second device. A second numerical value indicating a degree of relevance may be obtained, and a first device having a higher numerical value among the first numerical value and the second numerical value may be determined as the operation performing device.

The hub device 1000 may learn a matching model between an intent and an operation performing device using, for example, a rule-based system, but is not limited thereto. The artificial intelligence model used by the hub device 1000 is, for example, a neural network-based system (e.g., a convolution neural network; CNN, a recurrent neural network; RNN), and SVM ( Support Vector Machine), linear regression, logistic regression, Naive Bayes, random forest, decision tree, or k-nearest neighbor algorithm. It may be a combination of the above or other artificial intelligence models.

The updated device determination model may be a model trained to determine an operation performing device from among a newly registered new device and a plurality of previously registered devices according to a user account of the hub device 1000. In one embodiment, the updated device determination model analyzes text on a new device, obtains an intent as a result of the text analysis, and learns to determine a new device as an operation performing device based on the obtained intent. It may be an updated model through.

The updated device determination model represents the degree of relevance between the intent obtained from the text and the intent with each of a newly registered new device and a plurality of previously registered devices by logging in with the same user account as the user account of the hub device 1000. The numerical value may be analyzed, a numerical value indicating a degree of relevance to each of the plurality of devices may be calculated, and a device having a maximum value among the calculated numerical values may be determined as an operation performing device.

In an embodiment, the hub device 1000 may receive device information about each of a newly registered new device and a plurality of previously registered devices according to a user account from the voice assistant server 2000 (refer to FIG. 1 ). The device information may include, for example, at least one of identification information (eg, device ID information), device type, function performance capability, location information, and status information of each of a new device and a plurality of devices. have. The hub device 1000 may determine a device to perform an operation according to an intent among a new device and a plurality of devices based on the received device information. In an embodiment, the hub device 1000 may determine a new device as an operation performing device by analyzing text using the updated device determination model.

In step S860, the hub device 1000 provides at least a part of the text to the function determination model corresponding to the new device stored in the internal memory. Although not shown in the drawing, in step S860, the hub device 1000 may select a function determination model corresponding to the new device from among at least one function determination model previously stored in the internal memory. The'function determination model corresponding to the new device' is an operation regarding detailed operations for performing an operation according to the determined function by a new device determined as an operation performing device, a correlation relationship between the detailed operations, and an execution order of the detailed operations. It refers to the model used to acquire information.

The hub device 1000 provides at least a portion of the text to the selected function determination model. In an embodiment, the processor 1200 (refer to FIG. 2) of the hub device 1000 may not provide the entire text to the selected function determination model, but may provide only a part of the text. For example, if the new device is an air purifier, and the text converted from the voice input is "Execute the odor removal mode in the air purifier~",'in the air purifier' specifies the name of the device performing the operation. It may be unnecessary information in the function determination model corresponding to. The processor 1200 parses the text into words or phrases using the updated first natural language understanding model, recognizes a word or phrase specifying a device name, a common name, or an installation location, etc. , A part of the remaining text excluding words or phrases recognized in the entire text may be provided to the function determination model.

In step S870, the hub device 1000 obtains operation information about an operation to be performed by the new device by analyzing at least a part of the text using a function determination model corresponding to the new device. In an embodiment, the processor 1200 (refer to FIG. 2) of the hub device 1000 may analyze at least a portion of the text using the second natural language understanding model included in the function determination model corresponding to the new device. The processor 1200 parses the text into morphemes, words, or phrases by using the second natural language understanding model, grasps the meaning of the parsed morphemes, words, or phrases through grammatical and semantic analysis, and identifies the sensed meaning. The intent and parameter may be determined by matching to a predefined word. Here, the'parameter' means variable information for determining detailed operations of the device performing the operation related to the intent. For example, if the text provided in the function determination model corresponding to the new device is "Execute the odor removal mode~", the intent is "Run operation mode", and the parameter may be the "Odor removal mode", which is the mode to be operated. have.

The processor 1200 (refer to FIG. 2) may obtain operation information regarding at least one detailed operation related to the intent and the parameter by using the action plan management module of the function determination model corresponding to the new device. The action plan management module may manage detailed operations of a new device determined as an operation performing device and information about a relationship between detailed operations. By using the action plan management module, the processor 1200 may plan detailed operations to be performed by the new device and an execution order of detailed operations based on intents and parameters, and obtain operation information through this. .

The operation information may be information related to detailed operations to be performed by the new device and an execution order of the detailed operations. The operation information may include detailed operations to be performed by the new device, a relationship between each detailed operation and another detailed operation, and information related to an execution order. The operation information may include, for example, functions to be executed by a new device to perform a specific operation, an execution order of functions, an input value required to execute the functions, and an output value output as a result of execution of the functions. , Is not limited thereto.

In step S880, the hub device 1000 acquires a control command based on the acquired operation information. The control command refers to a command or a set of commands that can be read and executed by a new device so that the new device can perform detailed operations included in the operation information.

9 is a flowchart illustrating an operation method of each of the hub device 1000, the voice assistant server 2000, the IoT server 3000, and the new device 4300 of the present disclosure.

9 illustrates operations of each subject in a multi-device system environment including a hub device 1000, a voice assistant server 2000, an IoT server 3000, and an operation performing device 4000a. 9, the hub device 1000 includes an ASR module 1310, an NLG module 1320, a TTS module 1330, a device determination model 1340, a resource tracking module 1350, and a function determination model management module. (1360) may be included. In the embodiment illustrated in FIG. 9, the hub device 1000 may not store a function determination model or may not store a function determination model corresponding to a new device.

An ASR module 1310, an NLG module 1320, a TTS module 1330, a device determination model 1340, a resource tracking module 1350, and a function determination model management module 1360 shown in FIG. 9 are shown in FIG. Since they are the same as those shown, redundant descriptions will be omitted.

The voice assistant server 2000 may store a device determination model 2330, a voice assistant model update module 2340, an on-device model update module 2350, and a plurality of function determination models 2362, 2364, and 2368. For example, the first function determination model 2362 stored in the voice assistant server 2000 determines the function of the air conditioner, and obtains detailed operations related to the determined function and operation information regarding the relationship between the detailed operations. It may be the model used. For example, the second function determination model 2364 is a model used to determine a function of a TV and obtain detailed operations related to the determined function and operation information regarding a relationship between the detailed operations, and a new device function The determination model 2368 may be a model used to determine a function of an air purifier, which is a new device, and to obtain detailed operations related to the determined function and operation information regarding a relationship between the detailed operations.

In the embodiment shown in FIG. 9, the operation performing device is determined as a'air purifier', which is a new device 4300, and a new device function determination model 2368 corresponding to the air purifier is stored in the voice assistant server 2000. There may be.

In step S910, the new device 4300 transmits device information and user account information of the new device to the IoT server 3000. The new device 4300 is, when a user of the hub device 1000 acquires the new device 4300 and logs in by entering a user account after turning on the power of the new device 4300, the user account acquired during the login process. Information and device information may be transmitted to the IoT server 3000. The device information transmitted to the IoT server 3000 is, for example, identification information of the new device 4300 (for example, new device id information), the device type of the new device 4300, and the function of the new device 4300. It may include at least one of performance information, location information, and status information.

In step S920, the IoT server 3000 registers a new device in the received user account. The IoT server 3000 may classify and register a plurality of devices according to user accounts.

In the present disclosure, the new device 4300 refers to a device registered in the IoT server 3000 by newly acquired by the same user as the user of the hub device 1000 and logging in with the same user account as the hub device 1000 do.

In step S922, the IoT server 3000 transmits user account information and device information of a new device registered in the user account to the voice assistant server 2000.

In step S930, the IoT server 3000 transmits device information of the new device 4300 to the hub device 1000. The IoT server 3000 may identify the hub device 1000 previously registered for the received user account and transmit device information of the new device 4300 to the identified hub device 1000.

In step S932, the voice assistant server 2000 transmits device information of the new device 4300 to the hub device 1000. The voice assistant server 2000 may identify the hub device 1000 previously registered for the user account received from the IoT server 3000 and transmit device information of the new device 4300 to the identified hub device 1000. have.

In step S940, the hub device 1000 identifies whether the device determination model 1340 needs to be updated. In one embodiment, the hub device 1000 may check whether the device determination model 1340 stored in the memory 1300 (refer to FIG. 2) can determine the new device 4300 as an operation performing device from a user's voice input. have. For example, the hub device 1000 analyzes the text converted from the user's voice input by the first natural language understanding model 1342 included in the device determination model 1340, It is possible to check whether utterance data for obtaining a tent is included.

As a result of the check, if the update of the device determination model 1340 related to the new device 4300 is not required (step S944), the hub device 1000 determines the device performing the operation using the existing device determination model 1340. . In step S944, the existing device determination model 1340 is a model previously stored in the memory 1300 (refer to FIG. 2) before the hub device 1000 receives device information of the new device 4300, and is separately updated. Without a process, an intent related to the new device 4300 may be obtained from the text, and the new device 4300 may be determined as an operation performing device based on the acquired intent.

As a result of the confirmation, when the device determination model 1340 related to the new device 4300 needs to be updated (step S942), the hub device 1000 requests update of the user account information, the identification information of the hub device, and the device determination model. The requested query is transmitted to the voice assistant server 2000.

In step S950, the voice assistant server 2000 updates the device determination model 2330 by using the device information of the new device. The voice assistant server 2000 uses the voice assistant model update module 2340, and the device determination model 2330 analyzes the intent from the text based on the received device information of the new device 4300, and the analysis result is The device determination model 2330 may be updated to a new model through learning or the like so as to determine a new device 4300 as a device for performing an operation related to the tent. In one embodiment, the voice assistant model update module 2340 is a new model of the first natural language understanding model through learning so that the first natural language understanding model of the device determination model 2330 interprets text related to the new device 4300. Can be updated with

In step S952, the voice assistant server 2000 provides the updated device determination model 2330 to the hub device 1000. In an embodiment, the voice assistant server 2000 may transmit at least one file constituting the updated device determination model 2330 to the hub device 1000.

In step S960, the hub device 1000 stores the updated device determination model 2330 in the memory 1300 (refer to FIG. 2). In one embodiment, the updated device determination model 2330 received from the voice assistant server 2000 may be replaced by overwriting the device determination model 1330 previously stored in the hub device 1000. I can.

In step S970, the voice assistant server 2000 obtains function determination model information corresponding to the new device 4300 by using the device information of the new device 4300. In one embodiment, the voice assistant server 2000 uses the device information of the new device 4300 received from the IoT server 3000 in step S922, and uses the function determination model 2368 corresponding to the new device 4300. The acquired function determination model 2368 may be added to and stored in the existing function determination models 2362 and 2364 in the memory 2300 (refer to FIG. 3).

In step S972, the voice assistant server 2000 provides information on a function determination model corresponding to the new device 4300 to the hub device 1000. For information on the function determination model, see, for example, capacity information of the function determination model 2368, information on the occupied RAM occupied for reading the function determination model 2368, and the processor 1200 of the hub device 1000, FIG. ) Reads the function determination model 2368 to include at least one of information on the simulated processing time required to obtain operation information through detailed operations according to the function and the order of detailed operations. I can.

In step S980, the hub device 1000 identifies the capacity of the function determination model 2368 from the information of the function determination model corresponding to the new device 4300. In an embodiment, the hub device 1000 may check the capacity of at least one file constituting the function determination model 2368 from information on the function determination model received from the voice assistant server 2000. However, the present invention is not limited thereto, and the hub device 1000 includes information on occupied RAM occupied in order to read the function determination model 2368 from the information of the function determination model and the processor 1200 of the hub device 1000 (refer to FIG. 2 ). Information on the virtual processing time for reading the function determination model 2368 may be checked.

In step S990, based on the received information of the function determination model 2368 corresponding to the new device 4300, the hub device 1000 determines the function determination model 2368 corresponding to the new device 4300 to the hub device ( 1000) in the memory (1300, see FIG. 2) is determined whether or not to store. Since step S990 is the same as step S650 (including steps S710 to S740) shown in FIG. 7, a redundant description will be omitted.

10 is a flowchart illustrating a method of operating the hub device 1000 and the new device 4300 of the present disclosure.

10 is a flowchart illustrating operations of each subject in a multi-device system environment including the hub device 1000 and the new device 4300 after step ⓐ shown in FIG. 9. Step ⓐ shown in FIG. 9 represents a state in which the hub device 1000 determines to store the function determination model 2368 corresponding to the new device 4300 in the hub device.

In step S1010, the hub device 1000 may download the function determination model 2368 corresponding to the new device 4300 from the voice assistant server 2000 and store it in the internal memory 1300 (see FIG. 2). have. Referring to FIG. 10, the hub device 1000 may include a new device function determination model 1378. In an embodiment of step S1010, the hub device 1000 transmits a query requesting to transmit the function determination model 2368 corresponding to the new device 4300 to the voice assistant server 2000, and The function determination model 2368 may be downloaded from the secretary server 2000 and stored in the memory 1300. The new device function determination model 1378 included in the hub device 1000 may be the same as the function determination model 2368 corresponding to the new device 4300 received from the voice assistant server 2000.

In step S1020, the hub device 1000 receives a user's voice input.

In step S1030, the hub device 1000 converts the received voice input into text by performing ASR.

In step S1040, the hub device 1000 analyzes the text using the updated device determination model 1330, and determines a new device 4300 as a device for performing an operation related to the analyzed text.

In operation S1050, the hub device 1000 provides at least a part of the text to the new device function determination model 1378 stored in the internal memory 1300.

In step S1060, the hub device 1000 obtains operation information regarding an operation to be performed by the new device 4300 by analyzing at least a part of the text using the new device function determination model 1378.

Steps S1020 to S1060 are the same as steps S830 to S870 shown in FIG. 8, and thus redundant descriptions will be omitted.

In step S1062, the hub device 1000 transmits operation information by using the identification information of the new device 4300. In one embodiment, the hub device 1000 identifies the new device 4300 using identification information of the new device 4300, for example, the ID information of the new device 4300, and the identified new device 4300 Operation information can be transmitted to.

In step S1070, the new device 4300 acquires a control command based on the acquired operation information. In step S1070, the new device 4300 may itself include a control command conversion module that converts operation information into a control command. The new device 4300 may use the control command conversion module to convert the operation information into a command or a set of commands that the new device 4300 can read and execute.

In step S1080, the new device 4300 executes an operation according to the acquired control command.

In an embodiment, after executing the operation, the new device 4300 may transmit information about the operation execution result to the hub device 1000. However, the present invention is not limited thereto, and the new device 4300 may transmit information on the result of the operation execution to the IoT server 3000 (refer to FIG. 1 ).

11 is a flowchart illustrating an operation method of each of the hub device 1000, the voice assistant server 2000, the IoT server 3000, and the new device 4300 of the present disclosure.

11 illustrates operations of each subject in a multi-device system environment including a hub device 1000, a voice assistant server 2000, an IoT server 3000, and a new device 4300 after step ⓑ shown in FIG. 9. It is a flow chart showing. Step ⓑ shown in FIG. 9 represents a state in which the hub device 1000 determines that the function determination model 2368 corresponding to the new device 4300 is not stored in the hub device. Referring to the hub device 1000 illustrated in FIG. 11, unlike the hub device 1000 illustrated in FIG. 10, the new device function determination model 1378 is not included, which is not included in the new device 4300 in step ⓑ. This is because it is decided not to store the corresponding function determination model in the memory 1300 (refer to FIG. 2).

In step S1110, the hub device 1000 transmits a query requesting access information of the function determination model 2368 corresponding to the new device 4300 to the voice assistant server 2000. The access information includes, for example, identification information (eg, server id), location information, IP address information (IP address) of the voice assistant server 2000 in which the function determination model 2368 corresponding to the new device 4300 is stored. ), MAC address, API information for accessing the function determination model 2368 in the voice assistant server 2000 (Application Programming Interface), the language used for the function determination model 2368, and a corresponding new device ( 4300) may include at least one of the identification information. In an embodiment of step S1110, the hub device 1000 may transmit identification information and user account information of the hub device 1000 to the voice assistant server 2000.

In step S1120, the voice assistant server 2000 acquires access information for the function determination model 2368 corresponding to the new device 4300.

In step S1122, the voice assistant server 2000 transmits the access information of the function determination model 2368 corresponding to the new device 4300 to the hub device 1000.

In step S1130, the hub device 1000 stores the received access information in the internal memory 1300 (refer to FIG. 2).

In step S1140, the hub device 1000 receives a user's voice input.

In step S1150, the hub device 1000 converts the received voice input into text by performing ASR.

In step S1160, the text is analyzed using the updated device determination model 1330, and a new device 4300 is determined as a device for performing an operation corresponding to the analyzed text.

Steps S1140 to S1160 are the same as steps S830 to S850 shown in FIG. 8, and therefore, a duplicate description will be omitted.

In step S1162, the hub device 1000 transmits at least a part of the text to the function determination model 2368 corresponding to the new device 4300 based on the access information. In one embodiment, the hub device 1000 includes identification information (eg, server ID), location information, and IP address of the voice assistant server 2000 in which the function determination model 2368 corresponding to the new device 4300 is stored. Information (IP address), MAC address (MAC address), API information (Application Programming Interface) for accessing the function determination model 2368 in the voice assistant server 2000, language and correspondence of the function determination model 2368 The voice assistant server 2000 in which the function determination model 2368 is stored is identified based on at least one of the identification information of the new device 4300 to be performed, and at least the text is added to the function determination model 2368 of the voice assistant server 2000. Some can be transmitted.

In step S1170, the voice assistant server 2000 analyzes at least a part of the text using the function determination model 2368 corresponding to the new device 4300, so that the new device 4300 receives operation information about the operation to be performed. Acquire. In one embodiment, the voice assistant server 2000 selects a function determination model 2368 corresponding to the new device 4300 from among a plurality of function determination models 2362, 2364, 2368, and the selected function determination model 2368 The text is analyzed using the natural language understanding model 2368a of, and an intent is determined based on the analysis result. The voice assistant server 2000 may analyze at least a part of the text received from the hub device 1000 by using the natural language understanding model 2368a of the new device function determination model 2368. The natural language understanding model 2368a is an artificial intelligence model trained to analyze text related to the new device 4300, and may be a model trained to determine intents and parameters related to an operation intended by the user. The natural language understanding model 2368a may be a model trained to determine a function related to the type of the new device 4300 when text is input.

In one embodiment, the voice assistant server 2000 parses at least a part of the text into words or phrases using the natural language understanding model 2368a, and the parsed morphemes, words, or phrases have linguistic features (e.g. : A grammatical factor) can be used to infer the meaning of a word extracted from the parsed text, and match the inferred meaning to a predefined intent and parameter to obtain an intent and a parameter from the text. In one embodiment, the voice assistant server 2000 may determine only the intent from at least a portion of the text.

The voice assistant server 2000 acquires operation information on an operation to be performed by the new device 4300 based on the intent. In one embodiment, the voice assistant server 2000 uses the action plan management module 2368b of the new device function determination model 2368, based on the intent and the parameter, the operation information to be performed by the new device 4300. Plan. The action plan management module 2368b may analyze operations to be performed by the new device 4300 based on the intent and parameters. The action plan management module 2368b may select detailed operations related to interpreted operations among previously stored device operations, and may plan an execution order of the selected detailed operations. The action plan management module 2368b may acquire operation information regarding a detailed operation to be performed by the new device 4300 by using the planning result.

In step S1172, the voice assistant server 2000 transmits the acquired operation information and identification information of the new device 4300 to the IoT server 3000.

In step S1180, the IoT server 3000 acquires a control command based on the identification information of the new device 4300 and the received operation information. The IoT server 3000 may include a database in which control commands and operation information for a plurality of devices are stored. In one embodiment, the IoT server 3000 controls detailed operations of the new device 4300 based on identification information of the new device 4300 from among control commands for a plurality of devices previously stored in the database DB. You can select a control command for.

In step S1182, the IoT server 3000 transmits a control command to the new device 4300 by using the identification information of the new device 4300.

In step S1190, the new device 4300 executes operations corresponding to the control command according to the received control command.

In an embodiment, after executing the operation, the new device 4300 may transmit information about the operation execution result to the IoT server 3000.

12A is a conceptual diagram illustrating an operation of a hub device and a listener device according to the present disclosure.

Arrows shown in FIG. 12A indicate movement, transmission, and reception of data including text and voice signals between the first device 4100a and the second device 4200a. The numbers in the original letters indicate the order in which each action is performed.

The first device 4100a and the second device 4200a may be interconnected through wired communication or wireless communication, and may perform data communication. In one embodiment, the first device 4100a and the second device 4200a may be directly connected to each other through a communication network, but are not limited thereto. In an embodiment, the first device 4100a and the second device 4200a may be connected to the voice assistant server 2000 (refer to FIG. 3 ), and may be connected to each other through the voice assistant server 2000.

Referring to FIG. 12A, a first device 4100a is a listener device that receives a voice input from a user, and the second device 4200b determines a device for performing an operation by analyzing the voice input, and the determined device It may be a hub device that controls to perform an operation. For example, the first device 4100a may be an air conditioner, and the second device 4200a may be a TV.

The first device 4100a may be a listener device that receives a voice input composed of speech from a user. The listener device may be a device that only receives a voice input from a user, but is not limited thereto. In one embodiment, the listener device may be an operation performing device that receives a control command from the hub device (in the embodiment shown in FIG. 12A, the second device 4200a) and performs an operation related to a specific function.

In an embodiment, the listener device may receive a voice input related to a function performed by the listener device from a user. For example, the first device 4100a receives a voice input such as “Take the air conditioner temperature down to 20° C.” from the user (step ①).

In an embodiment, the first device 4100a may obtain a speech signal by converting sound received through a microphone into an acoustic signal and removing noise (eg, a non-speech component) from the acoustic signal.

The first device 4100a transmits a voice signal to the second device 4200a, which is a hub device (step ②).

The second device 4200a, which is a hub device, receives a voice signal from the first device 4100a, converts the voice signal to text, and interprets the text to determine the listener device as an operation performing device (step ③). In an embodiment, the second device 4200a may convert a speech signal into text by performing Automatic Speech Recognition (ASR) using data of an ASR module previously stored in a memory. The second device 4200a may detect an intent from the text and determine a device that performs an operation corresponding to the detected intent by analyzing the text using the device determination model previously stored in the memory. In the embodiment illustrated in FIG. 12A, the second device 4200a may determine the first device 4100a as the device performing the operation.

The function determination model corresponding to the operation performing device determined by the second device 4200a as the hub device is previously stored in the memory of the hub device, or is stored in the first device 4100a itself determined as the operation performing device, or It may be stored in the memory 2300 (see FIG. 3) of the voice assistant server 2000 (see FIG. 3). The function determination model corresponding to each device is a model used by the device to obtain detailed operations for performing an operation according to the determined function and operation information regarding a relationship between the detailed operations.

In the embodiment illustrated in FIG. 12A, the function determination model corresponding to the first device 4100a may be pre-stored in the memory of the first device 4100a. For example, the first device 4100a may store an air conditioner function determination model 4132 (refer to FIG. 5) used to obtain detailed operations corresponding to the air conditioner and operation information about a relationship between the detailed operations.

The second device 4200a, which is a hub device, provides the text to the function determination model of the listener device (step 4).

The first device 4100a executes an operation by interpreting the received text (step 5). In one embodiment, the first device 4100a analyzes at least a part of the text using the natural language understanding model 4134 (refer to FIG. 5) included in the function determination model 4132 of the air conditioner, and Operation information regarding an operation to be performed by the first device 4100a may be obtained based on the analysis result. The function determination model 4132 is an action plan management module 4136 (FIG. 5) that manages operation information related to a detailed operation of a device in order to generate detailed operations to be performed by the first device 4100a and an execution order of the detailed operations. Reference). The action plan management module 4136 may manage detailed operations executed by the first device 4100a and operation information regarding a relationship between the detailed operations. The action plan management module 4136 may plan detailed operations to be performed by the first device 4100a and an execution order of the detailed operations based on an analysis result of at least a part of the text. The first device 4100a may plan detailed operations and an execution order of the detailed operations based on an analysis result of at least a portion of the text by the function determination model 4132, and execute the operation based on the planning result.

12B is a conceptual diagram illustrating an operation of a hub device and a second device according to the present disclosure.

Arrows shown in FIG. 12B indicate movement, transmission, and reception of data including text and voice signals between the first device 4100b and the second device 4200b. The numbers in the original letters indicate the order in which each action is performed.

Like FIG. 12A, the first device 4100b and the second device 4200b illustrated in FIG. 12B are interconnected through wired communication or wireless communication, and may perform data communication.

Referring to FIG. 12B, the first device 4100b is a listener device that receives a voice input from a user and analyzes the received voice input to determine a device to perform an operation and control the determined device to perform the operation. It may be a hub device.

In the embodiment shown in FIG. 12B, the first device 4100b receives a voice input such as “Turn down the air conditioner temperature to 20°C” from the user (step ①).

The first device 4100b determines the hub device as the operation performing device by analyzing the voice input, and executes the operation (step 2). As a hub device, the first device 4100b converts voice input into text and interprets the converted text to determine an operation performing device, obtains operation information for the operation performing device to perform an operation, and obtains the operation. Since the method of executing the operation based on the information is the same as the method of the first device 4100a described with reference to FIG. 12A, a duplicate description will be omitted.

In the embodiment shown in FIGS. 12A and 12B, the device for performing an operation related to the intent to lower the set temperature of the air conditioner, for example, the ignition intention that the user intends to execute through a specific device, are all first devices 4100a and 4100b. It is the same as ). However, in the embodiment shown in FIG. 12A, since the first device 4100a is a listener device, it is not possible to directly analyze the voice input to determine the operation performing device. Therefore, a voice signal is transmitted to the second device 4200a, which is a hub device. Then, a further process of receiving the text from the second device 4200a is required. In the embodiment shown in FIG. 12B, since the first device 4100b is a hub device, unnecessary processes such as transmitting a voice signal to the second device 4200b or receiving a text from the second device 4200b are omitted. . Accordingly, in the embodiment shown in FIG. 12B, the latency may be less than that in the embodiment shown in FIG. 12A, and the processing time for the utterance may take less than that in the embodiment of FIG. 12A. The'processing time' refers to the time from the time when the user's utterance is received until the operation execution is completed. In an embodiment, the processing time of the utterance may mean a time from the time point when the user's utterance is received to the time point at which the operation execution completion message is output.

In an embodiment of the present disclosure, a device that a user uses more often and takes a shorter time to process an utterance among a plurality of devices may be determined as the hub device. In addition, when there is a device that is used by a user more often than a device that is currently used as a hub device and requires a shorter processing time for ignition, the hub device can be changed by replacing the device with a hub device. In the embodiment shown in FIGS. 12A and 12B, the user uses an air conditioner in the summer, and thus the frequency of ignition related to the air conditioner control increases. Considering the frequency of use and the processing time of ignition, the second device that is a TV When the hub device is changed to the first device 4100b, which is an air conditioner, instead of using 4200a as the hub device (see FIG. 12B), the latency can be shortened. In addition, the hub device can be replaced with another device in an exceptional situation where the hub device cannot operate normally, such as when the power of the device currently being used as a hub device is turned off or the network connection is disconnected. You can respond flexibly.

In an embodiment of the present disclosure, the hub device periodically acquires information on the frequency of use and processing time of utterances of each of the plurality of devices according to a preset time interval, and uses the obtained information to be used as the current hub device. The hub device can be changed periodically by comparing the frequency of use of the device and the processing time of the ignition.

13 is a block diagram illustrating components of a hub device 1002 according to an embodiment of the present disclosure.

The hub device 1002 is a device that receives a user's voice input and controls at least one of the plurality of devices 4000 based on the received voice input. The hub device 1002 may be a listener device that receives a voice input from a user.

Referring to FIG. 13, the hub device 1002 may be configured to include at least a microphone 1100, a processor 1200, a memory 1300, and a communication interface 1400. Among the constituent elements included in the hub device 1002, the microphone 1100, the processor 1200, and the communication interface 1400 are the same as the constituent elements shown in FIG.

The memory 1300 includes data about the ASR module 1310, data about the NLG module 1320, data about the TTS module 1330, data about the device determination model 1340, and the resource tracking module 1350. Data, data about the function determination model management module 1360, a plurality of function determination models 1370, and data about the hub device determination module 1380 may be stored. ASR module 1310, NLG module 1320, TTS module 1330, device determination model 1340, resource tracking module 1350, function determination model management module 1360, and a plurality of function determination models 1370 Since each is the same as the configuration shown in FIG. 2, redundant descriptions will be omitted.

The hub device determination module 1380 includes a plurality of devices 4000 previously registered in the IoT server 3000 (refer to FIG. 4) according to the hub device 1002 and the user account logged in the hub device 1002. 5) is a module configured to determine any one device as the hub device. The hub device determination module 1380 selects at least one hub device candidate from among the plurality of devices 4000, and selects at least one hub device candidate based on usage history information and performance information of each of the at least one hub device candidate. By determining any one of the hub device candidates, and replacing the current hub device 1002 with the determined device, data, instructions, or program code configured to change the hub device may be included.

The processor 1200 may select at least one hub device candidate from among the plurality of devices 4000 using data or program code about the hub device determination module 1380. In one embodiment, the processor 1200 uses data or program code for the hub device determination module 1380 to determine whether or not the plurality of devices 4000 are always powered, computing power, and power consumption. At least one of the plurality of devices 4000 may be obtained, and at least one hub device candidate may be determined based on at least one of whether power is always supplied, computing power, and power consumption.

The processor 1200 acquires usage history information and performance information of each of the at least one hub device candidate by using data or program code about the hub device determination module 1380, and based on the obtained usage history information and performance information. Thus, any one device among at least one hub device candidate may be determined. The processor 1200 may change the hub device by replacing the current hub device 1002 with the determined device. For example, if the current hub device 1002 is a TV, the processor 1200 determines an air conditioner among the plurality of devices 4000 as a new hub device using data on the hub device determination module 1380, and By replacing the TV, which is the hub device 1002, with an air conditioner, the hub device 1002 can be changed.

The hub device determination module 1380 may include a usage history log database 1382 and a device performance history database 1384. In FIG. 13, the usage history log database 1382 and the device performance history database 1384 are shown to be included in the hub device determination module 1380, but are not limited thereto. In an embodiment, the usage history log database 1382 and the device performance history database 1384 are not included in the hub device determination module 1380 and may be stored in the memory 1300.

In another embodiment, at least one of the usage history log database 1382 and the device performance history database 1384 is not included in the hub device 1002 and may be configured as an external database. In this case, the hub device 1002 may access at least one of the usage history log database 1382 and the device performance history database 1384 through a network.

The usage history log database 1382 is a database that stores information on the frequency of use and recent usage history of each of the hub device 1002 and the plurality of devices 4000 in the form of a log. The usage history log database 1382 includes, for example, the frequency of each of the hub device 1002 and the plurality of devices 4000 used as a listener device by a user, the frequency used as an executer device, and a listener device. At least one of a recent use history used as a device and a recent use history used as an operation performing device may be stored in a log format. The usage history log database 1382 may store, for example, a voice input frequently received from a user or text converted from a voice input.

The device performace history log database 1384 is a database that stores performance history information of each of the hub device 1002 and the plurality of devices 4000 in the form of a log. The device performance history database 1384 may store, for example, data about the hub device 1002 and the utterance processing time of each of the plurality of devices in a log format. The'processing time' refers to the time from the time when the device receives the user's speech to the time when the operation is completed. In an embodiment, the processing time of the utterance may mean a time from the time point when the user's utterance is received to the time point at which the operation execution completion message is output. The ignition processing time is inversely proportional to the device's performance. For example, the shorter the ignition processing time, the better the device's performance.

In an embodiment, the hub device 1002 may obtain information on a simulation result of the speech processing time from each of the plurality of devices 4000. In one embodiment, each of the plurality of devices 4000 interprets an example text such as an example of the user's ignition, for example, "Take the air conditioner temperature down to 20°" or "Play the movie Avengers on TV". Determine the operation performing device, obtain operation information on the operation to be performed by the determined operation execution device, generate a control command using the obtained operation information, and execute the operation based on the generated control command. It is possible to simulate the ignition processing time required until. In one embodiment, each of the plurality of devices 4000 may simulate an ignition processing time through a background process. 'Background process' refers to a process that is not an operation by a user, but a process that is executed in a background environment that is not recognized by the user by the device itself. Each of the plurality of devices 4000 may transmit the simulation result to the hub device 1002. The hub device 1002 may store a simulation result of the speech processing time acquired from each of the plurality of devices 4000 in the device performance history database 1384.

The processor 1200 analyzes the usage history log database 1382 using data or program code of the hub device determination module 1380, thereby providing information on the frequency of use of each of the hub device 1002 and at least one device candidate. Can be obtained. In addition, the processor 1200 analyzes the device performance history database 1384 using the data or program code of the hub device determination module 1380 to process the utterance of each of the hub device 1002 and at least one hub device candidate. You can obtain information about time. The processor 1200 may determine a device to replace the current hub device 1002 based on the frequency information and speech processing time information of each of the hub device 1002 and at least one hub device candidate. The processor 1200 uses the data or program code of the hub device determination module 1380 to replace the first device currently used as the hub device 1002 with a newly determined second device, thereby removing the hub device 1002. It can be changed to 2 devices. An embodiment in which the processor 1200 changes the hub device 1002 based on the information on the frequency of use and the information on the speech processing time will be described in detail with reference to FIG. 15.

However, the processor 1200 of the present disclosure is not limited to changing the hub device 1002 based on the information on the frequency of use and the information on the speech processing time. In an embodiment, the hub device 1002 may receive an input of a user selecting one of the plurality of devices 4000 and may determine the selected device as the hub device based on the received user input. For example, the hub device 1002 receives a user's voice input for selecting any one of the plurality of devices 4000 through the microphone 1100, and the processor 1200 receives data from the ASR module 1310. A device that the user intends to change may be identified by converting the voice input into text using and interpreting the text using data of the first natural language understanding model 1342. The processor 1200 may change the hub device to the identified device.

14 is a flowchart illustrating a method of operating the hub device 1002 according to an embodiment of the present disclosure.

In step S1410, the hub device 1002 selects at least one hub device candidates from among a plurality of devices 4000 (refer to FIG. 5) previously registered in the user account. In one embodiment, the hub device 1002 is pre-registered in the IoT server 3000 (see FIG. 4) according to a user account logged in to the hub device 1002, and through a network with the hub device 1002. Acquires at least one information of whether or not the connected plurality of devices 4000 are always supplied with power, computing power, and power consumption, and is selected from among whether or not the connected devices 4000 are always supplied with power, computing power, and power consumption. At least one hub device candidate among the plurality of devices 4000 may be determined based on at least one.

In step S1420, the hub device 1002 determines any one of the at least one hub device candidate based on the usage history information and the performance information of each of the at least one hub device candidate.

In one embodiment, the hub device 1002 analyzes the usage history log database 1382 (refer to FIG. 13) stored in the memory 1300 (see FIG. 13) to provide the hub device 1002 and at least one hub device candidate. Information on the frequency of use of each can be obtained. The hub device 1002 is from the usage history log database 1382, for example, the frequency of each of the hub device 1002 and at least one hub device candidate used as a listener device by a user, an executer device At least one of the use history information of the frequency used as, the recent use history used as the listener device, and the recent use history used as the operation performing device may be obtained.

In one embodiment, the hub device 1002 analyzes the device performance history database 1384 (refer to FIG. 13) stored in the memory 1300 to provide the hub device 1002 and the performance history of each of the at least one hub device candidate. Information can be obtained. The hub device 1002 may obtain, for example, information on the utterance processing time of each of the hub device 1002 and at least one hub device candidate from the device performance history database 1384. The'processing time' refers to the time from the time when the user's utterance is received until the execution of the operation is completed. In an embodiment, the processing time of the utterance may mean a time from the time point when the user's utterance is received to the time point at which the operation execution completion message is output.

In one embodiment, the hub device 1002 compares the frequency of use and the speech processing time information of the at least one hub device candidate with the frequency of use and the speech processing time of the current hub device 1002, so that at least one hub device candidate One of the devices to replace the hub device 1002 may be determined. For example, the hub device 1002 compares the maximum value of the frequency of use of each of the at least one hub device candidate with the current frequency of use of the hub device 1002, and the utterance processing time of each of the at least one hub device candidate Any one of the at least one hub device candidate may be determined by comparing the minimum value with the current processing time of the hub device 1002. A detailed method of determining any one of the at least one hub device candidate will be described in detail through steps S1510 to S1550 shown in FIG. 15.

In step S1430, the hub device 1002 changes the hub device by replacing the current hub device 1002 with the determined device. For example, if the current hub device 1002 is a TV, the hub device 1002 may change the hub device 1002 by replacing the TV with an air conditioner determined from among the hub device candidates.

15 is a flowchart illustrating a method of operating the hub device 1002 according to an embodiment of the present disclosure.

Steps S1510 to S1550 shown in FIG. 15 are exemplary embodiments in which step S1420 of FIG. 14 is embodied. Step S1510 shown in FIG. 15 is performed after step S1410 of FIG. 14 is performed.

In step S1510, the hub device 1002 obtains information on the frequency of use of the hub device 1002 and each of the at least one hub device candidate by analyzing the usage history log database 1382 (refer to FIG. 13). In one embodiment, the processor 1200 of the hub device 1002 (see FIG. 13) accesses the usage history log database 1382 stored in the memory 1300 (see FIG. 13), and the usage history log database The frequency of use including at least one of the frequency of each of the hub device 1002 and the at least one hub device candidate used as a listener device by a user by analyzing 1382, and a frequency used as an executer device You can obtain information about. In addition, the processor 1200 analyzes the usage history log database 1382, so that the hub device 1002 and at least one hub device candidate each have a recent usage history used as a listener device, and a recent usage history used as an operation performing device. It is also possible to obtain information on the recent use history including at least one of.

In step S1520, the hub device 1002 compares the frequency of use of the current hub device 1002 with a maximum value among the use frequencies of each of the at least one hub device candidate, and the frequency of use of the hub device 1002 is at least one hub. It is identified whether or not the device candidate is less than the maximum value among the frequency of use of the candidate device. When there is only one hub device candidate, the hub device 1002 may check whether the frequency of use of the device currently used as the hub device 1002 is less than the frequency of use of the hub device candidate.

When it is determined that the frequency of use of the device currently used as the hub device 1002 is less than the maximum value among the frequency of use of at least one hub device candidate (step S1530), the hub device 1002 includes the device performance history database 1384, 13), information on the processing time of the user of the hub device 1002 and each of the at least one hub device candidate is obtained. In one embodiment, the processor 1200 accesses the device performance history database 1384 stored in the memory 1300 and analyzes the device performance history database 1384 to provide the hub device 1002 and at least one Information on the speech processing time of each of the hub device candidates may be obtained. The'processing time' refers to the time from the time when the device receives the user's speech to the time when the operation is completed. In an embodiment, the processing time of the utterance may mean a time from the time point when the user's utterance is received to the time point at which the operation execution completion message is output. The ignition processing time is inversely proportional to the device's performance. For example, the shorter the ignition treatment time, the better the device's performance.

In step S1540, the hub device 1002 compares the ignition processing time of the device currently used as the hub device 1002 with a minimum value of ignition processing time of each of the at least one hub device candidate, thereby igniting the current hub device 1002. It is checked whether the processing time is greater than or equal to the minimum of the speech processing time of at least one hub device candidate. When there is only one hub device candidate, the hub device 1002 may check whether the ignition processing time of the device currently used as the hub device 1002 takes longer than the utterance processing time of the hub device candidate.

When it is determined that the ignition processing time of the device currently used as the hub device 1002 takes longer than the minimum ignition processing time of at least one hub device candidate (step S1550), the hub device 1002 is at least one hub device. A device to replace the hub device 1002 is determined based on the use frequency information and the speech processing time information among the candidates. In one embodiment, the hub device 1002 selects a device that is used as at least one of a listener device and an operation performing device among the at least one hub device candidate, which has the highest frequency and takes the least amount of time to process an utterance. It can be decided as the device to be replaced. In one embodiment, the hub device 1002 considers the frequency of use and the ignition processing time together, but may determine a device to replace the hub device 1002 with a large weight on the frequency of use. For example, the hub device 1002 may first select one or more devices having a high frequency of use among at least one hub device candidate, and then determine a device having the fastest utterance processing time among the selected one or more devices.

However, the present invention is not limited thereto, and the hub device 1002 first selects one or more devices whose ignition processing time is less than a preset threshold time among at least one hub device candidate, and the frequency of use is the highest among the selected one or more devices. You can also decide on many devices.

In step S1430, the hub device 1002 changes the hub device by replacing the device currently used as the hub device 1002 with the determined device.

In step S1520, when the frequency of use of the current hub device is greater than the maximum value of the use frequency of at least one hub device candidate (step S1440), the device currently used as the hub device 1002 is maintained as the hub device. Similarly, in step S1540, when the current utterance processing time of the hub device 1002 is less than or equal to the minimum value of utterance processing time of at least one hub device candidate (step S1440), the device currently used as the hub device 1002 continues as the hub device. maintain. If the frequency of use of the device currently used as the hub device 1002 is more than the frequency of use of other devices, and the ignition processing time of the device currently used as the hub device 1002 is less than the ignition processing time of other devices, Since there is no need to change the hub device, the device used as the hub device 1002 is kept.

16 is a flowchart illustrating a method of operating the hub device 1002 according to an embodiment of the present disclosure.

Steps S1610 to S1690 shown in FIG. 16 are steps performed between steps S610 and S620 shown in FIG. 6. Step S1610 is the same step as step S610 shown in FIG. 6. After step S1630, step S1652, step S1680, and step S1682 are performed, step S620 is performed.

In step S1610, the hub device 1002 receives device information of a new device registered in the user account from the IoT server 3000 (refer to FIG. 4). The new device is newly acquired by a user of the hub device 1002 through purchase or ownership transfer, and logs in with the same user account as the hub device 1002, so that the device registered in the IoT server 3000 is accessed. it means. The device information of the new device received by the hub device 1002 includes, for example, identification information of the new device (eg, device id information), the device type of the new device, and capability information of the new device. And at least one of location information. In an embodiment, the hub device 1000 may receive power on/off of a new device or status information regarding an operation currently being executed.

The hub device 1002 may receive device information of a new device from the IoT server 3000. However, the present invention is not limited thereto, and the hub device 1002 may receive device information of a new device from the voice assistant server 2000 (refer to FIG. 2 ).

In step S1620, the hub device 1002 identifies whether a hub device exists among a plurality of devices previously registered in the user account. In one embodiment, the hub device 1002 is a voice assistant server based on resource status information among a plurality of devices previously registered in the IoT server 3000 according to the same user account as the user account logged in to the hub device 1002. 2000, see Fig. 3). In an embodiment, the hub device may be a device capable of determining a device for performing an operation by including a device determination model 1340 (refer to FIG. 2) among a plurality of devices and interpreting a voice input received from a user.

When it is confirmed that the hub device does not exist among the plurality of devices (step S1630), a new device is determined as the hub device 1002.

When it is confirmed that the hub device 1002 exists among the plurality of devices (step S1640), the hub device 1002 is connected to the hub based on the constant power connection, computing power, and power consumption among the plurality of devices previously registered in the user account. Select a device candidate. In an embodiment, the hub device 1002 may determine one or more hub device candidates from among a plurality of devices. Step S1640 is the same step as step S1410 shown in FIG. 14, and redundant descriptions will be omitted.

In step S1650, the hub device 1002 identifies whether or not a voice input requesting change of the hub device is received from the user. For example, if the device currently used as the hub device 1002 is a TV, the hub device 1002 may receive a voice input requesting a hub device change such as "Change the air conditioner to a hub device." have. However, it is not limited to the reception of the voice input, and the hub device 1002 replaces the device currently used as the hub device 1002 with a new device to change the hub device by using a touch input, or an operation device such as a mouse or keyboard. It is also possible to receive an input through the user.

When receiving a voice input requesting change of the hub device (step S1652), the hub device 1002 determines a device to replace the hub device 1002 based on the received voice input. The hub device 1002 converts the voice input into text and interprets the text using a natural language understanding model to identify the name or type of the device that the user wants to change, and replace the identified device with the hub device 1002. It can be decided as a device to do. In one embodiment, the processor 1200 (see FIG. 13) of the hub device 1002 converts the received voice input into text using data or program code for the ASR module 1310 (see FIG. 13), and 1 Parse text into words or phrases using data or program code related to the natural language understanding model (1342 (see Fig. 13)), and a word specifying the device name, common name, or installation location, or Recognize phrases and identify devices from recognized words or phrases. The hub device 1002 may determine the identified device as a device to replace the hub device 1002.

When a voice input requesting a change of the hub device is not received from the user (step S1660), the hub device 1002 analyzes the device performance history database 1384 (refer to FIG. 13), and thus the hub device 1002 and at least one Acquires information about a user's speech processing time of each of the hub device candidates of.

In step S1670, the hub device 1002 compares the ignition processing time of the device currently used as the hub device 1002 and the ignition processing time of each of the at least one hub device candidate, thereby igniting the current hub device 1002. It is checked whether the processing time takes longer than the minimum value of the speech processing time of at least one hub device candidate.

Step S1660 and step S1670 are the same as steps S1530 and S1540 shown in FIG. 15, respectively, and redundant descriptions will be omitted.

When the ignition processing time of the device currently used as the hub device 1002 is less than or equal to the minimum ignition processing time of at least one hub device candidate (step S1680), the device that is the current hub device 1002 is determined as the hub device 1002 do.

When it is determined that the ignition processing time of the device currently used as the hub device 1002 takes longer than the minimum ignition processing time of at least one hub device candidate (step S1682), the hub device 1002 is A device to replace the hub device 1002 is determined based on the speech processing time information among the device candidates. In an embodiment, the hub device 1002 may determine a device having the smallest speech processing time among at least one hub device candidate as a device to replace the hub device 1002.

In step S620, the determined hub device 1002 requests the voice assistant server 2000 to update the device determination model 1340 (refer to FIG. 13) stored in the hub device 1002.

17 is a diagram illustrating a network environment including a hub device 1000, a plurality of devices 4000, and a voice assistant server 2000.

Referring to FIG. 17, a hub device 1000, a plurality of devices 4000, a voice assistant server 2000, and an IoT server 3000 may be interconnected through wired communication or wireless communication, and may perform communication. . In an embodiment, the hub device 1000 and the plurality of devices 4000 may be directly connected to each other through a communication network, but are not limited thereto.

The hub device 1000 and the plurality of devices 4000 may be connected to the voice assistant server 2000, and the hub device 1000 may be connected to the plurality of devices 4000 through a server. In addition, the hub device 1000 and the plurality of devices 4000 may be connected to the IoT server 3000. In another embodiment, each of the hub device 1000 and the plurality of devices 4000 may be connected to the voice assistant server 2000 through a communication network, and may be connected to an external IoT server 3000 through the voice assistant server 2000. have. In another embodiment, the hub device 1000 may be connected to a plurality of devices 4000 and the hub device 1000 may be connected to a plurality of devices 4000, and the hub device 1000 may be one or a plurality of access points located adjacent to each other. It may be connected to the plurality of devices 4000 through the (Access point). In addition, the hub device 1000 may be connected to a plurality of devices 4000 while maintaining a state of being connected to the voice assistant server 2000 or the IoT server 3000.

The hub device 1000, the plurality of devices 4000, the voice assistant server 2000, and the IoT server 3000 include a local area network (LAN), a wide area network (WAN), and a value-added communication network. It may be connected through an Added Network (VAN), a mobile radio communication network, a satellite communication network, or a combination thereof. Wireless communication methods include, for example, wireless LAN (Wi-Fi), Bluetooth, Bluetooth low energy, ZigBee, Wi-Fi Direct (WFD), ultra wideband (UWB), infrared communication (IrDA, infrared data). Association), NFC (Near Field Communication), and the like, but are not limited thereto.

In an embodiment, the hub device 1000 may receive a user's voice input. At least one of the plurality of devices 4000 may be a target device that performs a specific operation by receiving a control command from the voice assistant server 2000 and/or the IoT server 3000. At least one of the plurality of devices 4000 may be controlled to perform a specific operation based on a user's voice input received by the hub device 1000. In one embodiment, at least one of the plurality of devices 4000 may receive a control command from the hub device 1000 without receiving a control command from the voice assistant server 2000 and/or the IoT server 3000. .

The hub device 1000 may receive a voice input (eg, speech) from a user. In one embodiment, the hub device 1000 may include an ASR model. In one embodiment, the hub device 1000 may include an ASR model with limited functionality. For example, the hub device 1000 has a function of detecting a designated voice input (for example, a wake-up input such as'Hi Bixby','OK Google', etc.) or pre-processing a voice signal obtained from some voice inputs. It can include an ASR model that has a function. In FIG. 17, the hub device 1000 is illustrated as an AI speaker, but is not limited thereto. In an embodiment, any one device among the plurality of devices 4000 may be the hub device 1000. In addition, the hub device 1000 may include a first natural language understanding model, a second natural language understanding model, and a natural language generation model. In this case, the hub device 1000 may receive a user's voice input through a microphone, or may receive a user's voice input from at least one of the plurality of devices 4000. When the user's voice input is received, the hub device 1000 processes the user's voice input through an ASR model, a first natural language understanding model, a second natural language understanding model, and a natural language generation model, and responds to the user's voice input. Can provide.

The hub device 1000 may determine a type of a target device to perform an operation intended by the user based on the received voice signal. The hub device 1000 may receive a voice signal that is an analog signal and perform Automatic Speech Recognition (ASR) to convert the voice part into computer-readable text. The hub device 1000 may analyze the converted text using a first natural language understanding (NLU) model and determine a target device based on the analysis result. The hub device 1000 may determine at least one of the plurality of devices 4000 as a target device. The hub device 1000 may select a second natural language understanding model corresponding to the determined target device from among a plurality of stored second natural language understanding models. The hub device 1000 may determine an operation to be performed by the target device requested by the user by using the selected second natural language understanding model. When it is determined that there is no second natural language understanding model corresponding to the determined target device among the plurality of stored second natural language understanding models, the hub device 1000 transmits at least part of the converted text to the voice assistant server 2000 or a plurality of Can be transmitted to at least one of the devices 4000. The hub device 1000 transmits information on the determined operation to the target device so that the determined target device can execute the determined operation.

The hub device 1000 may receive information on a plurality of devices 4000 from the IoT server 3000. The hub device 1000 may determine a target device by using the received information of the plurality of devices 4000. In addition, the hub device 1000 may use the IoT server 3000 as a relay server for transmitting information on the determined operation to control the target device so that the target device can execute the determined operation.

The hub device 1000 may receive a user's voice input through a microphone and transmit the received voice input to the voice assistant server 2000. In an embodiment, the hub device 1000 may obtain a voice signal from the received voice input and transmit the voice signal to the voice assistant server 2000.

In the embodiment illustrated in FIG. 17, the plurality of devices 4000 include, but are not limited to, an air conditioner 4100, a TV 4200, a washing machine 4300, and a refrigerator 4400. For example, the plurality of devices 4000 includes a smartphone, a tablet PC, a mobile phone, a video phone, an e-book reader, and a desktop personal computer. computer), laptop personal computer, netbook computer, workstation, server, personal digital assistant (PDA), portable multimedia player (PMP), MP3 player, mobile medical device, camera ), or a wearable device. In one embodiment, the plurality of devices 4000 may be home appliances. Home appliances include, for example, televisions, digital video disk (DVD) players, audio, refrigerators, air conditioners, vacuum cleaners, ovens, microwave ovens, washing machines, air cleaners, set-top boxes, and home automation controls. It may include at least one of a home automation control panel, a security control panel, a game console, an electronic key, a camcorder, or an electronic frame.

The voice assistant server 2000 may determine a type of a target device to perform an operation intended by the user based on the received voice signal. The voice assistant server 2000 may receive a voice signal, which is an analog signal, from the hub device 1000 and convert the voice part into computer-readable text by performing ASR (Automatic Speech Recognition). The voice assistant server 2000 may analyze the converted text using a first natural language understanding (NLU) model and determine a target device based on the analysis result. Also, the voice assistant server 2000 may receive at least a part of the text from the hub device 1000 and information on a target device determined by the hub device 1000. In this case, the hub device 1000 converts the user's voice signal into text using the ASR model and the first natural language understanding model of the hub device 1000 and analyzes the converted text to determine a target device. In addition, the hub device 1000 transmits at least a part of the converted text and information on the determined target device to the voice assistant server 2000.

The voice assistant server 2000 may determine an operation to be performed by the target device requested by the user by using the second natural language understanding model corresponding to the determined target device. The voice assistant server 2000 may receive information on a plurality of devices 4000 from the IoT server 3000. The voice assistant server 2000 may determine a target device by using the received information of the plurality of devices 4000. In addition, the voice assistant server 2000 uses the IoT server 3000 as a relay server for transmitting information on the determined operation, and configures the IoT server 3000 and the target device so that the target device can execute the determined operation. Can be controlled. The IoT server 3000 may store information on a plurality of pre-registered devices 4000 connected through a network. In one embodiment, the IoT server 3000 performs identification information (eg, device ID information) of a plurality of devices 4000, a device type of each of the plurality of devices 4000, and a function of each of the plurality of devices 4000 At least one of capability information may be stored.

In an embodiment, the IoT server 3000 may store power on/off of each of the plurality of devices 4000 or state information regarding an operation being executed. The IoT server 3000 may transmit a control command for executing the determined operation to a target device among the plurality of devices 4000. The IoT server 3000 may receive information on the determined target device and information on the determined operation from the voice assistant server 2000, and transmit a control command to the target device based on the received information.

18A and 18B are diagrams illustrating an embodiment of a voice assistant model 200 executable in the hub device 1000 and the voice assistant server 2000 of the present disclosure.

18A and 18B, the voice assistant model 200 is implemented in software. The voice assistant model 200 may be configured to determine a user's intention from the user's voice input and control a target device related to the user's intention. The voice assistant model 200 includes a first assistant model 200a configured to update an existing model to a new model through training, etc., when a device controlled through the voice assistant model 200 is added, and the added device. It may include a second assistant model 200b configured to add the corresponding model to the existing model.

The first assistant model 200a is a model for determining a target device related to a user intention by analyzing a user's voice input. The first assistant model 200a may include an ASR model 202, an NLG model 204, a first natural language understanding model 300a, and a device determination model 310. In an embodiment, the device determination model 310 may be configured to include the first natural language understanding model 300a. In another embodiment, the device determination model 310 and the first natural language understanding model 300a may be configured as separate components.

The device determination model 310 is a model for performing an operation of determining a target device using the analysis result of the first natural language understanding model 300a. The device determination model 310 may include a plurality of detailed models, and one of the plurality of detailed models may be the first natural language understanding model 300a. The first natural language understanding model 300a or the device determination model 310 may be an artificial intelligence model.

When a device controlled through the voice assistant model 200 is added, the first assistant model 200a may update at least the device determination model 310 and the first natural language understanding model 300a through learning. Here, learning may mean learning by using both the training data used to train the existing device determination model and the first natural language understanding model and additional training data related to the added device. Further, learning may mean updating the device determination model and the first natural language understanding model using only additional learning data related to the added device.

The second assistant model 200b is a model specialized for a specific device, and is a model that determines an operation to be performed by a target device corresponding to a user's voice input among a plurality of operations that a specific device can perform. In FIG. 18A, the second assistant model 200b may include a plurality of second natural language understanding models 300b, an NLG model 206, and an action plan management model 210. The plurality of natural language understanding models 300b may correspond to a plurality of different devices, respectively. The second natural language understanding model, NLG model, and action plan management model may be a model implemented as a rule based system. In one embodiment, the second natural language understanding model, the NLG model, and the action plan management model may be artificial intelligence models. Each of the plurality of second natural language understanding models may be configured as a part of each of the plurality of function determination models.

When a device controlled through the voice assistant model 200 is added, the second assistant model 200b may be configured to add a second natural language understanding model corresponding to the added device. That is, the second assistant model 200b may be configured to further include a second natural language understanding model corresponding to a device added to the plurality of existing second natural language understanding models 300b. In this case, the second assistant model 200b is a plurality of second natural language understanding models including a second natural language understanding model added using information on the target device determined through the first assistant model 200a, It may be configured to select a second natural language understanding model corresponding to the determined target device.

Referring to FIG. 18B, the second assistant model 200b may include a plurality of action plan management models and a plurality of NLG models. 18B, each of the plurality of second natural language understanding models included in the second assistant model 200b corresponds to the second natural language understanding model 300b of FIG. 2A, and includes a plurality of second natural language understanding models included in the second assistant model 200b. Each of the NLG models corresponds to the NLG model 206 of FIG. 2A, and each of the plurality of action plan management models included in the second assistant model 200b may correspond to the action plan management model 210 of FIG. 2A.

In FIG. 18B, each of the plurality of action plan management models may be configured to correspond to each of the plurality of second natural language understanding models. In addition, each of the plurality of NLG models may be configured to correspond to each of the plurality of second natural language understanding models. In another embodiment, one NLG model may correspond to a plurality of second natural language understanding models, and one action plan management model may be configured to correspond to a plurality of second natural language understanding models.

In FIG. 18B, when a device controlled through the voice assistant model 200 is added, the second assistant model 200b adds a second natural language understanding model, an NLG model, and an action plan management model corresponding to the added device. Can be configured to

18B, the first natural language understanding model 300a may be configured to be updated to a new model through learning or the like when a device controlled through the voice assistant 200 is added. In addition, when the device determination model 310 is configured to include the first natural language understanding model 300a, the device determination model 310 is an existing model through learning when a device controlled through the voice assistant 200 is added. The whole can be configured to be updated with the new model. The first natural language understanding model 300a or the device determination model 310 may be an artificial intelligence model. Here, learning may mean learning by using both the training data used to train the existing device determination model and the first natural language understanding model and additional training data related to the added device. Further, learning may mean updating the device determination model and the first natural language understanding model using only additional learning data related to the added device.

18B, the second assistant model 200b includes a second natural language understanding model, an NLG model, and an action plan management model corresponding to the added device when a device controlled through the voice assistant 200 is added to the existing model. It can be updated by adding. The second natural language understanding model, NLG model, and action plan management model may be a model implemented as a rule based system.

In the embodiment of FIG. 18B, the second natural language understanding model, the NLG model, and the action plan management model may be an artificial intelligence model. Each of the second natural language understanding model, the NLG model, and the action plan management model may be managed as one module according to a corresponding device. In this case, the second assistant model 200-b may include a plurality of second assistant models 200b-1, 200b-2, and 200b-3 respectively corresponding to a plurality of devices. For example, the second natural language understanding model corresponding to the TV, the NLG model corresponding to the TV, and the action plan management model corresponding to the TV may be managed as the second assistant model 200b-1 corresponding to the TV. In addition, the second natural language understanding model corresponding to the speaker, the NLG model corresponding to the speaker, and the action plan management model corresponding to the speaker may be managed as the second assistant model 200b-2 corresponding to the speaker. In addition, the second natural language understanding model corresponding to the refrigerator, the NLG model corresponding to the refrigerator, and the action plan management model corresponding to the refrigerator may be managed as the second assistant model 200b-3 corresponding to the refrigerator.

When a device controlled through the voice assistant model 200 is added, the second assistant model 200b may be configured to add a second assistant model corresponding to the added device. That is, the second assistant model 200b may be configured to further include a second assistant model corresponding to a device added to the existing plurality of second assistant models 200b-1 to 3. In this case, the second assistant model 200b is a plurality of second assistant models including a second assistant model corresponding to the added device using information on the target device determined through the first assistant model 200a. Among them, it may be configured to select a second assistant model corresponding to the determined target device.

Programs executed by the hub device 1000, the voice assistant server 2000, and the plurality of devices 4000 described herein are a hardware component, a software component, and/or a hardware component and a software component. It can be implemented in combination. A program can be executed by any system capable of executing computer-readable instructions.

The software may include a computer program, code, instructions, or a combination of one or more of these, configuring the processing unit to operate as desired or processed independently or collectively. You can command the device.

The software may be implemented as a computer program including instructions stored in a computer-readable storage media. Computer-readable recording media include, for example, magnetic storage media (e.g., read-only memory (ROM), random-access memory (RAM), floppy disk, hard disk, etc.) and optical reading media (e.g., CD-ROM (CD-ROM) and DVD (Digital Versatile Disc)). The computer-readable recording medium is distributed over networked computer systems, so that computer-readable codes can be stored and executed in a distributed manner. The medium is readable by a computer, stored in memory, and executed on a processor.

The computer-readable storage medium may be provided in the form of a non-transitory storage medium. Here,'non-transient' means that the storage medium does not contain a signal and is tangible, but does not distinguish between semi-permanent or temporary storage of data in the storage medium.

In addition, a program according to the embodiments disclosed in the present specification may be provided by being included in a computer program product. Computer program products can be traded between sellers and buyers as commodities.

The computer program product may include a software program and a computer-readable storage medium in which the software program is stored. For example, a computer program product is a product (for example, a downloadable application) in the form of a software program that is electronically distributed through a device manufacturer or an electronic market (for example, Google Play Store, App Store). It may include. For electronic distribution, at least a part of the software program may be stored in a storage medium or may be temporarily generated. In this case, the storage medium may be a storage medium of a manufacturer's server, an electronic market server, or a relay server temporarily storing a software program.

The computer program product may include a storage medium of a server or a storage medium of a device in a system composed of a server and a device. Alternatively, when there is a server or a third device (eg, a smartphone) that is communicatively connected to the device, the computer program product may include a storage medium of the third device. Alternatively, the computer program product may include a software program itself transmitted from a server to a device or a third device, or transmitted from a third device to the device.

In this case, one of the server, the device, and the third device may execute the computer program product to perform the method according to the disclosed embodiments. Alternatively, two or more of a server, a device, and a third device may execute a computer program product to distribute and implement the method according to the disclosed embodiments.

For example, a server (eg, an IoT server or a voice assistant server) may execute a computer program product stored in the server, and control a device connected to the server to perform the method according to the disclosed embodiments.

As another example, a third device may execute a computer program product, and a device connected in communication with the third device may be controlled to perform a method according to the disclosed embodiment.

When the third device executes the computer program product, the third device may download the computer program product from the server and execute the downloaded computer program product. Alternatively, the third device may perform a method according to the disclosed embodiments by executing a computer program product provided in a preloaded state.

As described above, although the embodiments have been described by the limited embodiments and drawings, various modifications and variations are possible from the above description to those of ordinary skill in the art. For example, the described techniques are performed in a different order from the described method, and/or components such as a computer system or module described are combined or combined in a form different from the described method, or other components or equivalents Even if substituted or substituted by, appropriate results can be achieved.

Claims (30)

A method for storing a voice assistant model in which a hub device controls a device, the method comprising:
Receiving information on a new device connected to the hub device;
Upon receiving information on the new device, requesting, from a voice assistant server, an update of a device determination model stored in the hub device;
Receiving and storing the updated device determination model from the voice assistant server;
Requesting, from the voice assistant server, information on a function determination model corresponding to the new device;
Receiving information on a function determination model corresponding to the new device from the voice assistant server, and determining whether to store the function determination model in the hub device based on the received information; And
Storing the function determination model corresponding to the new device when it is determined to store the function determination model in the hub device;
Including, the method. The method of claim 1,
When it is determined not to store the function determination model in the hub device, receiving access information for a function determination model corresponding to the new device;
The method further comprising. The method of claim 2,
The access information includes identification information, location information, IP address information (IP address), MAC address (MAC address) of the voice assistant server in which a function determination model corresponding to the new device is stored, and the function determination in the voice assistant server. A method comprising at least one of API information for accessing a model (Application Programming Interface), a language used for the function determination model, and identification information of the new device. The method of claim 1,
The method of determining whether to store the function determination model in the hub device is determined based on a resource state of the hub device and information of a function determination model corresponding to the new device. The method of claim 1,
The hub device is determined by the voice assistant server based on resource state information for each of a plurality of devices pre-registered according to a user account. The method of claim 1,
Selecting at least one hub device candidate from among a plurality of devices previously registered in the IoT server according to a user account logged in to the hub device;
Determining any one of the at least one hub device candidate based on usage history information and performance information of each of the at least one hub device candidate; And
Changing the hub device by replacing the hub device with the determined device;
The method further comprising. The method of claim 6,
Selecting the at least one hub device candidate,
The method of claim 1, wherein the at least one hub device candidate is selected from among the plurality of devices based on at least one of whether the plurality of devices are always powered on, computing power, and power consumption. The method of claim 6,
Determining any one device among the at least one hub device candidate,
Acquiring information on a frequency of use of the hub device and each of the at least one hub device candidate by analyzing a usage history log database stored in the hub device;
Analyzing a performance history log database stored in the hub device, thereby obtaining information on a speech processing time of each user of the hub device and the at least one hub device candidate; And
Determining a device to replace the hub device based on frequency information and speech processing time information of each of the hub device and the at least one hub device candidate;
Including, the method. The method of claim 6,
Selecting the at least one hub device candidate from among the plurality of devices,
The method performed after receiving information on a new device connected to the hub device. The method of claim 6,
Receiving a user input for selecting one of a plurality of devices pre-registered in the IoT server according to a user account logged in to the hub device; And
Determining a selected device as the hub device based on the user input;
The method further comprising. In the hub device (Hub device) for storing a voice assistant model for controlling the device,
A communication interface for performing data communication with at least one of a plurality of devices, a voice assistant server, and an IoT server;
A memory storing a program including one or more instructions; And
A processor that executes one or more instructions of a program stored in the memory;
Including,
The processor,
Receiving information on a new device connected to the hub device, and upon receiving the information on the new device, requests the voice assistant server to update a device determination model previously stored in the memory, and the voice assistant server Controlling the communication interface to receive the updated device determination model from
Storing the received updated device determination model in the memory,
Requesting information on a function determination model corresponding to the new device from the voice assistant server, and controlling the communication interface to receive information on a function determination model corresponding to the new device from the voice assistant server,
Determine whether to store the function determination model in the hub device based on the received information,
When it is determined to store the function determination model in the hub device, storing a function determination model corresponding to the new device in the memory. The method of claim 11,
The processor, when it is determined not to store the function determination model in the hub device, controls the communication interface to receive access information for a function determination model corresponding to the new device. The method of claim 12,
The access information includes identification information, location information, IP address information (IP address), MAC address (MAC address) of the voice assistant server in which a function determination model corresponding to the new device is stored, and the function determination in the voice assistant server. A hub device comprising at least one of API information (Application Programming Interface) for accessing a model, a language used for the function determination model, and identification information of the new device. The method of claim 11,
The processor, based on a resource state of the hub device and information on a function determination model corresponding to the new device, determines whether to store the function determination model in the memory. The method of claim 11,
The hub device is determined by the voice assistant server based on resource state information for each of a plurality of devices previously registered according to a user account. The method of claim 11,
The processor,
Selecting at least one hub device candidate from among a plurality of devices pre-registered with the IoT server according to a user account logged in to the hub device,
Determine any one of the at least one hub device candidate based on usage history information and performance information of each of the at least one hub device candidate,
Changing the hub device by replacing the hub device with the determined device. The method of claim 16,
The processor,
The hub device, wherein the at least one hub device candidate is selected from among the plurality of devices based on at least one of whether each of the plurality of devices is always powered, computing power, and power consumption. The method of claim 16,
A usage history log database for storing information on a frequency of use of the hub device and each of the at least one hub device candidate; And
A performance history log database for storing information on speech processing time of each user of the hub device and the at least one hub device candidate;
The hub device further comprising. The method of claim 18,
The processor,
The hub device and the at least one hub device candidate by analyzing the usage history log database to obtain information on the frequency of use of each of the hub device and the at least one hub device candidate, and by analyzing the performance history log database Obtaining information about each user's utterance processing time,
The hub device to determine a device to replace the hub device based on frequency information and speech processing time information of each of the hub device and the at least one hub device candidate. The method of claim 16,
A voice input unit for receiving a voice input from a user selecting one of a plurality of devices pre-registered in the IoT server according to a user account logged in to the hub device;
Including more,
The processor converts the user's voice input received from the voice input unit into text by performing Automatic Speech Recognition (ASR), and interprets the converted text using a Natural Language Understanding model (NLU). Determining the selected device as the hub device according to the voice input of the device. In a method for storing a voice assistant model in which a voice assistant server and a hub device control a device,
As the hub device receives information on the new device, requesting an update of a device determination model stored in the hub device from the voice assistant server;
The voice assistant server, upon receiving an update request for the device determination model, performing an update on the device determination model stored in the hub device;
The hub device receiving and storing the updated device determination model from the voice assistant server;
The hub device, requesting information on a function determination model corresponding to the new device from the voice assistant server;
Obtaining, by the voice assistant server, information on a function determination model corresponding to the new device and transmitting it to the hub device; And
The hub device receiving information on a function determination model corresponding to the new device from the voice assistant server, and determining whether to store the function determination model in the hub device based on the received information;
Including, the method. The method of claim 21,
The hub device, when it is determined to store the function determination model in the hub device, storing a function determination model corresponding to the new device;
The method further comprising. The method of claim 21,
The hub device, when it is determined not to store the function determination model in the hub device, requesting and receiving access information for a function determination model corresponding to the new device from the voice assistant server;
The method further comprising. The method of claim 23,
The access information includes identification information, location information, IP address information (IP address), MAC address (MAC address) of the voice assistant server in which a function determination model corresponding to the new device is stored, and the function determination in the voice assistant server. A method comprising at least one of API information for accessing a model (Application Programming Interface), a language used for the function determination model, and identification information of the new device. The method of claim 21,
The hub device is determined by the voice assistant server based on resource state information for each of a plurality of devices pre-registered according to a user account. A system comprising a hub device and a voice assistant server,
The hub device,
A communication interface for performing data communication with the voice assistant server;
A memory storing a program including one or more instructions; And
A processor that executes one or more instructions of a program stored in the memory;
Including,
The processor,
Receiving information on the new device, controlling the communication interface to request the voice assistant server to update the device determination model previously stored in the memory,
Receiving the updated device determination model from the voice assistant server, storing the received updated device determination model in the memory,
Requesting information on a function determination model corresponding to the new device from the voice assistant server, and controlling the communication interface to receive information on a function determination model corresponding to the new device from the voice assistant server,
Determine whether to store the function determination model in the hub device based on the received information,
The voice assistant server,
Upon receiving an update request for the device determination model from the hub device, the hub device performs an update to the device determination model previously stored in the hub device and obtains information on a function determination model corresponding to the new device. To transfer to, the system. The method of claim 26,
The processor, when it is determined to store the function determination model in the hub device, stores a function determination model corresponding to the new device in the memory. The method of claim 26,
When it is determined that the function determination model is not stored in the hub device, the processor requests access information for a function determination model corresponding to the new device to the voice assistant server, and the access information is sent to the voice assistant server. Controlling the communication interface to receive from. The method of claim 28,
The access information includes identification information, location information, IP address information (IP address), MAC address (MAC address) of the voice assistant server in which a function determination model corresponding to the new device is stored, and the function determination in the voice assistant server. A system comprising at least one of API information (Application Programming Interface) for accessing a model, a language used for the function determination model, and identification information of the new device. The method of claim 26,
The hub device is determined by the voice assistant server based on resource state information for each of a plurality of devices previously registered according to a user account.

Images