KR20210014053A - System and method for registering device for voice assistant service - Google Patents

Abstract

Provided are a system and method for registering a device for a voice assistant service. The method for registering a novel device for a voice assistant service by a server comprises the operations of: comparing functions of a previously registered device and functions of the novel device; identifying functions corresponding to the functions of the previously registered device among the functions of the novel device based on the comparison result; obtaining previously registered utterance data related to at least a part of the identified functions; and generating action data for the novel device based on the identified functions and the obtained previously registered utterance data.

Description

System and method for registering devices for voice assistant service {SYSTEM AND METHOD FOR REGISTERING DEVICE FOR VOICE ASSISTANT SERVICE}

The present disclosure relates to a system and method for registering a new device for voice assistant service.

With the development of multimedia technology and network technology, users can receive various services using devices. In particular, with the development of speech recognition technology, a user can input a voice (for example, speech) to a device and receive a response message according to the voice input through a service providing agent.

When determining the intention included in the user's voice input in the voice assistant service, artificial intelligence (AI) technology can be used, and rule-based natural language understanding technology (NLU) can be used. May be.

However, when a new device is added in a home network environment including a plurality of devices in providing the voice assistant service, it is difficult to provide device control according to the user's voice input in consideration of the functions of the new device. there was. In particular, even if the new device is not a device registered in advance in the voice assistant service, it is necessary to effectively reflect the functions of the new device in the voice assistant service.

An embodiment of the present disclosure may provide a system and method capable of registering a new device using functions of a previously registered device for a voice assistant service.

In addition, an embodiment of the present disclosure may provide a system and method capable of registering a function of a new device by combining or deleting functions of at least one previously registered device.

In addition, an embodiment of the present disclosure may provide a system and method for registering a new device by using speech data related to functions of a previously registered device.

In addition, an embodiment of the present disclosure may provide a system and method capable of obtaining utterance data and action data related to functions of a new device by using utterance data and action data for a previously registered device. .

In addition, an embodiment of the present disclosure may provide a system and method capable of generating or updating a voice assistant model specialized for a new device by using speech data and action data related to functions of a new device.

As a technical means for achieving the above technical problem, a first aspect of the present disclosure includes an operation of obtaining specification information indicating functions of at least one pre-registered device; Comparing functions of the previously registered device and functions of the new device based on the specification information; Identifying functions corresponding to functions of the previously registered device among functions of the new device based on the comparison result; Obtaining pre-registered speech data related to at least some of the identified functions; Generating action data for the new device based on the identified functions and the acquired pre-registered speech data; And storing the obtained speech data and the generated action data in association with the new device, wherein the action data is included in a series of detailed functions of the new device corresponding to the obtained speech data. The server may provide a method of registering a new device for voice assistant service, including data relating to it.

In addition, a second aspect of the present disclosure is a communication interface; A memory storing a program including one or more instructions; And a processor that executes one or more instructions of the program stored in the memory. Including, wherein the processor obtains specification information representing functions of at least one pre-registered device, and compares functions of the pre-registered device and functions of the new device based on the specification information And, based on the comparison result, identify functions corresponding to functions of the previously registered device among functions of the new device, and acquire pre-registered speech data related to at least some of the identified functions, and , On the basis of the identified functions and the acquired pre-registered speech data, action data for the new device is generated, and the obtained speech data and the generated action data are associated with the new device. Server for registering a new device for voice assistant service, which is stored in a database (DB), and the action data includes data on a series of detailed functions of the new device corresponding to the acquired speech data Can provide.

In addition, a third aspect of the present disclosure may provide a computer-readable recording medium in which a program for executing the method of the first aspect on a computer is recorded.

1 is a schematic diagram of a system for providing a voice assistant service according to an embodiment of the present disclosure.
2 is a diagram illustrating an example in which a voice assistant server according to an embodiment of the present disclosure registers a new device based on a function of a previously registered device.
3 is a flowchart of a method for registering a new device by the voice assistant server 3000 according to an embodiment of the present disclosure.
4 is a flowchart illustrating a method of comparing, by a server, a function of a preset device and a function of a new device according to an embodiment of the present disclosure.
5A is a diagram illustrating an example of comparing a function of a previously registered device and a function of a new device according to an embodiment of the present disclosure.
5(b) is a diagram illustrating an example of comparing a function set of a previously registered device and a function of a new device according to an embodiment of the present disclosure.
5C is a diagram illustrating an example of comparing a combination of a function and a function set of a previously registered device with a function of a new device according to an embodiment of the present disclosure.
5(d) is a diagram illustrating an example of comparing a combination of a function and a function set of a plurality of previously registered devices with a function of a new device according to an embodiment of the present disclosure.
5(e) is a diagram illustrating an example of comparing a function of a new device with a function remaining after some of the functions of a previously registered device according to an embodiment of the present disclosure are deleted.
6 is a flowchart of a method for generating, by a voice assistant server, utterance data and action data related to a function different from a function of a previously registered device among functions of a new device according to an embodiment of the present disclosure.
7A is a diagram illustrating an example of a query output from a voice assistant server to generate speech data and action data related to a function of a new device according to an embodiment of the present disclosure.
7B is a diagram illustrating an example in which a query recommending a speech sentence is output to generate speech data and action data related to a function of a new device according to an embodiment of the present disclosure.
8 is a flowchart of a method for expanding speech data by the voice assistant server 3000 according to an embodiment of the present disclosure.
9A is a diagram illustrating an example in which similar speech data is generated from speech data according to an embodiment of the present disclosure.
9B is a diagram illustrating an example in which a representative speech and a similar speech are matched with action data according to an embodiment of the present disclosure.
10A is a diagram illustrating an example in which similar speech data is generated from speech data according to an embodiment of the present disclosure.
10B is a diagram illustrating an example in which a representative speech and a similar speech are matched with action data according to an embodiment of the present disclosure.
11A is a diagram illustrating an example of speech data according to an embodiment of the present disclosure.
11(b) is a diagram illustrating an example of speech data according to an embodiment of the present disclosure.
12 is a diagram illustrating an example of specification information of a device according to an embodiment of the present disclosure.
13 is a block diagram of a voice assistant server according to an embodiment of the present disclosure.
14 is a diagram illustrating another example of a voice assistant server according to an embodiment of the present disclosure.
15 is a conceptual diagram illustrating an action plan management model according to an embodiment of the present disclosure.
16 is a diagram illustrating a capsule database stored in an action plan management model according to an embodiment of the present disclosure.
17 is a block diagram of an IoT cloud server according to an embodiment of the present disclosure.
18 is a block diagram of a client device according to an embodiment of the present disclosure.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art may easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In addition, in the drawings, parts not related to the description are omitted in order to clearly describe the present disclosure, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a part is said to be "connected" to another part, this includes not only "directly connected" but also "electrically connected" with another element interposed therebetween. . In addition, when a part "includes" a certain component, it means that other components may be further included rather than excluding other components unless specifically stated to the contrary.

Functions related to artificial intelligence according to the present disclosure are operated through a processor and a memory. The processor may be composed of one or a plurality of processors. In this case, one or more processors may be a general-purpose processor such as a CPU, AP, or Digital Signal Processor (DSP), a graphics-only processor such as a GPU, a Vision Processing Unit (VPU), or an artificial intelligence-only processor such as an NPU. One or more processors control to process input data according to a predefined operation rule or an artificial intelligence model stored in the memory. Alternatively, when one or more processors are dedicated artificial intelligence processors, the artificial intelligence dedicated processor may be designed with a hardware structure specialized for processing a specific artificial intelligence model.

A predefined motion rule or an artificial intelligence model is characterized by being created through learning. Here, to be made through learning means that the basic artificial intelligence model is learned using a plurality of learning data by a learning algorithm, so that a predefined motion rule or artificial intelligence model set to perform a desired characteristic (or purpose) is created. Means Jim. Such learning may be performed in the device itself performing artificial intelligence according to the present disclosure, or may be performed through a separate server and/or system. Examples of the learning algorithm include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but are not limited to the above-described examples.

The artificial intelligence model may be composed of a plurality of neural network layers. Each of the plurality of neural network layers has a plurality of weight values, and a neural network operation is performed through an operation between the operation result of a previous layer and a plurality of weights. The plurality of weights of the plurality of neural network layers can be optimized by the learning result of the artificial intelligence model. For example, a plurality of weights may be updated to reduce or minimize a loss value or a cost value acquired from the artificial intelligence model during the learning process. The artificial neural network may include a deep neural network (DNN), for example, CNN (Convolutional Neural Network), DNN (Deep Neural Network), RNN (Recurrent Neural Network), RBM (Restricted Boltzmann Machine), DBN (Deep Belief Network), BRDNN (Bidirectional Recurrent Deep Neural Network), or deep Q-Networks (Deep Q-Networks), and the like, but is not limited to the above-described example.

Hereinafter, the present disclosure will be described in detail with reference to the accompanying drawings.

1 is a schematic diagram of a system for providing a voice assistant service according to an embodiment of the present disclosure.

Referring to FIG. 1, a system for providing a voice assistant service according to an embodiment of the present disclosure includes a client device 1000, at least one device 2000, a voice assistant server 3000, and Internet Of Things (IoT). It may include a cloud server (4000). The at least one device 2000 may be a device previously registered in the voice assistant server 3000 or the IoT cloud server 4000 for the voice assistant service.

The client device 1000 may receive a voice input (eg, speech) from a user. In one embodiment, the client device 1000 may include a voice recognition module. In one embodiment, the client device 1000 may include a voice recognition module having limited functionality. For example, the client device 1000 has a function of detecting a designated voice input (for example, a wake-up input such as'Hi Bixby','Okay Google', etc.) or pre-processing a voice signal obtained from some voice inputs. It may include a speech recognition module having a function. The client device 1000 may be an artificial intelligence speaker, but is not limited thereto. In an embodiment, some of the at least one device 2000 may be the client device 1000.

The at least one device 2000 may be a target device that performs a specific operation according to a control command from the voice assistant server 3000 and/or the IoT cloud server 4000. The at least one device 2000 may be controlled to perform a specific operation based on a user's voice input received by the client device 1000. In one embodiment, at least some of the at least one device 2000 may receive a control command from the client device 1000 without receiving a control command from the voice assistant server 3000 and/or the IoT cloud server 4000. May be.

The client device 1000 may receive a user's voice input through a microphone and transmit the received voice input to the voice assistant server 3000. In an embodiment, the client device 1000 may obtain a voice signal from the received voice input and transmit the voice signal to the voice assistant server 3000.

The voice assistant server 3000 receives a user's voice input from the client device 1000 and analyzes the received voice input to select a target device from among at least one device 2000 to perform actions according to the user's intention. In addition, information on the selected target device and operations to be performed by the target device may be provided to the IoT cloud server 4000 or the target device.

The IoT cloud server 4000 may register and manage information on the device 2000 for the voice assistant service, and may provide device information for the voice assistant service to the voice assistant server 3000. Device information is information related to a device used to provide a voice assistant service, and may include, for example, identification information (device id information), function performance information (capability), location information, and status information of the device. have. In addition, the IoT cloud server 4000 may receive information on a target device and operations to be performed by the target device from the voice assistant server 3000 and provide control information for controlling the operations to the target device.

When a new device 2900 for the voice assistant service is added, the voice assistant server 3000 utilizes the functions of the previously registered device 2000, speech data, and operations corresponding to the speech data. 2900) may generate speech data and action data. In addition, the voice assistant server 3000 may generate or update a voice assistant model to be used with respect to the new device 2900 by using speech data and action data about the new device 2900.

The speech data is data related to the voice spoken by the user in order to receive the voice assistant service, and may be data representing the speech of the user. The speech data may be data used to interpret the user's intention related to the operation of the device 2000. The speech data may include, for example, at least one of speech parameters in the form of a speech sentence in a text format or an output value of the NLU model. The utterance parameter is data output from the NLU model and may include an intent and a parameter. The intent is information determined by interpreting the text using the NLU model, and may indicate the user's speech intention. The intent may be, for example, information indicating an operation of a device intended by the user. The intent may include not only information indicating the user's utterance intention (hereinafter, intention information), but also a numerical value corresponding to information indicating the user's intention. Numerical values may indicate the probability that the text will be associated with information indicating a specific intent. When a plurality of pieces of information indicating the user's intention are obtained as a result of analyzing the text using the NLU model, intention information having a maximum numerical value corresponding to each intention information may be determined as the intent. Also, the parameter may be variable information for determining detailed operations of the device related to the intent. The parameter is information related to the intent, and a plurality of types of parameters may correspond to one intent. The parameter may include not only variable information for determining device operation information, but also a numerical value indicating a probability that the text is related to the variable information. As a result of analyzing the text using the natural language understanding model, a plurality of variable information indicating a parameter may be obtained. In this case, variable information having a maximum numerical value corresponding to each variable information may be determined as a parameter.

The action data may be data regarding a series of detailed operations of the device 2000 corresponding to predetermined speech data. For example, the action data may include detailed operations to be performed by the device in response to predetermined speech data, a relationship between each detailed operation and other detailed operations, and information related to an execution order of the detailed operations. . The association relationship between a detailed operation and another detailed operation includes information about another detailed operation to be executed before executing the detailed operation in order to execute one detailed operation. For example, when the operation to be performed is “music playback”, “power on” may be another detailed operation that must be executed before the “music playback” operation. In addition, the action data may include, for example, functions to be executed by the target device in order to perform a specific operation, an execution order of functions, an input value required to execute the functions, and an output value output as a result of execution of the functions. However, it is not limited thereto.

In addition, when the new device 2900 is identified, the voice assistant server 3000 acquires function information of the new device 2900, and stores function information of the previously registered device 2000 and the function information of the new device 2900. By comparing, it is possible to determine pre-registered speech data usable in relation to the function of the new device 2900. In addition, the voice assistant server 3000 may edit pre-registered speech data and functions corresponding thereto, and generate action data using the edited speech data and data related to the functions.

The device 2000 includes a smart phone, a tablet PC, a PC, a smart TV, a mobile phone, a personal digital assistant (PDA), a laptop, a media player, a micro server, a global positioning system (GPS) device, an e-book terminal, a digital broadcasting terminal, Navigation, kiosk, MP3 player, digital camera, and other mobile or non-mobile computing devices may be, but are not limited thereto. In addition, the device 2000 includes a lamp, air conditioner, TV, robot cleaner, washing machine, weight scale, refrigerator, set-top box, and home automation control panel with communication and data processing functions. , A security control panel, a game console, an electronic key, a camcorder, or an electronic frame. In addition, the device 2000 may be a wearable device such as a watch, glasses, hair band, and ring having a communication function and a data processing function. However, the present invention is not limited thereto, and the device 1000 may include all types of devices capable of transmitting and receiving data from the voice assistant server 3000 and/or the IoT cloud server 4000 through a network.

The network 200 includes a local area network (LAN), a wide area network (WAN), a value added network (VAN), a mobile radio communication network, a satellite communication network, and a mutual It includes a combination, and is a data communication network in a comprehensive meaning that enables each network member shown in FIG. 1 to communicate with each other smoothly, and includes a wired Internet, a wireless Internet, and a mobile wireless communication network. Wireless communication is, for example, wireless LAN (Wi-Fi), Bluetooth, Bluetooth low energy (Bluetooth low energy), Zigbee, WFD (Wi-Fi Direct), UWB (ultra wideband), infrared communication (IrDA, infrared Data Association). ), NFC (Near Field Communication), and the like, but are not limited thereto.

2 is a diagram illustrating an example in which a voice assistant server according to an embodiment of the present disclosure registers a new device based on a function of a previously registered device.

Referring to FIG. 2, when the new device 2900, the air conditioner B, is identified, the voice assistant server 3000 acquires function information of the air conditioner B, and the previously registered device 2000, the function information of the air conditioner A, and the dehumidification device A The function information of air conditioner B can be compared with the function information of air conditioner B

The voice assistant server 3000 may compare the functions of the air conditioner B with the functions of the air conditioner A and the functions of the dehumidification device A, and among the functions of the air conditioner B, the functions of the air conditioner A and the functions of the dehumidification device A are the same or Similar functions can be identified. For example, the voice assistant server 3000, among the functions of air conditioner B, is “power ON/OFF”, “cooling mode ON/OFF”, “dehumidification mode ON/OFF”, “temperature UP/DOWN” and “humidity control” It can be identified that this is consistent with the functions of the air conditioner A and the functions of the dehumidifying device A.

The voice assistant server 3000 may determine speech data corresponding to at least one of the identified functions, and generate action data corresponding to the determined speech data. For example, the voice assistant server 3000, the ignition data "power on" corresponding to the "power ON" of the air conditioner A, and the "temperature", which is the ignition data corresponding to the "cooling mode ON, temperature down" of the air conditioner A Lower it.”, “Temperature increase”, which is the ignition data corresponding to “Cooling mode ON, temperature UP” of air conditioner A, “Power on”, which is the ignition data corresponding to “Power ON” of dehumidifying unit A, and dehumidifying unit A By using "lower humidity", which is the ignition data corresponding to "Humidity DOWN", ignition data corresponding to at least one of the functions of the air conditioner B can be created or edited.

3 is a flowchart of a method for registering a new device by the voice assistant server 3000 according to an embodiment of the present disclosure.

In operation S300, the voice assistant server 3000 may acquire function information regarding functions of the new device 2900 and the previously registered device 2000. When a new device 2900 is added to the system for the voice assistant service, the voice assistant server 3000 is a new device 2900 or a new device 2900 obtained from an external server (not shown) connected to the new device 2900. Functions supported by the new device 2900 may be identified from the specification information of. For example, as shown in FIG. 12, the voice assistant server 3000 is a new device from specification information including information on a device identifier, a name of a function that can be performed, a description of a function that can be performed, and a factor necessary for performing the function. The functions supported by (2900) can be identified.

In addition, the voice assistant server 3000 may identify functions of the device 2000 from specifications of the previously registered device 2000. The voice assistant server 3000 may identify functions of the device 2000 from specification information stored in a DB (not shown) of the voice assistant server 3000. Alternatively, the voice assistant server 3000 receives specification information of the device 2000 stored in a DB (not shown) of the IoT cloud server 4000 from the IoT cloud server 4000, and the device 2000 from the received specification information Can identify the functions of The specification information of the previously registered device 2000, like the specification information of the new device 2900, includes an identifier of a device, a name of a function that can be performed, a description of a function that can be performed, and information on factors necessary for performing the function. I can.

In operation S310, the voice assistant server 3000 may determine whether the function of the previously registered device 2000 and the function of the new device 2900 are the same or similar. The voice assistant server 3000 compares the functions of the previously registered device 2000 with the functions of the new device 2900, so that the functions of the new device 2900 are the same as or similar to the functions of the previously registered device 2000. Function can be identified.

The voice assistant server 3000 identifies a name representing a function supported by the new device 2900 from the specification information of the new device 2900, and the identified name is of the function supported by the previously registered device 2000. It can be determined whether it is the same or similar to the name. In this case, the voice assistant server 3000 may pre-store information on names and similar words indicating a predetermined function, and based on the stored similar word information, the functions of the previously registered device 2000 and the functions of the new device 2900 It can be determined whether these are the same or similar.

Also, the voice assistant server 3000 may determine whether or not the functions are identical or similar with reference to the speech data. The voice assistant server 3000 may determine whether the function of the new device 2900 is the same as or similar to the function of the previously registered device 2000 by using speech data related to the function of the previously registered device 2000. . In this case, the voice assistant server 3000 may determine whether the function of the new device 2900 is the same as or similar to the function of the previously registered device 2000 based on the meaning of words included in the speech data.

The voice assistant server 3000 may determine whether a single function of the previously registered device 2000 and a single function of the new device 2900 are the same or similar. The single function may be, for example, one function such as'power ON','power OFF','temperature UP' and'temperature DOWN'. The voice assistant server 3000 may determine whether the function set of the previously registered device 2000 and the function set of the new device 2900 are the same or similar. The function set is a set of single functions, for example, may be a combination of functions such as "Power ON + Temperature UP" and "Temperature DOWN + Humidity DOWM".

If it is determined in operation S310 that the function of the previously registered device 2000 and the function of the new device 2900 are the same or similar, in operation S320, the voice assistant server 3000 uses pre-registered speech data related to the same or similar functions. Can be obtained.

The voice assistant server 3000 may extract speech data corresponding to a function determined to be the same as or similar to the function of the new device 2900 among functions of the previously registered device 2000 from a DB (not shown).

The voice assistant server 3000 may extract speech data corresponding to a function set determined to be the same or similar to the function of the new device 2900 from among the function sets of the previously registered device 2000 from a DB (not shown). have.

In this case, speech data corresponding to a function of the previously registered device 2000 and speech data corresponding to a function set of the previously registered device 2000 may be previously stored in a DB (not shown).

Meanwhile, the voice assistant server 3000 may edit a function and a set of functions determined to be identical or similar, and may generate speech data corresponding to the edited functions. The voice assistant server 3000 may combine functions determined to be the same or similar, and generate speech data corresponding to the combined functions. In addition, the voice assistant server 3000 may combine a function and a set of functions determined to be the same or similar, and generate speech data corresponding to the combined functions. In addition, the voice assistant server 3000 may delete some functions from among functions in the function set determined to be identical or similar, and may generate speech data corresponding to the function set from which some functions have been deleted.

The voice assistant server 3000 may expand speech data. The voice assistant server 3000 may generate similar speech data having the same meaning as the extracted or generated speech data but having a different expression by modifying the expression of the extracted or generated speech data.

In operation S330, the voice assistant server 3000 may generate action data for the new device 2900 based on the same or similar functions and speech data. The action data may be data indicating detailed operations of the device and an execution order of detailed operations according to the speech data. The action data may include, for example, an identification value of detailed operations, an execution order of detailed operations, a control command for executing a detailed operation, and the like, but is not limited thereto.

For example, when the function corresponding to the speech data is a single function, the voice assistant server 3000 may generate action data including a detailed operation indicating a single function. For example, when a function corresponding to speech data is a function set, the voice assistant server 3000 may generate detailed operations representing functions in the function set, and an execution order of the detailed operations.

In operation S340, the voice assistant server 3000 may create or update a voice assistant model related to the new device 2900 using speech data and action data.

The voice assistant server 3000 includes speech data corresponding to the functions of the previously registered device 2000 related to the functions of the new device 2900, speech data newly generated in relation to the functions of the new device 2900, and extended Using the speech data and action data, a voice assistant model related to the new device 2900 may be created or updated. The voice assistant server 3000 may accumulate and store speech data and action data related to the new device 2900. In addition, the voice assistant server 3000 may create or update a CAN (Concept Action Network), which is a capsule-type database included in an action plan management model (not shown).

The voice assistant model related to the new device 2900 is a model used for the voice assistant service and is a model specialized for the new device 2900 and may be a model that determines an operation to be performed by a target device corresponding to a user's voice input. . The voice assistant model related to the new device 2900 may include, for example, an NLU model, an NLG model, and an action plan management model. The NLU model related to the new device 2900 is an artificial intelligence model for interpreting the user's input voice in consideration of the function of the new device 2900, and the NLG model related to the new device 2900 considers the function of the new device. It may be an artificial intelligence model for generating natural language for dialogue with a user. Also, the action plan management model related to the new device 2900 may be a model for planning operation information to be performed by the new device 2900 in consideration of functions of the new device 2900. The action plan management model may select detailed operations to be performed by the new device 2900 from the interpreted user's spoken voice, and plan an execution order of the selected detailed operations. The action plan management model may acquire operation information on a detailed operation to be performed by the new device 2900 by using the planning result. The operation information may be information related to detailed operations to be performed by the device, a correlation relationship between detailed operations, and an execution order of the detailed operations. The operation information includes, for example, functions to be executed by the new device 2900 in order to perform detailed operations, an execution order of functions, an input value required to execute the functions, and an output value output as a result of execution of the functions. It can, but is not limited thereto.

When a voice assistant model that can be used for the new device 2900 already exists, the voice assistant server 3000 may update the voice assistant model.

In addition, the voice assistant server 3000 includes utterance data corresponding to the function of the previously registered device 2000 related to the function of the new device 2900, utterance data newly generated in relation to the function of the new device 2900, Using the expanded speech data and action data, a voice assistant model related to the new device 2900 may be created or updated.

In addition, the action plan management model may manage information about a plurality of detailed operations of the new device 2900 and a relationship between the plurality of detailed operations. The relationship between each detailed operation and another detailed operation among a plurality of detailed operations may include information on other detailed operations that must be executed before executing the detailed operation in order to execute one detailed operation. have.

In an embodiment, the action plan management model may include a Concept Action Network (CAN), which is a capsule-type database representing the actions of the device and the relationship between the actions. CAN (Concept Action Network) includes functions to be executed by the device to perform a specific operation, order of execution of functions, input values necessary to execute functions, and output values output as a result of execution of functions. It can be implemented as an ontology graph composed of knowledge triples representing the relationship between concepts.

If it is determined in operation S310 that the function of the previously registered device 2000 and the function of the new device 2900 are not the same or similar, the voice assistant server 3000 in operation S350 is different from the function of the previously registered device 2000. Input of speech data and action data for a function can be requested. The voice assistant server 3000 may register different functions of the new device 2900 and output a query message for generating and editing speech data related to the different functions. The query message may be provided to the client device 1000, the new device 2900, or the developer's device (not shown). The voice assistant server 3000 may receive a response to the query message from the client device 1000, the new device 2900, or the developer's device (not shown). The voice assistant server 3000 may provide a software development kit (SDK) tool for registering the function of the new device 2900 to the client device 1000, the new device 2900, or a developer's device (not shown). . In addition, the voice assistant server 3000 provides a list of functions different from those of the previously registered device 2000 among the functions of the new device 2900 to the user's device 2000 or the developer's device (not shown). I can. The voice assistant server 3000 may provide recommended speech data related to at least some of different functions to the user's device 2000 or the developer's device (not shown).

In operation S360, the voice assistant server 3000 may acquire speech data and action data. The voice assistant server 3000 may analyze a response to a query using the NLU model. The voice assistant server 3000 may interpret a user's response or a developer's response by using an NLU model trained for function registration and generation of speech data. The voice assistant server 3000 may generate speech data related to functions of the new device 2900 based on the interpreted response. The voice assistant server 3000 may generate speech data related to functions of the new device 2900 by using an interpreted user's response or an interpreted developer's response, and recommend the generated speech data. The voice assistant server 3000 may select some of the functions of the new device 2900 and generate speech data related to each of the selected partial functions. Also, the voice assistant server 3000 may select some of the functions of the new device 2900 and generate speech data related to a combination of the selected partial functions. In addition, the voice assistant server 3000 may generate similar speech data having the same meaning as the generated speech data and different expressions. The voice assistant server 3000 may generate action data using the generated speech data. The voice assistant server 3000 may generate action data corresponding to the generated speech data by identifying functions of the new device 2900 related to the generated speech data and determining an execution order of the identified functions.

4 is a flowchart illustrating a method of comparing, by a server, a function of a preset device and a function of a new device according to an embodiment of the present disclosure.

In operation S400, the voice assistant server 3000 may compare the functions of the previously registered device 2000 and the functions of the new device 2900. The voice assistant server 3000 may compare the name of the function supported by the new device 2900 with the name of the function supported by the previously registered device 2000. In this case, the voice assistant server 3000 may pre-store information on names and similar words indicating a predetermined function, and based on the stored similar word information, the functions of the previously registered device 2000 and the functions of the new device 2900 Can be compared.

In addition, the voice assistant server 3000 may compare the functions of the previously registered device 2000 and the functions of the new device 2900 with reference to speech data stored in the IoT cloud server 4000. The voice assistant server 3000 may determine whether the function of the new device 2900 is the same as or similar to the function of the previously registered device 2000 by using speech data related to the function of the previously registered device 2000. . In this case, the voice assistant server 3000 may determine whether the function of the new device 2900 is the same as or similar to the function of the previously registered device 2000 based on the meaning of words included in the speech data.

In operation S405, the voice assistant server 3000 may determine whether a function that does not match the function of the previously registered device 2000 exists among the functions of the new device 2900. The voice assistant server 3000 may determine whether the functions of the at least one previously registered device 2000 and the functions of the new device 2900 all match. For example, the voice assistant server 3000 includes functions of the new device 2900 among functions of the first device 2100, functions of the second device 2200, and functions of the third device 2300. Matching functions can be identified.

When the name of the function of the new device 2900 is the same as the name of the function of the previously registered device 2000, the voice assistant server 3000 uses the function of the previously registered device 2000 and the function of the new device 2900. It can be judged that this coincides.

In addition, it is determined that the function name of the new device 2900 is similar to the function name of the previously registered device 2000 and that the function of the previously registered device 2000 and the function of the new device 2900 are functions for the same purpose. In this case, the voice assistant server 3000 may determine that the function of the previously registered device 2000 and the function of the new device 2900 match.

If it is determined in operation S405 that there is no function that does not match the function of the previously registered device 2000 among the functions of the new device 2900, the voice assistant server 3000 may perform operations S320 to S340. The voice assistant server 3000 uses the utterance data and action data related to the function of the previously registered device 2000 that matches the function of the new device 2900, and uses the utterance data and action data related to the function to the new device 2900. Action data may be generated, and a voice assistant model for providing a voice assistant service for the new device 2900 may be generated or updated.

If it is determined in operation S405 that a function that does not match the function of the previously registered device 2000 exists among the functions of the new device 2900, the voice assistant server 3000 performs the function of the previously registered device 2000 in operation S410. You can combine them.

The voice assistant server 3000 may combine single functions of at least one device 2000. For example, the voice assistant server 3000 may combine a first function of the first device 2100 and a second function of the first device 2100. Also, for example, the voice assistant server 3000 may combine a first function of the first device 2100 and a third function of the second device 2200.

The voice assistant server 3000 may combine function sets of at least one device 2000. For example, the voice assistant server 3000 may combine a first function set of the first device 2100 and a second function set of the first device 2100. Also, for example, the voice assistant server 3000 may combine a first function set of the first device 2100 and a third function set of the second device 2200.

The voice assistant server 3000 may combine a single function and a function set of at least one device 2000. For example, the voice assistant server 3000 may combine a first function of the first device 2100 and a first function set of the first device 2100. Also, for example, the voice assistant server 3000 may combine a first function of the first device 2100 and a third function set of the second device 2200.

The voice assistant server 3000 may combine functions of the previously registered device 2000 from speech data corresponding to functions of the previously registered device 2000. For example, the voice assistant server 3000 extracts first speech data corresponding to a first function of the first device 2100 and second speech data corresponding to a second function from a DB (not shown), and It may be determined to combine the first function and the second function based on the meaning of the first speech data and the second speech data. In addition, for example, the voice assistant server 3000 may store first speech data corresponding to the first function of the first device 2100 and third speech data corresponding to the third function of the second device 2200 to a DB. It may be extracted from (not shown) and determined to combine the first function and the third function based on the meaning of the first speech data and the third speech data.

In operation S415, the voice assistant server 3000 may compare the combined functions with the functions of the new device 2900. The voice assistant server 3000 may compare names of the combined functions with names of functions supported by the previously registered device 2000. In addition, the voice assistant server 3000 may compare the combined functions and functions of the new device 2900 with reference to the speech data stored in the IoT cloud server 4000.

In operation S420, after performing operation S415, the voice assistant server 3000 may determine whether a function that does not match the function of the previously registered device 2000 exists among the functions of the new device 2900. When the names of the combined functions are the same as the names of the functions of the previously registered device 2000, the voice assistant server 3000 may determine that the combined functions match the functions of the new device 2900.

In addition, when the names of the combined functions are similar to those of the previously registered device 2000 and it is determined that the combined functions and functions of the new device 2900 are functions of the same purpose, the voice assistant server The 3000 may determine that the combined functions match the functions of the new device 2900.

If it is determined in operation S420 that there is no function that does not match the function of the previously registered device 2000 among the functions of the new device 2900, the voice assistant server 3000 may perform operations S320 to S340. The voice assistant server 3000 uses utterance data and action data related to a function of a previously registered device 2000 that matches the function of the new device 2900, and utterance data and action data related to a combined function, Speech data and action data related to functions may be generated in the new device 2900, and a voice assistant model may be generated or updated to provide a voice assistant service for the new device 2900.

If it is determined in operation S420 that a function that does not match the function of the previously registered device 2000 exists among the functions of the new device 2900, the voice assistant server 3000 performs some of the functions of the device 2000 in operation S425. Can be deleted.

The voice assistant server 3000 may delete some of the single functions of at least one device 2000. The voice assistant server 3000 may delete a single function that is determined not to be supported by the new device 2900 from among the single functions of the device 2000.

The voice assistant server 3000 may delete some of the functions in the function sets of the at least one device 2000. The voice assistant server 3000 may delete a function that is determined not to be supported by the new device 2900 from among functions in the function sets of the device 2000.

The voice assistant server 3000 may delete some of the function sets of at least one device 2000. The voice assistant server 3000 may delete function sets determined to be not supported by the new device 2900 from among the function sets of the device 2000.

In operation S430, the voice assistant server 3000 may compare functions remaining after deletion with functions of the new device 2900. The voice assistant server 3000 may compare names of functions remaining after deletion with names of functions supported by the previously registered device 2000. In addition, the voice assistant server 3000 may compare the functions remaining after deletion with the functions of the new device 2900 by referring to the speech data stored in the IoT cloud server 4000.

In operation S435, after performing operation S430, the voice assistant server 3000 may determine whether a function that does not match the function of the previously registered device 2000 exists among the functions of the new device 2900. When the names of the functions remaining after deletion are the same as the names of the functions of the previously registered device 2000, the voice assistant server 3000 may determine that the functions remaining after the deletion match the functions of the new device 2900. have.

In addition, if the names of functions remaining after deletion are similar to the names of functions of the previously registered device 2000, and it is determined that the functions remaining after deletion and the functions of the new device 2900 are functions of the same purpose, The voice assistant server 3000 may determine that functions remaining after deletion are consistent with functions of the new device 2900.

If it is determined in operation S435 that there is no function that does not match the function of the previously registered device 2000 among the functions of the new device 2900, the voice assistant server 3000 may perform operations S320 to S340. The voice assistant server 3000 includes utterance data and action data related to a function of a previously registered device 2000 that matches the function of the new device 2900, utterance data and action data related to a combined function, and functions remaining after deletion. Creating a voice assistant model for generating speech data and action data related to functions in the new device 2900, and providing a voice assistant service for the new device 2900, using the speech data and action data related to Can be updated.

If it is determined in operation S435 that a function that does not match the function of the previously registered device 2000 exists among the functions of the new device 2900, the voice assistant server 3000 may perform operation S350.

In FIG. 4, S400, S410, S415, S425 and S430 have been described as being sequentially performed, but are not limited thereto. For example, before comparing the functions of the new device 2900 and the functions of the previously registered device 2000 (S400), the functions of the previously registered device 2000 are combined (S410) or a part of them is deleted (S425). It is possible to establish a DB (not shown) in advance by performing the operation. In this case, the function of the new device 2900, the function of the previously registered device 2000, the combined function of the previously registered device 2000, and the deletion function of the previously registered device 2000 using the built DB Compared with (S400, S415, S430), it may be determined whether a function that does not match the function of the previously registered device 2000 exists among the functions of the new device 2900.

5A is a diagram illustrating an example of comparing a function of a previously registered device and a function of a new device according to an embodiment of the present disclosure.

Referring to FIG. 5A, the voice assistant server 3000 may compare functions of a pre-registered air conditioner A, functions of a pre-registered dehumidifier A, and functions of a new air conditioner B.

For example, the functions supported by the new air conditioner B are power ON/OFF, cooling mode ON/OFF, dehumidification mode ON/OFF, temperature setting, temperature UP/DOWN, humidity setting, humidity UP/DOWN, artificial intelligence mode. It may include ON/OFF, etc. Further, for example, functions supported by the previously registered air conditioner A may include power ON/OFF, cooling mode ON/OFF, temperature setting, temperature UP/DOWN, and the like. Also, for example, functions supported by the previously registered dehumidifier A may include power ON/OFF, humidity setting, humidity UP/DOWN, and the like.

Among the functions of the new air conditioner B, the voice assistant server 3000 is an air conditioner A and dehumidifier A in which power ON/OFF, cooling mode ON/OFF, temperature setting, temperature UP/DOWN, humidity setting, humidity UP/DOWN are previously registered. It can be judged that it matches the functions of The voice assistant server 3000 may use ignition data related to functions of air conditioner A and ignition data regarding functions of dehumidifier A in order to determine whether the functions match.

The voice assistant server 3000 acquires ignition data corresponding to the functions provided by the pre-registered air conditioner A, such as power ON/OFF, cooling mode ON/OFF, temperature setting, and temperature UP/DOWN, and pre-registered dehumidifier It is possible to acquire ignition data corresponding to the functions provided by A, such as power ON/OFF, humidity setting, and humidity UP/DOWN. Also, the voice assistant server 3000 may generate action data of a new air conditioner B by using matching functions and acquired ignition data.

5B is a diagram illustrating an example of comparing a function set of a previously registered device and a function of a new device according to an embodiment of the present disclosure.

Referring to FIG. 5(b), the voice assistant server 3000 uses'cooling mode ON + temperature UP', which is a function set of pre-registered air conditioner A, and'cooling mode ON/OFF' and'temperature UP/ It can be identified that it matches'DOWN'.

The voice assistant server 3000 may acquire “temperature increase”, which is ignition data corresponding to “cooling mode ON + temperature UP”, which is a function set of pre-registered air conditioner A. In addition, the voice assistant server 3000 uses'cooling mode ON/OFF' and'temperature UP/DOWN', which are functions of the new air conditioner B, and'cooling mode ON' function, using the acquired ignition data'temperature increase'. After executing'Temperature UP', action data for executing the'Temperature UP' function can be created.

5C is a diagram illustrating an example of comparing a combination of a function and a function set of a previously registered device with a function of a new device according to an embodiment of the present disclosure.

5(c), the voice assistant server 3000 is a new air conditioner with a combination of'power ON' which is a function of a pre-registered air conditioner A and'cooling mode ON + temperature down' which is a function set of a pre-registered air conditioner A. It can be identified that it matches'Power ON/OFF','Cooling Mode ON/OFF' and'Temperature UP/DOWN' of B.

The voice assistant server 3000 corresponds to'power on', which is ignition data corresponding to'power ON', which is a function of pre-registered air conditioner A, and'cooling mode ON + temperature down', which is a function set of pre-registered air conditioner A. It is possible to obtain the ignition data'lower temperature'. Also, the voice assistant server 3000 may edit the acquired speech data. For example, the voice assistant server 3000 may generate ignition data representing'turn on the air conditioner to lower the temperature' from'turn on the power' and'lower the temperature'.

In addition,'Power ON/OFF','Cooling Mode ON/OFF' and'Temperature UP/DOWN' of the Voice Assistant's new air conditioner B, and'Power ON' by using the generated ignition data'Turn on the air conditioner to lower the temperature'. After executing the function, the'cooling mode ON' function is executed, and after that, action data for executing the'temperature down' function can be generated.

FIG. 5D is a diagram illustrating an example of comparing a combination of a function and a function set of a plurality of previously registered devices with a function of a new device according to an embodiment of the present disclosure.

Referring to FIG. 5(d), the voice assistant server 3000 includes i)'Power ON', which is a function of pre-registered air conditioner A, ii)'Cooling Mode ON + Temperature DOWN', which is a function set of pre-registered air conditioner A, And iii) the combination of'Power ON + Humidity DOWN', which is the function set of dehumidifying device A, is the combination of'Power ON/OFF','Cooling Mode ON/OFF','Temperature UP/DOWN', and'Dehumidification Mode' of the new air conditioner B. It can be identified that it matches'ON/OFF' and'Humidity UP/DOWN'.

The voice assistant server 3000 includes i)'Power ON', which is the ignition data corresponding to'Power ON', which is a function of pre-registered air conditioner A, ii)'Cooling Mode ON + Temperature DOWN', which is a function set of pre-registered air conditioner A. 'Temperature lowering', which is ignition data corresponding to and iii)'lower humidity', which is ignition data corresponding to'Power ON + Humidity DOWN', which is a function set of the previously registered dehumidifying device A, may be obtained. Also, the voice assistant server 3000 may edit the acquired speech data. For example, the voice assistant server 3000 can generate ignition data representing'turn on the air conditioner to lower the temperature and lower the humidity' from'turn on the power','lower the temperature', and'lower the humidity'.

In addition,'Power ON/OFF','Cooling Mode ON/OFF','Temperature UP/DOWN','Dehumidification Mode ON/OFF' and'Humidity UP/DOWN', and generated ignition data of Voice Assistant's new air conditioner B After executing the'Power ON' function, execute the'Cooling Mode ON' function, execute the'Temperature DOWN' function, and execute the'Dehumidification Mode ON' function by using'Turn on the air conditioner to lower the temperature and lower the humidity'. Action data can be created to execute the'Humidity DOWN' function.

5(e) is a diagram illustrating an example of comparing a function remaining after deleting some of functions of a previously registered device with a function of a new device according to an embodiment of the present disclosure.

Referring to FIG. 5(e), the voice assistant server 3000 performs'Temperature 26° Set + Temperature Check' in'Temperature 26° Setting + Temperature Check + Artificial Intelligence Mode ON', which is a function set of the registered air conditioner A. You can acquire the leftover function,'Artificial Intelligence Mode ON'. In addition, the voice assistant server 3000 is a combination of i)'Artificial Intelligence Mode ON' which is the remaining function and ii)'Power ON' which is the function of the previously registered air conditioner A, and'Power ON/OFF' and'Artificial Intelligence Mode ON' of the new air conditioner B. It can be identified that it matches'intelligence mode ON/OFF'.

The voice assistant server 3000 may i) acquire “power on”, which is ignition data corresponding to “power ON,” which is a function of the previously registered air conditioner A. In addition, the voice assistant server 3000 is from'Turn on the artificial intelligence function at a temperature of 26 degrees', which is ignition data corresponding to'Temperature 26 degrees setting + temperature check + artificial intelligence mode ON', which is a function set of pre-registered air conditioner A. , It is possible to extract the utterance data'Turn on artificial intelligence function' corresponding to the remaining function'Artificial Intelligence Mode ON'. In addition, the voice assistant server 3000 may generate utterance data indicating'turn on the power and turn on the artificial intelligence function' from'turn on the power' and'turn on the artificial intelligence function'.

In addition, the voice assistant server 3000 uses'power ON/OFF' and'artificial intelligence mode ON/OFF' of the new air conditioner B, and'power on and turn on the artificial intelligence function', which is the generated ignition data. After executing the'ON' function, action data for executing the'Artificial Intelligence Mode ON' function can be created.

6 is a flowchart of a method for generating, by a voice assistant server, utterance data and action data related to a function different from a function of a previously registered device among functions of a new device according to an embodiment of the present disclosure.

In operation S600, the voice assistant server 3000 may output a query for registering an additional function and generating or editing speech data using the NLG model. The voice assistant server 3000 may register a function of the new device 2900 and provide a GUI for generating speech data to the user's device 2000 or the developer's device (not shown). The developer's device (not shown) may install a software development kit (SDK) for registering a new device, and receive a GUI from the voice assistant server 3000 through the installed software development kit. .

The voice assistant server 3000 may register a function of the new device 2900 and provide guidance text or guidance voice data for guiding the generation of speech data to the user's device 2000 or the developer's device (not shown). have. The voice assistant server 3000 may generate a query for registering additional functions and generating speech data by using an NLG model trained for function registration and speech data generation.

In addition, the voice assistant server 3000 provides a list of functions different from those of the previously registered device 2000 among the functions of the new device 2900 to the user's device 2000 or the developer's device (not shown). I can. The voice assistant server 3000 may provide recommended speech data related to at least some of different functions to the user's device 2000 or the developer's device (not shown).

In operation S610, the voice assistant server 3000 may analyze a response to the query using the NLU model. The voice assistant server 3000 may receive the user's response to the query from the user's device 2000 or the developer's response to the query from the developer's device (not shown). The voice assistant server 3000 may interpret a user's response or a developer's response by using an NLU model trained for function registration and generation of speech data.

In addition, the voice assistant server 3000 receives the user's response input through the GUI provided to the user's device 2000 from the user's device 2000, or the developer's response through the GUI provided to the developer's device (not shown). Input may be received from the developer's device (not shown).

In operation S620, the voice assistant server 3000 may generate speech data related to functions of the new device 2900 based on the analyzed response. The voice assistant server 3000 may generate speech data related to functions of the new device 2900 by using an interpreted user's response or an interpreted developer's response, and recommend the generated speech data. The voice assistant server 3000 may select some of the functions of the new device 2900 and generate speech data related to each of the selected partial functions. Also, the voice assistant server 3000 may select some of the functions of the new device 2900 and generate speech data related to a combination of the selected partial functions.

The voice assistant server 3000 may generate speech data related to the function of the new device 2900 using an NLG model for generating speech data, based on the identification values and attributes of the functions of the new device 2900 . For example, the voice assistant server 3000 may input data representing identification values and attributes of functions of a new device into an NLG model for generating speech data, and obtain speech data output from the NLG model. Not limited.

In addition, the voice assistant server 3000 may select at least some of the generated speech data based on the user's response input input through the GUI and the developer's response input input through the GUI. In addition, the voice assistant server 3000 may generate similar speech data having the same meaning as the generated speech data and different expressions.

In operation S630, the voice assistant server 3000 may generate action data using the generated speech data. The voice assistant server 3000 may generate action data corresponding to the generated speech data by identifying functions of the new device 2900 related to the generated speech data and determining an execution order of the identified functions. The generated action data may be matched with speech data and similar speech data.

7A is a diagram illustrating an example of a query output from a voice assistant server to generate speech data and action data related to a function of a new device according to an embodiment of the present disclosure.

Referring to FIG. 7(a), the voice assistant server 3000 requests a query for inputting an utterance related to a new function of an air conditioner, a new device 2900, of a user's device 2000 or a developer's device (not shown). Can be provided to. For example, a text or a query voice saying “Please input a speech related to the automatic drying function” may be output from the user's device 2000 or the developer's device (not shown).

In addition, the voice assistant server 3000, the user's device 2000 or the developer's device inputs the utterance "Remove the smell of air conditioner" input from the user's device 2000 or the developer's device (not shown). It can be received from (not shown).

The voice assistant server 3000 may modify "Remove the smell of air conditioner." to "Remove the smell of air conditioner." In addition, the voice assistant server 3000 may generate “current operation OFF + drying function ON”, which is action data corresponding to “remove air conditioner odor”, which is a modified speech statement.

7B is a diagram illustrating an example in which a query recommending a speech sentence is output to generate speech data and action data related to a function of a new device according to an embodiment of the present disclosure.

Referring to FIG. 7(b), the voice assistant server 3000 provides text or voice for notifying the new function of the air conditioner, which is the new device 2900, to the user's device 2000 or the developer's device (not shown). I can. For example, the voice assistant server 3000 may output a text or voice saying “The automatic drying function is a new function” from the user's device 2000 or the developer's device (not shown). In addition, the voice assistant server 3000 may generate a recommended utterance related to the automatic drying function, which is a new function, and provide text or voice representing the recommended utterance to the user's device 2000 or the developer's device (not shown). have. For example, the voice assistant server 3000 asks, "Do you want to register an utterance saying "When you turn off the air conditioner, execute the drying function"?" is the user's device 2000 or the developer's device (not shown). Can be output from

In addition, the voice assistant server 3000 may receive an input for selecting a recommended speech from the user's device 2000 or the developer's device (not shown).

The voice assistant server 3000 checks the reception of the power OFF input + drying function, which is action data corresponding to the recommended utterance, "Do you want to register the utterance ""Execute the drying function when turning off the air conditioner." + Air conditioner power off” can be generated.

8 is a flowchart of a method for expanding speech data by the voice assistant server 3000 according to an embodiment of the present disclosure.

In operation S800, the voice assistant server 3000 may acquire similar speech data related to the generated speech data by inputting the generated speech data into the artificial intelligence model. The voice assistant server 3000 may acquire similar utterance data output from the artificial intelligence model by inputting the generated utterance data to the trained artificial intelligence model to generate utterance data similar to the utterance data. The artificial intelligence model may be, for example, a model trained by using a set of spoken sentences and similar spoken sentences for controlling a device as training data.

For example, as shown in Fig. 9(a), when the ignition statement "Remove the smell of air conditioner" is input as an artificial intelligence model, "Remove the smell of air conditioner", "I smell the air conditioner", "I smell moldy", Similar utterances, such as "remove the smell of mold," can be output from the artificial intelligence model. In addition, “remove air conditioner odor”, which is an utterance sentence input to the artificial intelligence model, may be set as the representative utterance sentence. The representative speech sentence may be set in consideration of, for example, the frequency of use of the user and the accuracy of grammar, but is not limited thereto.

In addition, for example, as shown in Fig. 10(a), when the ignition statement “when the air conditioner is turned off, execute the drying function” is input as the artificial intelligence model, “when the air conditioner is turned off, the drying function is executed”, “when the air conditioner is turned off, Similar utterances such as, "Smell no smell", "If the air conditioner is turned off, no smell", etc. can be output from the artificial intelligence model. Also, when the air conditioner is turned off, the drying function is executed” can be determined as the representative fire statement. In addition, one of the similar speech statements output from the artificial intelligence model, “If the air conditioner is turned off, execute the drying function” may be set as the representative speech statement. The representative speech sentence may be set in consideration of, for example, the frequency of use of the user and the accuracy of grammar, but is not limited thereto.

In operation S810, the voice assistant server 3000 may match action data with utterance data and similar utterance data. The voice assistant server 3000 may match action data corresponding to the speech data input to the artificial intelligence model with the pseudo speech data output from the artificial intelligence model.

For example, as shown in FIG. 9(b), representative fire statements “Remove the smell of air conditioner”, and similar fire statements “Remove the smell of air conditioner”, “I smell air conditioner”, “I smell mold”, “ Remove mold odor” can be mapped to action data “Current operation OFF -> Dry function ON”.

In addition, for example, as shown in FIG. 10(b), the representative fire statements "If you turn off the air conditioner, perform the drying function" and similar fire statements, "Remove the smell of air conditioner", "When you turn off the air conditioner, please do not make the smell", "Air conditioner If you turn it off, let me not smell it” can be mapped to the action data “Check reception of power OFF input -> Dry function ON -> Air conditioner power OFF”.

11A is a diagram illustrating an example of speech data according to an embodiment of the present disclosure.

Referring to FIG. 11A, speech data may be speech in text form. For example, the representative speech “Turn on TV”, and similar speech phrases “Turn on TV”. “Turn on the TV” or “Turn on the TV” may be speech data.

11(b) is a diagram illustrating an example of speech data according to an embodiment of the present disclosure.

Referring to FIG. 11B, speech data may include speech parameters and speech sentences. The utterance parameter is an output value of the NLU model and may include an intent and a parameter. For example, the utterance parameter included in the utterance data may include an intent “power on” and a parameter “TV”. Also, for example, the utterances included in the utterance data are “Turn on the TV” or “Turn on the TV”. You can include text such as “Turn on TV” or “Turn on TV”. In FIG. 11B, it has been described that the speech data includes a speech parameter and a speech sentence, but the present invention is not limited thereto, and the speech data may include only speech parameters.

13 is a block diagram of a voice assistant server according to an embodiment of the present disclosure.

Referring to FIG. 13, the voice assistant server 3000 includes a communication interface 3100, a processor 3200, and a storage unit 3300, and the storage unit 3300 includes a first voice assistant model 3310, at least one The second voice assistant model 3320, SDK interface module 3330, and DB 3340 may be included.

The communication interface 3100 communicates with the client device 1000, the device 2000, and the IoT cloud server 4000. The communication interface 3100 may include one or more components for communication with the client device 1000, the device 2000, and the IoT cloud server 4000.

The processor 3200 typically controls the overall operation of the voice assistant server 3000. For example, the processor 3200 may control a function of the voice assistant server 3000 in the present specification by executing programs stored in the storage unit 3300.

The storage unit 3300 may store a program for processing and control of the processor 3200 and may store data related to functions of the new device 2900. The storage unit 3300 includes a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (eg, SD or XD memory, etc.), RAM (Random Access Memory) SRAM (Static Random Access Memory), ROM (ROM, Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, magnetic It may include at least one type of storage medium among a disk and an optical disk.

The programs stored in the storage unit 3300 can be classified into a plurality of modules according to their functions, for example, a first voice assistant model 3310, at least one second voice assistant model 3320, and an SDK It may be classified as an interface module 3330 or the like.

The first voice assistant model 3310 is a model that determines a target device related to user intention by analyzing a user's voice input. The first voice assistant model 3310 is an ASR (Automatic Speech Recognition) model 3311, a first NLU model 3312, a first NLG model 3313, a device determination module 3314, a function comparison module 3315, A speech data acquisition module 3316, an action data generation module 3317, and a model updater 3318 may be included.

The ASR model 3311 converts the speech signal into text by performing ASR. The ASR model 3311 may perform ASR for converting a speech signal into a computer-readable text using a predefined model such as an acoustic model (AM) or a language model (LM). When an acoustic signal from which noise is not removed from the client device 110 is received, the ASR model 3311 may obtain a voice signal by removing noise from the received acoustic signal, and perform ASR on the voice signal. .

The first NLU model 3312 analyzes the text and determines a first intent related to the user's intention based on the analysis result. The first NLU model 3312 may be a model trained to obtain a first intent corresponding to the text by analyzing the text. The intent may be information indicating a user's speech intention included in the text.

The device determination model 3314 determines a user's first intent from the converted text by performing a syntactic analysis or semantic analysis using the first NLU model 3312. I can. In one embodiment, the device determination model 3314 uses the first NLU model 3312 to parse the converted text in units of morphemes, words, or phrases, and the parsed The meaning of the word extracted from the parsed text can be inferred by using the linguistic features of the morpheme, word, or phrase (eg, grammatical element). The device determination model 3314 may determine a first intent corresponding to the meaning of the inferred word by comparing the meaning of the inferred word with predefined intents provided by the first NLU model 3312 . The device determination model 3314 may determine the type of the target device based on the first intent. In an embodiment, the device determination model 3314 may determine the type of the target device by using the first intent obtained using the first NLU model 3312. The device determination model 3314 provides the parsed text and target device information to the second voice assistant model 3320. In an embodiment, the device determination model 3314 may provide the determined identification information (eg, device ID) of the target device to the second voice assistant model 3320 together with the parsed text.

The first NLG model 3313 may register a function of the new device 2900 and generate a query message for generating or editing speech data.

The function comparison module 3315 may compare a function of the previously registered device 2000 with a function of the new device 2900. The function comparison module 3315 may determine whether a function of the previously registered device 2000 and a function of the new device 2900 are the same or similar. The function comparison module 3315 may identify a function identical or similar to a function of the previously registered device 2000 among functions of the new device 2900.

The function comparison module 3315 identifies a name representing a function supported by the new device 2900 from the specification information of the new device 2900, and the identified name is of the function supported by the previously registered device 2000. It can be determined whether it is the same or similar to the name. In this case, the DB 3340 may previously store information on names and similar words representing a predetermined function, and the functions of the previously registered device 2000 and the new device 2900 are the same based on the stored similar word information. Or you can judge whether they are similar.

In addition, the function comparison module 3315 may determine whether or not the functions are identical or similar by referring to speech data stored in the DB 3340. The function comparison module 3315 may determine whether the function of the new device 2900 is the same as or similar to the function of the previously registered device 2000 by using speech data related to the function of the previously registered device 2000. . In this case, the function comparison module 3315 analyzes the speech data using the first NLU model, and the function of the new device 2900 is registered in the device 2000 based on the meaning of words included in the speech data. It can be determined whether it is the same or similar to the function.

The function comparison module 3315 may determine whether a single function of the previously registered device 2000 and a single function of the new device 2900 are the same or similar. The function comparison module 3315 may determine whether the function set of the previously registered device 2000 and the function set of the new device 2900 are the same or similar.

The speech data acquisition module 3316 may acquire speech data related to a function of the new device 2900. The speech data acquisition module 3316 may extract speech data corresponding to a function determined to be the same or similar to the function of the new device 2900 from the speech data DB 3331 among functions of the previously registered device 2000. have.

The speech data acquisition module 3316 extracts speech data corresponding to the function set determined to be the same or similar to the function of the new device 2900 from the speech data DB 3331 among the function sets of the previously registered device 2000 can do. In this case, speech data corresponding to a function of the previously registered device 2000 and speech data corresponding to a function set of the previously registered device 2000 may be previously stored in the speech data DB 341.

The speech data acquisition module 3316 may edit a function and a set of functions determined to be the same or similar, and generate speech data corresponding to the edited functions. The speech data acquisition module 3316 may combine functions determined to be identical or similar, and generate speech data corresponding to the combined functions. Further, the speech data acquisition module 3316 may combine functions and function sets determined to be identical or similar, and generate speech data corresponding to the combined functions. Also, the speech data acquisition module 3316 may delete some functions among functions in the function set determined to be identical or similar, and generate speech data corresponding to the function set from which some functions are deleted.

The speech data acquisition module 3316 may expand speech data. The speech data acquisition module 3316 may generate similar speech data having the same meaning as the extracted or generated speech data but having different expressions by modifying the expression of the extracted or generated speech data.

The speech data acquisition module 3316 may output a query for registering an additional function and generating or editing speech data using the first NLG model 3313. The speech data acquisition module 3316 registers the function of the new device 2900 and provides guidance text or guidance voice data for guiding the generation of speech data to the user's device 2000 or the developer's device (not shown). I can. The speech data acquisition module 3316 may provide a list of functions different from those of the previously registered device 2000 among functions of the new device 2900 to the user's device 2000 or the developer's device (not shown). have. The speech data acquisition module 3316 may provide recommended speech data related to at least some of different functions to the user's device 2000 or the developer's device (not shown).

The speech data acquisition module 3316 may analyze a response to the query by using the first NLU model 3312. The speech data acquisition module 3316 may generate speech data related to functions of the new device 2900 based on the analyzed response. The speech data acquisition module 3316 may generate speech data related to functions of the new device 2900 by using the interpreted user's response or the interpreted developer's response, and recommend the generated speech data. The speech data acquisition module 3316 may select some of functions of the new device 2900 and generate speech data related to each of the selected partial functions. The speech data acquisition module 3316 may select some of the functions of the new device 2900 and generate speech data related to a combination of some of the selected functions. The speech data acquisition module 3316 may generate speech data related to the function of the new device 2900 using the first NLG model 3313 based on the identification values and attributes of the functions of the new device 2900. .

The action data generation module 3317 may generate action data for the new device 2900 based on the same or similar functions and speech data. For example, when the function corresponding to the utterance data is a single function, the action data generation module 3317 may generate action data including a detailed operation representing the single function. For example, when the function corresponding to the speech data is a function set, the action data generation module 3317 may generate detailed operations representing functions in the function set, and an execution order of the detailed operations. The action data generation module 3317 may generate action data using speech data generated in connection with a new function of the new device 2900. The action data generation module 3317 may generate action data corresponding to the generated speech data by identifying new functions of the new device 2900 related to the speech data and determining an execution order of the identified functions. The generated action data may be matched with speech data and similar speech data.

The model updater 3318 may create or update a second voice assistant model 3320 related to the new device 2900 using speech data and action data. The model updater 3318 includes speech data corresponding to the functions of the previously registered device 2000 related to the functions of the new device 2900, speech data newly generated in relation to the functions of the new device 2900, and extended speech. Using the data and action data, a second voice assistant model 3320 related to the new device 2900 may be generated or updated. The model updater 3318 may accumulate and store speech data and action data related to the new device 2900 in the speech data DB 341 and the action data DB 3342. In addition, the model updater 3318 may create or update a CAN (Concept Action Network), which is a capsule-type database included in the action plan management model 3323.

The second voice assistant model 3320 is a model specialized for a specific device and may determine an operation to be performed by a target device corresponding to a user's voice input. The second voice assistant model 3320 may include a second NLU model 3321, a second NLG model 3322, and an action plan management model 3323. The voice assistant server 3000 may include a second voice assistant model 3320 for each device type.

The second NLU model 3321 is an NLU model specialized for a specific device, and analyzes text and determines a second intent related to the user's intention based on the analysis result. The second NLU model 3321 may interpret a user's input voice in consideration of a device function. The second NLU model 3321 may be a model trained to obtain a second intent corresponding to the text by analyzing the text.

The second NLG model 3322 is an NLG model specialized for a specific device, and may generate a query message required to provide a voice assistant service to a user. The second NLG model 3322 may generate a natural language for conversation with a user in consideration of the function of the device.

The action plan management model 3323 is a device-specific model and may be a model that determines an operation to be performed by a target device corresponding to a user's voice input. The action plan management model 3323 may plan operation information to be performed by the new device 2900 in consideration of the function of the new device 2900.

The action plan management model 3323 may select detailed operations to be performed by the new device 2900 from the interpreted user's spoken voice, and plan an execution order of the selected detailed operations. The action plan management model 3323 may acquire operation information about a detailed operation to be performed by the new device 2900 by using the planning result. The operation information may be information related to detailed operations to be performed by the device, a correlation relationship between detailed operations, and an execution order of the detailed operations. The operation information includes, for example, functions to be executed by the new device 2900 in order to perform detailed operations, an execution order of functions, an input value required to execute the functions, and an output value output as a result of execution of the functions. It can, but is not limited thereto.

The action plan management model 3323 may manage information about a plurality of detailed operations of the new device 2900 and a relationship between the plurality of detailed operations. The relationship between each detailed operation and another detailed operation among a plurality of detailed operations may include information on other detailed operations that must be executed before executing the detailed operation in order to execute one detailed operation. have.

The action plan management model 3323 may include a concept action network (CAN), which is a database in the form of a capsule representing the actions of a device and a relationship between the actions. CAN (Concept Action Network) includes functions to be executed by the device to perform a specific operation, order of execution of functions, input values necessary to execute functions, and output values output as a result of execution of functions. It can be implemented as an ontology graph composed of knowledge triples representing the relationship between concepts.

The SDK interface module 3330 may transmit and receive data through the communication interface 3100 with the client device 1000 or a developer's device (not shown). The client device 1000 or the developer's device (not shown) can install a software development kit (SDK) for registering a new device, and from the voice assistant server 3000 through the installed software development kit. You can receive the GUI. The processor 3200 may register functions of the new device 2900 and provide a GUI for generating speech data to the user's device 2000 or the developer's device (not shown) through the SDK interface module 3330. have. The processor 3200 receives the user's response input through the GUI provided to the user's device 2000 from the user's device 2000 through the SDK interface module 3330, or provides a GUI provided to the developer's device (not shown). The developer's response input may be received from the developer's device (not shown) through the SDK interface module 3330. The SDK interface module 3330 may transmit and receive data through the IoT cloud server 4000 and the communication interface 3100.

The DB 3340 may store various types of information for the voice assistant service. The DB 3340 may include a speech data DB 341 and an action data DB 3342.

The speech data DB 3341 may store speech data related to functions of the client device 1000, the device 2000, and the new device 2900.

The action data DB 3342 may store action data related to functions of the client device 1000, the device 2000, and the new device 2900. The speech data stored in the speech data DB and the action data stored in the action data DB 3332 may be mapped to each other.

14 is a diagram illustrating another example of a voice assistant server according to an embodiment of the present disclosure.

Referring to FIG. 14, the voice assistant server 3000 may include one second voice assistant model 3320. In this case, the second voice assistant model 3320 may include a plurality of second NLU models 3324, 3325, and 3326. The plurality of second NLU models 3324, 3325, and 3326 may be NLU models specialized for each device type.

15 is a conceptual diagram illustrating an action plan management model according to an embodiment of the present disclosure.

Referring to FIG. 15, the action plan management model 3323 may include a speaker CAN 212, a mobile CAN 214, and a TV CAN 216. The speaker CAN 212 may include information on detailed operations including speaker control, media playback, weather, and TV control, and an action plan for storing concepts corresponding to each of the detailed operations in a capsule form. The mobile CAN 214 may include an action plan that stores information on detailed operations including SNS, mobile control, map, and QA, and concepts corresponding to each of the detailed operations in a capsule form. The TV CAN 216 may include information on detailed operations including shopping, media playback, education, and TV playback, and an action plan for storing concepts corresponding to each of the detailed operations in a capsule form. In one embodiment, a plurality of capsules included in each of the speaker CAN 212, the mobile CAN 214, and the TV CAN 216 are stored in a function registry, which is a component in the action plan management model 3323. Can be.

In one embodiment, the action plan management model 3323 determines detailed actions corresponding to the second intent and parameters determined by the voice assistant server 3000 interpreting the text converted from the voice input through the second NLU model. It may include a strategy registry that is necessary for this. When there are a plurality of action plans related to the text, the strategy registry may include reference information for determining one action plan. In one embodiment, the action plan management model 3323 may include a follow up registry in which information of a follow-up operation for suggesting a follow-up operation to a user in a specified situation is stored. The subsequent operation may include, for example, a subsequent speech.

In an embodiment, the action plan management model 3323 may include a layout registry that stores layout information output by the target device.

In one embodiment, the action plan management model 3323 may include a vocabulary registry in which vocabulary information included in capsule information is stored. In one embodiment, the action plan management model 3323 may include a dialog registry in which information about a dialog (or interaction) with a user is stored.

16 is a diagram illustrating a capsule database stored in an action plan management model according to an embodiment of the present disclosure.

Referring to FIG. 16, the capsule database stores detailed operations and relationship information about concepts corresponding to the detailed operations. The capsule database can be implemented in the form of a CAN (Concept Action Network). The capsule database may store a plurality of capsules 230, 240, 250. The capsule database may store detailed operations for executing operations related to a user's voice input, and input parameter values and output result values required for detailed operations in a concept action network (CAN) format.

The capsule database may store information related to operation for each device. In the embodiment shown in FIG. 19, the capsule database may store a plurality of capsules 230, 240, 250 related to operations performed by a specific device, for example, a TV. In one embodiment, one capsule (eg, capsule A 230) may correspond to one application. One capsule may include at least one or more detailed operations and at least one or more concepts for performing a specified function. For example, capsule A 230 includes a detailed operation 231a and a concept 231b corresponding to the detailed operation 231a, and the capsule B 240 includes a plurality of detailed operations 241a, 242a, 243a. ), and a plurality of concepts 241b, 242b, and 243b corresponding to each of the plurality of detailed operations 241a, 242a, and 243a.

The action plan management model 210 may generate an action plan for performing an operation related to a user's voice input by using a capsule stored in a capsule database. For example, the action plan management model 210 may generate an action plan using a capsule stored in a capsule database. For example, the action plan management model 210 includes detailed operations 231a and concept 231b of capsule A 230, detailed operations 241a, 242a, 243a and concepts 241b of capsule B 240. , 242b, 243b) and the detailed operation 251a and concept 251b of the capsule C 250, an action plan 260 related to operations to be performed by the device may be generated.

17 is a block diagram of an IoT cloud server according to an embodiment of the present disclosure.

Referring to FIG. 17, the IoT cloud server 4000 according to an embodiment of the present disclosure includes a communication interface 4100, a processor 4200, and a storage unit 4300, and the storage unit 4300 is an SDK interface module. 4310, a function comparison module 4320, a device registration module 4330, and a DB 4340 may be included. Further, the DB 4340 may include a device function DB 441 and an action data DB 4432.

The communication interface 4100 communicates with the client device 1000, the device 2000, and the voice assistant server 3000. The communication interface 4100 may include one or more components for communication with the client device 1000, the device 2000, and the voice assistant server 3000.

The processor 4200 typically controls the overall operation of the IoT cloud server 4000. For example, the processor 4200 may control functions of the IoT cloud server 4000 in the present specification by executing programs stored in the storage unit 4300.

The storage unit 4300 may store a program for processing and control of the processor 4200 and may store data related to functions of the device 2000. The storage unit 4300 includes a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (eg, SD or XD memory, etc.), RAM (Random Access Memory) SRAM (Static Random Access Memory), ROM (ROM, Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, magnetic It may include at least one type of storage medium among a disk and an optical disk.

The programs stored in the storage unit 4300 can be classified into a plurality of modules according to their functions, for example, the SDK interface module 4310, the function comparison module 4320, and the device registration module 4330. It can be classified as

The SDK interface module 4310 may transmit and receive data through the voice assistant server 3000 and the communication interface 4100. The processor 4200 may provide functional information of the device 2000 to the voice assistant server 3000 through the SDK interface module 4310.

When the function comparison module 4320 is included in the IoT cloud server 4000, the function comparison module 4320 may serve as the function comparison model 3315 of the voice assistant server 3000 described above.

In this case, the function comparison module 4320 may compare the function of the previously registered device 2000 and the function of the new device 2900. The function comparison module 4320 may determine whether a function of the previously registered device 2000 and a function of the new device 2900 are the same or similar. The function comparison module 4320 may identify a function identical or similar to a function of the previously registered device 2000 among functions of the new device 2900.

The function comparison module 4320 identifies a name representing a function supported by the new device 2900 from the specification information of the new device 2900, and the identified name is of the function supported by the previously registered device 2000. It can be determined whether it is the same or similar to the name. In this case, the DB 4340 may previously store information on names and similar words representing a predetermined function, and the functions of the previously registered device 2000 and the new device 2900 are the same based on the stored similar word information. Or you can judge whether they are similar.

In addition, the function comparison module 4320 may determine whether or not the functions are identical or similar by referring to speech data stored in the DB 4340. The function comparison module 3315 may determine whether the function of the new device 2900 is the same as or similar to the function of the previously registered device 2000 by using speech data related to the function of the previously registered device 2000. . The function comparison module 4320 may determine whether a single function of the previously registered device 2000 and a single function of the new device 2900 are the same or similar. The function comparison module 4320 may determine whether the function set of the previously registered device 2000 and the function set of the new device 2900 are the same or similar.

The device registration module 4330 may register a device for voice assistant service. When a new device 2900 is identified, the device registration module 4330 receives information on functions of the new device 2900 from the voice assistant server 3000, and registers the received information in the DB 4340. can do. The information on the functions of the new device 2900 may include, for example, functions supported by the new device 2900, action data related to the functions, but is not limited thereto.

The DB 4340 may store device information necessary for the voice assistant service. The DB 4340 may include a device function DB 441 and an action data DB 4432. The device function DB 4340 may store function information of the client device 1000, the device 2000, and the new device 2900. The function information may include a device function identification value, a function name, and information on a function property, but is not limited thereto. The action data DB 4432 may store action data related to functions of the client device 1000, the device 2000, and the new device 2900.

18 is a block diagram of a client device according to an embodiment of the present disclosure.

Referring to FIG. 18, a client device 1000 according to an embodiment of the present disclosure includes an input unit 1100, an output unit 1200, a processor 1300, a memory 1400, and a communication interface 1500, The memory 1400 may include an SDK module 1420.

According to an embodiment, the device 2000 may operate as the client device 1000, or the new device 2900 registered after the new device 2900 is registered may operate as the client device 1000. Device 2000 or new device 2900 may include the configurations of FIG. 18.

The input unit 1100 refers to a means for a user to input data for controlling the client device 1000. For example, the input unit 1100 includes a key pad, a dome switch, and a touch pad (contact type capacitance method, pressure type resistive film method, infrared detection method, surface ultrasonic conduction method, integral tension type). Measurement method, piezo effect method, etc.), a jog wheel, a jog switch, and the like, but are not limited thereto.

The input unit 1100 may receive a user input for registering the new device 2900.

The output unit 1200 may output an audio signal, a video signal, or a vibration signal, and the output unit 1200 may include at least one of a display unit, an audio output unit, and a vibration motor.

The processor 1300 controls the overall operation of the client device 1000. For example, the processor 1300 may generally control the input unit 1100, the output unit 1200, the memory 1400, and the communication interface 1500 by executing programs stored in the memory 1400. .

The processor 1300 may request an input for registering a function of the new device 2900 from the user. The processor 1300 may perform an operation for registering a new device 2900 together with the voice assistant server 300 by controlling the SDK module 1420 to be described later.

The processor 1300 may receive and output a query message for generating and editing speech data related to a function of the new device 2900 from the voice assistant server 3000. The processor 1300 may provide a user with a list of functions different from those of the previously registered device 1300 among functions of the new device 2900. The processor 1300 may provide the user with recommended speech data related to at least some of the functions of the new device 2900.

The processor 1300 may receive a user's response to the query message. The processor 1300 may provide the user's response to the voice assistant server 3000 to allow the voice assistant server 3000 to generate speech data and action data related to functions of the new device 2900.

The communication interface 1500 may include one or more components for communicating with the voice assistant server 3000, the IoT cloud server 4000, the device 2000, and the new device 2900. For example, the communication interface 1500 may include a short range communication unit, a mobile communication unit, and a broadcast reception unit.

The short-range wireless communication unit includes a Bluetooth communication unit, a Bluetooth Low Energy (BLE) communication unit, a Near Field Communication unit, a WLAN (Wi-Fi) communication unit, a Zigbee communication unit, an infrared (IrDA, infrared) communication unit. Data Association) communication unit, WFD (Wi-Fi Direct) communication unit, UWB (ultra wideband) communication unit, and may include an Ant+ communication unit, but is not limited thereto.

The mobile communication unit transmits and receives a radio signal with at least one of a base station, an external terminal, and a server on a mobile communication network. Here, the wireless signal may include a voice call signal, a video call signal, or various types of data according to transmission/reception of text/multimedia messages.

The broadcast receiver receives a broadcast signal and/or broadcast-related information from outside through a broadcast channel. Broadcast channels may include satellite channels and terrestrial channels. According to an implementation example, the first device 1000 may not include the broadcast receiver 1530.

The memory 1400 may store a program for processing and controlling the processor 1300 and may store data input to the client device 1000 or output from the client device 1000.

The memory 1400 is a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (for example, SD or XD memory), and RAM. (RAM, Random Access Memory) SRAM (Static Random Access Memory), ROM (ROM, Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, magnetic disk And at least one type of storage medium among optical disks.

Programs stored in the memory 1400 can be classified into a plurality of modules according to their functions. For example, the SDK module 1420, a UI module (not shown), a touch screen module (not shown), and a notification module ( Not shown), etc.

The SDK module 1420 may be executed by the processor 1300 to perform an operation required to register a new device 2900. The SDK module 1420 may be downloaded from the voice assistant server 3000 and installed in the client device 1000. The SDK module 1420 may output a GUI for registering the new device 2900 on the screen of the client device 1000. If the client device 1000 is a device that does not include a display device, the SDK module 1420 may cause the client device 1000 to output a voice message for registration of the new device 2900. The SDK module 1420 may allow the client device 1000 to receive a response from a user and provide it to the voice assistant server 3000.

An embodiment of the present disclosure may also be implemented in the form of a recording medium including instructions executable by a computer, such as a program module executed by a computer. Computer-readable media can be any available media that can be accessed by a computer, and includes both volatile and nonvolatile media, removable and non-removable media. Further, the computer-readable media may include computer storage media and communication media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Communication media may typically contain computer readable instructions, data structures, or other data in a modulated data signal such as a program module.

In addition, in the present specification, the “unit” may be a hardware component such as a processor or a circuit, and/or a software component executed by a hardware configuration such as a processor.

In addition, in the present specification, “including at least one of a, b, or c” means “including only a, only b, only c, including a and b, or including b and c, It may mean including a and c, or including all of a, b and c.

The above description of the present disclosure is for illustrative purposes only, and those skilled in the art to which the present disclosure pertains will be able to understand that it is possible to easily transform it into other specific forms without changing the technical spirit or essential features of the present disclosure will be. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not limiting. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as being distributed may also be implemented in a combined form.

The scope of the present disclosure is indicated by the claims to be described later rather than the detailed description, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present disclosure. do.

Claims (20)

In the method for the server to register a new device for voice assistant service,
Acquiring specification information indicating functions of at least one pre-registered device;
Comparing functions of the previously registered device and functions of the new device based on the specification information;
Identifying functions corresponding to functions of the previously registered device among functions of the new device based on the comparison result;
Obtaining pre-registered speech data related to at least some of the identified functions;
Generating action data for the new device based on the identified functions and the acquired pre-registered speech data; And
Storing the obtained speech data and the generated action data in association with the new device;
Including,
The action data includes data on a series of detailed functions of the new device corresponding to the acquired speech data.
The method of claim 1,
The operation of identifying the corresponding functions,
Among the functions of the new device, functions of the same or similar purpose as those of the previously registered device are identified.
The method of claim 1,
The comparing operation,
Identifying a combination of functions of the at least one pre-registered device corresponding to the combination of functions of the new device by comparing a combination of functions of the at least one previously registered device with a combination of functions of the new device Being, the way.
The method of claim 3,
The at least one pre-registered device includes a plurality of pre-registered devices,
The comparing operation,
And comparing a combination of a function of a first device among the plurality of pre-registered devices and a function of a second device among the plurality of pre-registered devices with a combination of functions of the new device.
The method of claim 1,
The comparing operation,
And deleting some of the functions in the function set of the at least one pre-registered device, and comparing the function remaining after the deletion with the function of the new device.
The method of claim 1,
The operation of obtaining the pre-registered speech data may include: extracting speech data corresponding to the identified functions from a DB; And
Editing the extracted speech data;
The method comprising a.
The method of claim 1,
Identifying functions different from those of the previously registered device among functions of the new device; And
Providing a list of the different functions to a client device;
The method further comprising.
The method of claim 7,
The method, wherein the list of different functions provided to the client device is used to generate speech data corresponding to the different functions by an SDK (Software Developing kit) module installed in the client device.
The method of claim 1,
The obtained speech data and the generated action data are used to create or update a voice assistant model specialized for the new device.
The method of claim 1,
The obtained speech data and the generated action data are used to generate or update an ontology graph including knowledge triples representing actions of the new device and relationships between actions.
In the server for registering a new device for voice assistant service,
Communication interface;
A memory storing a program including one or more instructions; And
A processor that executes one or more instructions of a program stored in the memory; Including,
The processor,
Obtaining specification information representing functions of at least one pre-registered device, comparing functions of the pre-registered device and functions of the new device based on the specification information, and based on the comparison result , Among the functions of the new device, identify functions corresponding to the functions of the previously registered device, obtain pre-registered speech data related to at least some of the identified functions, and the identified functions and the Based on the acquired pre-registered speech data, action data for the new device is generated, and the obtained speech data and the generated action data are stored in a predetermined DB (DataBase) in association with the new device,
The action data includes data on a series of detailed functions of the new device corresponding to the acquired speech data.
The method of claim 11,
The processor, among the functions of the new device, identifies functions of the same or similar purpose as the functions of the previously registered device.
The method of claim 11,
The processor, by comparing the combination of functions of the at least one previously registered device with the combination of functions of the new device, the combination of functions of the at least one pre-registered device corresponding to the combination of functions of the new device To identify the server.
The method of claim 13,
The at least one pre-registered device includes a plurality of pre-registered devices,
Wherein the processor compares a combination of a function of a first device among the plurality of pre-registered devices and a function of a second device among the plurality of pre-registered devices with a combination of functions of the new device.
The method of claim 11,
Wherein the processor deletes some of the functions in the function set of the at least one pre-registered device, and compares the function remaining after the deletion with the function of the new device.
The method of claim 11,
The processor, the operation of obtaining the pre-registered speech data, extracts speech data corresponding to the identified functions from the DB, and edits the extracted speech data.
The method of claim 11,
The processor, among functions of the new device, identifies functions different from those of the previously registered device, and provides a list of the different functions to a client device.
The method of claim 17,
The list of the different functions provided to the client device is used to generate speech data corresponding to the different functions by a software developing kit (SDK) module installed in the client device.
The method of claim 11,
The obtained speech data and the generated action data are used to create or update a voice assistant model specialized for the new device.
A computer-readable recording medium storing a program for executing the method of claim 1 on a computer.

Images