KR20200127814A - Server for seleting a target device according to a voice input, and controlling the selected target device, and method for operating the same - Google Patents

Abstract

Disclosed are a server for controlling a device based on a voice input of a user and an operation method thereof. According to one embodiment of the present disclosure, the server determines a type of target device related to a voice input of a user received from a client device in a network environment including a plurality of devices, determines the target device based on the determined type and device information related to the plurality of devices, and obtains operation information so that the determined target device performs an operation.

Description

A server that determines the target device based on the user's input and controls the target device and its operation method {SERVER FOR SELETING A TARGET DEVICE ACCORDING TO A VOICE INPUT, AND CONTROLLING THE SELECTED TARGET DEVICE, AND METHOD FOR OPERATING THE SAME}

The present disclosure relates to a server for determining a target device to perform an operation according to a user's intention included in a voice input received from a user in a network environment including a plurality of devices, and controlling the determined target device, and an operation method thereof. .

With the development of multimedia technology and network technology, users can receive various services using devices. In particular, with the development of speech recognition technology, a user can input a voice (for example, speech) to a device and receive a response message according to the voice input through a service providing agent.

However, in a home network environment including a plurality of devices, when a user wants to receive a service through a device other than a client device that interacts through voice input, etc., it is troublesome to select a service agent directly. have. In particular, since the types of services that can be provided by each of a plurality of devices are different, there is a demand for a technology capable of effectively providing a service by grasping an intention included in a user's voice input.

When determining the intention included in the user's voice input, artificial intelligence (AI) technology may be used, and rule-based natural language understanding technology (NLU) may be used.

The present disclosure relates to a server and a method of operation thereof, and more specifically, to determine a user's intention from a voice input in a network environment including a plurality of devices, and automatically determine a device to perform an operation according to the identified intention. It is an object of the present invention to provide a server for controlling the determined device and an operation method thereof.

In order to solve the above technical problem, an embodiment of the present disclosure includes the steps of receiving a user's voice input from a client device, performing ASR (Automatic Speech Recognition), and converting the received voice input into text, the Determining a type of a target device related to the text by applying a first natural language understanding model to the text, a second natural language understanding model corresponding to the determined type of the target device among a plurality of second natural language understanding models 2 Selecting a natural language understanding model, determining a target device based on the determined type and device information of the plurality of devices, and analyzing the text by applying the selected second natural language understanding model to at least part of the text It provides a method of controlling a device by a server based on a voice input, comprising the step of obtaining operation information on an operation to be performed by the target device, and transmitting the obtained operation information to the target device. .

In order to solve the above technical problem, another embodiment of the present disclosure provides a client device and a communication interface for performing data communication with a plurality of devices, a memory for storing a program including one or more instructions, and a memory. And a processor that executes one or more instructions of the stored program, wherein the processor controls the communication interface to receive a user's voice input from the client device, performs Automatic Speech Recognition (ASR), and receives the received voice input. Is converted to text, and a first natural language understanding model is applied to the text to determine a type of a target device related to the text, and to the determined type among a plurality of second natural language understanding models. Select a corresponding second natural language understanding model, determine a target device based on the determined type and device information of the plurality of devices, and analyze the text by applying the selected second natural language understanding model to at least part of the text By doing so, it provides a server, which obtains operation information on an operation to be performed by the target device and controls the communication interface to transmit the obtained operation information to the target device.

In order to solve the above technical problem, another embodiment of the present disclosure provides a computer-readable recording medium in which a program for execution on a computer is recorded, the recording medium receiving a user's voice input from a client device. , Performing ASR (Automatic Speech Recognition) to convert the received speech input into text, by analyzing the text using a first natural language understanding model, the type of target device related to the text Determining, determining a target device based on the determined type and device information of the plurality of devices, analyzing the text using a second natural language understanding model corresponding to the determined type, It includes obtaining motion information, and transmitting the obtained motion information to an IoT cloud server.

The present disclosure may be easily understood by a combination of the following detailed description and accompanying drawings.
1 is a diagram illustrating a network environment including a server and a plurality of devices.
2A and 2B are diagrams showing an embodiment of a voice assistant executable in the server of the present disclosure.
3 is a conceptual diagram showing an embodiment in which the server of the present disclosure recognizes the user's intention from the user's voice input, determines a target device in relation to the user's intention, and controls the target device.
4 is a flowchart illustrating a method of determining a target device to perform an operation according to a user's intention from a user's voice input and controlling the target device by the server of the present disclosure.
5 is a flowchart illustrating a method of determining, by the server of the present disclosure, a target device to perform an operation according to a user's intention from a user's voice input and controlling the target device.
6 is a flowchart illustrating a specific method of determining a type of a target device through text converted from a voice input by the server of the present disclosure.
7 is a flowchart illustrating a method of determining a target device based on information on a plurality of devices and a target device type determined from text by the server of the present disclosure.
8 is a flowchart illustrating a method of determining a target device based on a user's response input when the server of the present disclosure cannot determine the target device even when considering information of a plurality of devices.
9 is a flowchart illustrating a method of obtaining, by the server of the present disclosure, operation information regarding an operation to be performed by a target device using a second natural language understanding model.
10 is a diagram illustrating a method of determining, by a server of the present disclosure, a first intent and a second intent from text and obtaining operation information from an action plan management model.
11 is a flowchart illustrating the operation of a client device, a server, an artificial intelligence assistant engine, and an IoT cloud server of the present disclosure.
12 is a flowchart illustrating operations of a client device, a server, and an IoT cloud server according to the present disclosure.
13 is a flowchart illustrating operations of a client device, a server, and an IoT cloud server according to the present disclosure.
14 is a block diagram illustrating a client device, a server, and an IoT cloud server according to the present disclosure.
15 is a diagram showing an example of a system architecture of a program executed by a server of the present disclosure.
16 is a diagram illustrating an example of a device type classifier of a program executed by a server of the present disclosure.
17 is a diagram showing a device disambiguation unit of a program executed by the server of the present disclosure.
18 is a diagram showing an example of a response message control unit (Response Execute Manager) of a program executed by the server of the present disclosure.
19 is a diagram illustrating an example of an intelligent device resolver of a program executed by the server of the present disclosure.
20 is a conceptual diagram illustrating an action plan management model according to an embodiment of the present disclosure.
21 is a diagram illustrating a capsule database according to an embodiment of the present disclosure.
22 is a block diagram showing components of a device according to the present disclosure.

The terms used in the embodiments of the present specification have selected general terms that are currently widely used as possible while taking the functions of the present disclosure into consideration, but this may vary depending on the intention or precedent of a technician working in the field, the emergence of new technologies, etc. . In addition, in certain cases, there are terms arbitrarily selected by the applicant, and in this case, the meaning of the terms will be described in detail in the description of the corresponding embodiment. Therefore, the terms used in the present specification should be defined based on the meaning of the term and the contents of the present disclosure, not the name of a simple term.

Singular expressions may include plural expressions unless the context clearly indicates otherwise. Terms used herein, including technical or scientific terms, may have the same meaning as commonly understood by a person of ordinary skill in the technical field described herein.

When a part of the specification is said to "include" a certain component, it means that other components may be further included rather than excluding other components unless otherwise stated. In addition, terms such as "... unit" and "... module" described in this specification mean a unit that processes at least one function or operation, which is implemented as hardware or software, or is a combination of hardware and software. Can be implemented.

The expression "configured to (configured to)" used in this specification is, for example, "suitable for", "having the capacity to" depending on the situation. It can be used interchangeably with ", "designed to", "adapted to", "made to", or "capable of". The term "configured to (or set)" may not necessarily mean only "specifically designed to" in hardware. Instead, in some situations, the expression "a system configured to" may mean that the system "can" along with other devices or parts. For example, the phrase “a processor configured (or configured) to perform A, B, and C” means a dedicated processor (eg, an embedded processor) for performing the operation, or by executing one or more software programs stored in memory, It may mean a generic-purpose processor (eg, a CPU or an application processor) capable of performing corresponding operations.

In the present specification, the'first natural language understanding model' is a model trained to obtain a first intent corresponding to the text by analyzing the text converted from the speech signal. The first natural language understanding model may be a model trained to determine a device type to perform an operation intended by the user when text is input. The first natural language understanding model may determine a first intent by interpreting the input text, and the first intent may be used to determine a device type. Further, the first natural language understanding model may be used to determine a target device by interpreting the input text.

In the present specification, the'second natural language understanding model' is an artificial intelligence model learned to analyze text related to a specific type of device. In the present specification, there are a plurality of second natural language understanding models, and each of the plurality of second natural language understanding models corresponds to a plurality of different device types. When the device type is determined through the first natural language understanding model, a second natural language understanding model corresponding to the device type is selected from among the plurality of second natural language understanding models. Text converted from the speech signal is input to the selected second natural language understanding model. In this case, the input text may be a part of the text converted from the voice signal. The second natural language understanding model may be a model trained to determine second intents and parameters related to an operation intended by the user by analyzing the input text. The second natural language understanding model may be a model trained to determine a function related to a specific device type when text is input. The storage capacity of the plurality of second natural language understanding models may be larger than the storage capacity of the first natural language understanding model.

In this specification, the'intent' is information indicating the intention of the user determined by interpreting the text. The intent is information indicating the user's intention to speak, and may be information indicating the operation of the target device requested by the user. The intent may be determined by interpreting the text using a natural language understanding (NLU) model. For example, if the text converted from the user's voice input is "Play Movie Avengers on TV", the intent may be "Play content". For another example, when the text converted from the user's voice input is "Turn down the air conditioner temperature to 18 °C", the intent may be "temperature control".

The intent may include not only information indicating the user's utterance intention (hereinafter, intention information), but also a numerical value corresponding to information indicating the user's intention. The numerical value may indicate the probability that the text is associated with information indicating a specific intent. When a plurality of pieces of information representing the user's intention are obtained as a result of analyzing the text using the natural language understanding model, intention information having a maximum numerical value corresponding to each intention information may be determined as the intent.

The'first intent' is information indicating the user's speech intention included in the text, and may be used to determine at least one type of a plurality of target device types. The first intent may be determined by analyzing the text using the first natural language understanding model.

The'second intent' is information indicating the user's speech intention included in the text, and may be used to determine an operation to be performed by a specific type of device. In addition, the second natural language understanding model may be a model trained to determine an operation of a device related to a user's voice input by analyzing text. The second intent may be determined by analyzing the text using the second natural language understanding model.

In the present specification,'parameter' means variable information for determining detailed operations of a target device related to an intent. The parameter is information related to the intent, and a plurality of types of parameters may correspond to one intent. The parameter may include not only variable information for determining operation information of the target device, but also a numerical value indicating a probability that the text is related to the variable information. As a result of analyzing the text using the second natural language understanding model, a plurality of variable information indicating parameters may be obtained. In this case, variable information having a maximum numerical value corresponding to each variable information may be determined as a parameter. For example, if the text converted from the user's voice input is "Play Movie Avengers", the intent is'play content', and the parameter is'movie' which is the genre of the content to be played and/or the content to be played. It may be the title of'Avengers'.

In the present specification, the'action plan management model' may be a model for managing operation information related to a detailed operation of a device in order to generate detailed operations to be performed by the target device and an execution order of the detailed operations. The action plan management model may manage detailed operations of a device for each device type and operation information regarding a relationship between the detailed operations. The action plan management model may plan detailed operations to be performed by the device and an execution order of detailed operations based on the second intent and parameters output from the text.

In this specification, an operation of a device may mean at least one action performed by the device by executing a specific function in the device. The operation may represent at least one action performed by the device by executing an application on the device. The operation may represent, for example, playing a video, playing music, writing an email, receiving weather information, displaying news information, executing a game, taking a picture, and the like performed by the device through execution of an application. However, the operation is not limited to the above-described example.

The operation of the device may be performed based on information on detailed operations output from the action plan management model. The device may perform at least one action by executing a function corresponding to the detailed operation output from the action plan management model. The device may store a command for executing a function corresponding to a detailed operation, and when a detailed operation is determined, the device may execute a specific function by determining a command corresponding to the detailed operation and executing the command.

Also, the device may store a command for executing an application corresponding to a detailed operation. The instructions for executing the application may include instructions for executing the application itself and instructions for executing detailed functions constituting the application. When a detailed operation is determined, the device may execute an application by executing a command for executing an application corresponding to the detailed operation, and execute a detailed function by executing a command for executing a detailed function of the application corresponding to the detailed operation.

In this specification,'action information' may be information related to detailed operations to be performed by the device, a relationship between each detailed operation and other detailed operations, and an execution order of the detailed operations. The association relationship between each detailed operation and another detailed operation includes information about another operation that must be executed before executing the operation in order to execute one operation. For example, when the operation to be performed is “music playback”, “power on” may be another detailed operation that must be essentially executed before the “music playback” operation. The operation information may include, for example, functions to be executed by the target device in order to perform a specific operation, an execution order of functions, an input value necessary to execute the functions, and an output value output as a result of execution of the functions. , Is not limited thereto.

The IoT cloud server is a server that acquires, stores, and manages device information for each of a plurality of devices. The IoT cloud server may acquire, determine, or generate a control command capable of controlling a device using stored device information. The IoT cloud server may transmit a control command to a device determined to perform an operation based on the operation information. The IoT cloud server may receive a result of performing the operation according to the control command from the device that performed the operation. The IoT cloud server may be configured as a hardware device independent from the'server' described herein, but is not limited thereto. The IoT cloud server may be a component of the'server' of the present specification, or may be a server designed to be classified by software.

Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art may easily implement the embodiments. However, the present disclosure may be implemented in various different forms and is not limited to the exemplary embodiments described herein.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings.

1 is a diagram illustrating a network environment including a client device 110, a plurality of devices 121 to 127, and a server 300.

Referring to FIG. 1, a client device 110, a plurality of devices 121 to 127, a server 300, and an IoT cloud server 400 are interconnected by wired communication or wireless communication, and may perform communication. have. In one embodiment, the client device 110 and the plurality of devices 121 to 127 may be directly connected to each other through a communication network, but are not limited thereto. The client device 110 and the plurality of devices 121 to 127 may be connected to the server 300, and the client device 110 may be connected to the plurality of devices 121 to 127 through the server. In addition, the client device 110 and the plurality of devices 121 to 127 may be connected to the IoT cloud server 400. In another embodiment, each of the client device 110 and the plurality of devices 121 to 127 may be connected to the server 300 through a communication network, and may be connected to an external IoT cloud server 400 through the server 300. . In another embodiment, the client device 110 and the plurality of devices 121 to 127 may be connected through a gateway or a relay.

The client device 110, a plurality of devices 121 to 127, the server 300 and the IoT cloud server 400 include a local area network (LAN), a wide area network (WAN), and a value-added communication network ( Value Added Network (VAN), a mobile radio communication network, a satellite communication network, or a combination thereof. Wireless communication methods include, for example, wireless LAN (Wi-Fi), Bluetooth, Bluetooth low energy, Zigbee, WFD (Wi-Fi Direct), UWB (ultra wideband), infrared communication (IrDA, infrared data). Association), NFC (Near Field Communication), and the like, but are not limited thereto.

In one embodiment, the client device 110 may receive a user's voice input. At least one of the plurality of devices 121 to 127 may be a target device that performs a specific operation by receiving a control command from the server 300 and/or the IoT cloud server 400. At least one of the plurality of devices 121 to 127 may be controlled to perform a specific operation based on a user's voice input received by the client device 100. In one embodiment, at least one of the plurality of devices 121 to 127 may receive a control command from the client device 110 without receiving a control command from the server 300 and/or the IoT cloud server 400 have.

The client device 110 may receive a voice input (eg, speech) from a user. In one embodiment, the client device 110 may include a voice recognition module. In one embodiment, the client device 110 may include a speech recognition module with limited functionality. For example, the client device 110 has a function of detecting a designated voice input (for example, a wake-up input such as'Hi Bixby','OK Google', etc.) or preprocessing a voice signal obtained from some voice inputs. It may include a speech recognition module having a function. In FIG. 1, the client device 110 is illustrated as an AI speaker, but is not limited thereto. In an embodiment, any one of the plurality of devices 121 to 127 may be the client device 110.

The client device 110 may receive a user's voice input through a microphone and transmit the received voice input to the server 300. In an embodiment, the client device 110 may obtain a voice signal from the received voice input and transmit the voice signal to the server 300.

In the embodiment shown in FIG. 1, the plurality of devices 121 to 127 are a light 121, an air conditioner 122, a TV 123, a robot cleaner 124, a washing machine 125, a weight scale 126, and a refrigerator. (127), but is not limited thereto. For example, the plurality of devices 121 to 127 include a smartphone, a tablet personal computer, a mobile phone, a video phone, an e-book reader, and a desktop PC ( desktop personal computer), laptop PC (laptop personal computer), netbook computer, workstation, server, personal digital assistant (PDA), portable multimedia player (PMP), MP3 player, mobile medical device, camera It may include at least one of (camera) or a wearable device. In one embodiment, the plurality of devices 121 to 127 may be home appliances. Home appliances include, for example, television, digital video disk (DVD) player, audio, refrigerator, air conditioner, vacuum cleaner, oven, microwave oven, washing machine, air cleaner, set-top box, and home automation control. It may include at least one of a home automation control panel, a security control panel, a game console, an electronic key, a camcorder, or an electronic frame.

The server 300 may determine a type of a target device to perform an operation intended by the user based on the received voice signal. The server 300 may receive a voice signal, which is an analog signal, from the client device 110 and convert the voice part into computer-readable text by performing Automatic Speech Recognition (ASR). The server 300 may analyze the converted text using a first natural language understanding (NLU) model and determine the type of the target device based on the analysis result.

The server 300 may determine an operation to be performed in the target device requested by the user by using the second natural language understanding model corresponding to the determined type of the target device.

The server 300 may receive information on a plurality of devices 121 to 127 from the IoT cloud server 400. The server 300 may determine a target device using the received information of the plurality of devices 121 to 127 and the determined target device type. In addition, the server 300 may control the target device through the IoT cloud server 400 so that the target device may execute the determined operation. The operation of the server 300 will be described in detail with reference to FIGS. 4 to 9.

The IoT cloud server 400 may store information on a plurality of pre-registered devices 121 to 127 connected through a network. In one embodiment, the IoT cloud server 400 includes identification values (eg, device ID information) of a plurality of devices 121 to 127, a device type of each of the plurality of devices 121 to 127, and a plurality of devices 121 To 127) at least one of respective function performance capability information (capability) may be stored. In an embodiment, the IoT cloud server 400 may store power on/off of each of the plurality of devices 121 to 127 or state information regarding an operation being executed.

The IoT cloud server 400 may transmit a control command for executing the determined operation to a target device among the plurality of devices 121 to 127. The IoT cloud server 400 may receive information on the determined target device and information on the determined operation from the server 300, and transmit a control command to the target device based on the received information.

2A and 2B are diagrams illustrating an embodiment of a voice assistant 200 executable in the server 300 of the present disclosure.

2A and 2B, the voice assistant 200 may be implemented in software. The voice assistant 200 may be configured to determine a user's intention from the user's voice input and control a target device related to the user's intention. When a device controlled through the voice assistant 200 is added, the voice assistant 200 includes a first assistant model 200a configured to update an existing model to a new model through learning, etc., and corresponding to the added device. It may include a second assistant model 200b configured to add the model to the existing model.

The first assistant model 200a is a model for determining a target device related to a user intention by analyzing a user voice input. The first assistant model 200a may include an ASR model 202, an NLG model 204, a first natural language understanding model 300a, and a device dispatcher model 310. In an embodiment, the device dispatcher model 310 may include a first natural language understanding model 300a. In another embodiment, the device dispatcher model 310 and the first natural language understanding model 300a may be configured as separate components. The device dispatcher model 310 is a model for determining a target device by using the analysis result of the first natural language understanding model 300a. The device dispatcher model 310 may include a plurality of detailed models.

The second assistant model 200b is a model specialized for a specific device and is a model that determines an operation to be performed by a target device corresponding to a user's voice input. In FIG. 2A, the second assistant model 200b may include a plurality of second natural language understanding models 300b, an NLG model 206, and an action plan management model 210. The plurality of natural language understanding models 300b may respectively correspond to a plurality of different devices.

Referring to FIG. 2B, the second assistant model 200b may include a plurality of action plan management models and a plurality of NLG models. In FIG. 2B, each of the plurality of second natural language understanding models included in the second assistant model 200b corresponds to the second natural language understanding model 300b of FIG. 2A, and includes a plurality of second natural language understanding models included in the second assistant model 200b. Each of the NLG models corresponds to the NLG model 206 of FIG. 2A, and each of the plurality of action plan management models included in the second assistant model 200b may correspond to the action plan management module 210 of FIG. 2A.

In FIG. 2B, each of the plurality of action plan management models may be configured to correspond to each of the plurality of second natural language understanding models. In addition, each of the plurality of NLG models may be configured to correspond to each of the plurality of second natural language understanding models. In another embodiment, one NLG model may correspond to a plurality of second natural language understanding models, and one action plan management model may be configured to correspond to a plurality of second natural language understanding models.

In FIG. 2B, when a device controlled through the voice assistant 200 is added, the first natural language understanding model 300a may be configured to be updated to a new model through learning or the like. In addition, when the device dispatcher model 310 is configured to include the first natural language understanding model 300a, the device dispatcher model 310 is performed through learning when a device controlled through the voice assistant 200 is added. The entire existing model can be configured to be updated with a new model. The first natural language understanding model 300a or the device dispatcher model 310 may be an artificial intelligence model.

2B, the second assistant model 200b includes a second natural language understanding model, an NLG model, and an action plan management model corresponding to the added device when a device controlled through the voice assistant 200 is added to the existing model. It can be updated by adding. The second natural language understanding model, NLG model, and action plan management model may be models implemented with a rule-based system.

In the embodiment of FIG. 2B, the second natural language understanding model, the NLG model, and the action plan management model may be an artificial intelligence model. The second natural language understanding model, the NLG model, and the action plan management model may be managed as one module according to each corresponding device. In this case, the second assistant model 200-b may include a plurality of second assistant models 200b-1, 200b-2, and 200b-3 respectively corresponding to a plurality of devices. For example, the second natural language understanding model corresponding to the TV, the NLG model corresponding to the TV, and the action plan management model corresponding to the TV may be managed by the second assistant model 200b-1 corresponding to the TV. In addition, the second natural language understanding model corresponding to the speaker, the NLG model corresponding to the speaker, and the action plan management model corresponding to the speaker may be managed as the second assistant model 200b-2 corresponding to the speaker. In addition, the second natural language understanding model corresponding to the refrigerator, the NLG model corresponding to the refrigerator, and the action plan management model corresponding to the refrigerator may be managed by the second assistant model 200b-3 corresponding to the refrigerator.

3 is a conceptual diagram showing an embodiment in which the server 300 of the present disclosure determines a user's intention from a user's voice input and controls a target device in relation to the user's intention.

Referring to FIG. 3, the client device 110 receives a voice input from a user through a microphone. In the embodiment shown in FIG. 2, the client device 110 may be an artificial intelligence speaker, but is not limited thereto. The client device 110 may include a voice recognition module.

In an embodiment, the client device 110 may receive a voice input from the user without specifying the name of the target device to perform an operation of increasing the volume, such as "Raise the volume." In another embodiment, the client device 110 may receive a voice input including the name of the target device, such as "Play Bohemian Rhapsody on TV" from the user.

The client device 110 obtains a voice signal from a voice input received through a microphone, and transmits the obtained voice signal to the server 300.

The server 300 includes program code and application instructions including instructions on functions or operations of the voice assistant 200 for determining the target device 123 based on the received voice signal. ), an algorithm, a routine, a set of instructions, a natural language understanding engine, or an artificial intelligence model.

The server 300 converts the speech signal received from the client device 110 into text by performing Automatic Speech Recognition (ASR). In an embodiment, the server 300 may convert a speech signal into a computer-readable text using a predefined model such as an acoustic model (AM) or a language model (LM).

The server 300 may determine the first intent by analyzing the text using the first natural language understanding model (first NLU) 300a. The server 300 may determine the type of the target device related to the determined first intent.

The server 300 may determine a second natural language understanding model corresponding to the type of the target device from among the plurality of second natural language understanding models (second NLU) 300b in consideration of the determined type of the target device. The server 300 may determine the second intent by analyzing the text using a second natural language understanding model (second NLU) corresponding to the type of the target device among the plurality of second natural language understanding models 300b. In this case, the text input to the second natural language understanding model may be at least a part of the text converted from the speech signal. In an embodiment, the server 300 may determine the second intent and parameter from the text using the second natural language understanding model.

The server 300 receives information on a plurality of devices 120 from the IoT cloud server 400. The plurality of devices 120 may be devices previously registered with the IoT cloud server 400 in relation to user account information.

The server 300 may receive device information about a plurality of previously registered devices 120 from the IoT cloud server 400. In one embodiment, the server 300 may receive device information including at least one of capability information, location information, and status information of the plurality of devices 120 from the IoT cloud server 400. have. In an embodiment, the server 300 may receive status information including information on whether a plurality of devices 120 are powered on or off or operation information currently being performed from the IoT cloud server 400. .

The server 300 may determine a target device based on the determined type of the target device and device information received from the IoT cloud server 400. The server 300 may determine a target device by selecting a plurality of candidate devices corresponding to the determined type among the plurality of devices 120 and considering state information of the selected plurality of candidate devices. In the embodiment shown in FIG. 3, the server 300 may determine the third device 123 as a target device.

The server 300 may determine a second intent and a parameter from the text using a second natural language understanding model corresponding to the determined type of the target device. In this case, the text input to the second natural language understanding model may be at least a part of the text converted from the speech signal. The server 300 may determine an operation to be performed by the target device based on the second intent and parameter. In one embodiment, the server 300 transmits the identification information of the target device, the second intent and the parameter to the action plan management model, and details for performing an operation related to the second intent and the parameter from the action plan management model. It is possible to obtain motion information for the motion.

The server 300 transmits operation information to the IoT cloud server 400. The IoT cloud server 400 may convert operation information obtained from the server 300 into a control command that can be read by the target device. Here, the control command means a command that the target device can read and execute so that the target device can perform detailed operations included in the operation information.

The IoT cloud server 400 transmits a control command to the third device 123 determined as a target device.

Conventionally, in a network environment including a plurality of devices 120, when a user wants to receive a service through a device other than the client device 110 that performs an interaction such as conversation through voice input, There was a hassle of having to select another device and run another device directly.

The server 300 of the present disclosure automatically determines the target device by determining the intention included in the user's voice input received through the client device 110, thereby providing an automated system environment to the user and improving user convenience. I can make it. In addition, the server 300 according to the embodiment shown in FIG. 3 considers the state information of the device even when the user does not speak or speaks indefinitely about the device name or device type. It is decided that it can provide an intuitive and convenient user experience (User eXperience, UX). Specific operations performed by the server 300 will be described in detail with reference to FIGS. 4 and 5.

FIG. 4 is a flowchart illustrating a method of determining a target device to perform an operation according to a user's intention from a user's voice input and controlling the target device by the server 300 of the present disclosure.

In step S410, the server 300 receives a user's voice signal. In one embodiment, the server 300 may receive a voice signal from the client device 110. In an embodiment, the server 300 may obtain identification information (id information) of the client device 110 or identification information of a user from the client device 110. In an embodiment, upon receiving the identification information of the client device 110, the server 300 may search for user account information related to the identification information of the client device 110.

In step S420, the server 300 converts the received speech signal into text by performing ASR (Automatic Speech Recognition). In one embodiment, the server 300 performs ASR for converting a speech signal into a computer-readable text using a predefined model such as an acoustic model (AM) or a language model (LM). I can.

In step S430, the server 300 analyzes the converted text using a first natural language understanding model (NLU), and determines a target device based on the analysis result of the text. The first natural language understanding model may be a model trained to determine at least one target device related to a user's voice input by analyzing the text converted through ASR. According to an embodiment, the device dispatcher model including the first natural language understanding model determines at least one target device related to a user's voice input. For example, when the text converted from the user's voice input is "Play Bohemian Rhapsody~", the target device may be a TV.

In step S440, the server 300 acquires information on an operation to be performed by the target device by using a second natural language understanding model corresponding to the determined target device among the plurality of second natural language understanding models.

In an embodiment, the server 300 may select a second natural language understanding model corresponding to the determined target device from among the plurality of second natural language understanding models. The second natural language understanding model is a model specialized for a specific device, and may be an artificial intelligence model trained to determine an operation, which is related to a device determined by the first natural language understanding model and corresponds to a text. An operation may mean at least one action performed by the device by executing a specific function in the device. The operation may represent at least one action performed by the device by executing an application on the device.

In step S450, the server 300 transmits the obtained operation information to the target device. The server 300 may transmit the motion information to the target device using the motion information and the determined identification information (device id) of the target device. In one embodiment, the server 300 is a target device determined in step S430 among device information about a plurality of devices of a user that is previously registered in relation to the user's account information (eg, user ID) acquired in step S410. Check the identification information of. Device information about a plurality of devices may be obtained through the IoT cloud server 400.

In one embodiment, the server 300 may directly transmit motion information to the target device, in which case the motion information may include an operation command. In another embodiment, the server 300 may transmit operation information to the target device through the IoT cloud server 400. In this case, the IoT cloud server 400 may check a control command to be transmitted to the target device using the operation information received from the server 300. The IoT cloud server 400 may transmit the identified control command to the target device using the identification information of the target device.

5 is a flowchart illustrating a method of determining a target device to perform an operation according to a user's intention from a user's voice input and controlling the target device by the server 300 of the present disclosure.

In step S510, the server 300 receives a user's voice signal from the client device 110. In one embodiment, the client device 110 receives a user's voice input (eg, speech) through a microphone, obtains a voice signal from the user's voice input, and the server 300 transmits the voice signal to the client device. It can be received from (110). The client device 110 converts a sound received through a microphone into an acoustic signal and removes noise (eg, a non-speech component) from the acoustic signal to obtain a speech signal.

In one embodiment, when the client device 110 transmits the voice signal to the server 300, identification information of the client device 110 (eg, ID information of the client device) or account information of the user (eg, , User ID information) may be transmitted to the server 300. The server 300 may obtain identification information (id information) of the client device 110 or identification information of a user from the client device 110. In an embodiment, upon receiving the identification information of the client device 110, the server 300 may search for user account information related to the identification information of the client device 110.

In step S520, the server 300 performs ASR (Automatic Speech Recognition) to convert the received speech signal into text. In one embodiment, the server 300 performs ASR for converting a speech signal into a computer-readable text using a predefined model such as an acoustic model (AM) or a language model (LM). I can. When the server 300 receives a sound signal from which noise has not been removed from a client device, the server 300 may obtain a voice signal by removing noise from the received sound signal, and perform ASR on the voice signal.

In step S530, the server 300 analyzes the converted text using a first natural language understanding model (NLU), and determines a type of the target device based on the analysis result of the text. The first natural language understanding model may be a model trained to obtain a first intent corresponding to the text by analyzing the text converted through the ASR. Here, the first intent is information indicating the user's utterance intention included in the text, and may be used to determine at least one type of a plurality of target device types. For example, when the text converted from the user's voice input is "Play Bohemian Rhapsody~", the first intent may be "play content". In an embodiment, the first natural language understanding model may be a model trained to determine the type of at least one target device related to the user's voice input by analyzing the text converted through the ASR. In another embodiment, the device dispatcher model including the first natural language understanding model may determine the type of at least one target device related to the user's voice input based on an analysis result of the first natural language understanding model.

The server 300 may determine the user's first intent from the converted text by performing a syntactic analysis or a semantic analysis using the first natural language understanding model. In an embodiment, the server 300 parses the converted text in units of morphemes, words, or phrases using the first natural language understanding model. The server 300 may infer the meaning of the word extracted from the parsed text by using the parsed morpheme, word, or linguistic feature (eg, grammatical element) of the parsed morpheme, word, or phrase using the first natural language understanding model. The server 300 may determine a first intent corresponding to the meaning of the inferred word by comparing the meaning of the inferred word with predefined intents provided in the first natural language understanding model.

The server 300 may determine the type of the target device based on the first intent. In an embodiment, the server 300 may determine the type of the target device related to the first intent using the first natural language understanding model. The'type' refers to a category of devices classified according to a preset criterion. The type of device may be determined, for example, based on the function or use of the device. For example, the device includes an audio device (eg, a speaker) that outputs an acoustic signal, an image device (eg, a TV) that outputs both sound and video signals, an air conditioner that controls the air temperature (eg, an air conditioner), It may be classified into a type such as a cleaning device (eg, a robot cleaner), but is not limited thereto.

In yet another embodiment, the server 300 may determine a target device type related to the first intent recognized from the text, based on a matching model capable of determining the relationship between the first intent and the target device type. The relationship between the first intent and the target device type may be calculated as a predetermined numerical value. In an embodiment, the relationship between the first intent and the target device type may be calculated as a probability value.

The server 300 may determine a type of a target device related to the first intent from among the types of a plurality of target devices by applying the matching model to the first intent obtained from the first natural language understanding model. In one embodiment, the server 300 obtains a plurality of numerical values representing the degree of association between the first intent and a plurality of target device types by applying a matching model to the first intent, and the obtained plurality of numerical values The target device type having the maximum value among them may be determined as the final type. For example, if the first intent is related to the first device type and the second device type, respectively, the server 300 may provide a first numerical value indicating a degree of association between the first intent and the first device type, and A second numerical value indicating a degree of correlation between the first intent and the second device type may be obtained, and a first device type having a higher numerical value among the first numerical value and the second numerical value may be determined as the target device type.

For example, the first numerical value representing the degree of relation between the first intent and the "TV" of "content reproduction" calculated by applying the matching model is different target device types, for example "refrigerator" and "content reproduction" It may be greater than the second numerical value indicating the degree of relationship between the liver. In this case, the server 300 may determine the TV as a target device type related to the first intent of content playback. In another example, when the first intent is "temperature control", a numerical value representing the degree of relation between the first intent and the "air conditioner" may be higher than a numerical value representing the degree of relationship between the first intent and the "TV". I can. In this case, the server 300 may determine the air conditioner as a target device type related to the first intent of temperature control.

However, it is not limited to the above-described example, and the server 300 may determine a predetermined number of device types by arranging numerical values representing degrees of association between the first intent and the plurality of device types in high order. In an embodiment, the server 300 may determine a device type in which a numerical value indicating a degree of relevance is equal to or greater than a predetermined threshold as a device type related to the first intent. In this case, a plurality of device types may be determined as target device types.

The server 300 may learn a matching model between the first intent and the target device type using, for example, a rule-based system, but is not limited thereto. The artificial intelligence model used by the server 300 is, for example, a neural network-based system (e.g., a convolution neural network; CNN, a recurrent neural network; RNN), and SVM (Support Vector Machine), linear regression, logistic regression, Naive Bayes, random forest, decision tree, or k-nearest neighbor algorithm. It could be a combination of one or another AI model.

The server 300 determines a target device based on the type of the target device determined using the first natural language understanding model and information on a plurality of devices. The server 300 transmits the user's account information (eg, user ID) obtained from the client device 110 in step S510 to the IoT cloud server 400, and the user's account information to the IoT cloud server 400 In connection with, it is possible to request device information on a plurality of devices of a user who has been previously registered. The IoT cloud server 400 may be a server that stores device information about a plurality of devices of a user, which are previously registered for each user account. The IoT cloud server 400 is separate from the server 300 and may be connected to the server 300 through a network. The server 300 may receive device information on a plurality of devices registered in relation to the user's account information from the IoT cloud server 400. For example, the device information may include device identification information (device id information), function performance capability information (capability), location information, and status information.

In an embodiment, the server 300 may receive device information about a plurality of devices from the IoT cloud server 400 by using the user's account information previously stored in the server 300. The server 300 may receive device information from the IoT cloud server 400 even before a voice input is received, and may receive device information according to a preset period.

The function performance capability information may be information on a function of a device predefined to perform an operation. For example, when the device is an air conditioner, the function performance information of the air conditioner may indicate a function such as temperature up, temperature down, or air purification, and when the device is a speaker, volume up/down, music playback ) Can represent the same function. The function performance capability information may be previously stored in the IoT cloud server 400. However, the present invention is not limited thereto, and function performance capability information may be stored in the server 300.

Further, the location information is information indicating the location of the device, and may include, for example, a name of a place where the device is located and a location coordinate value indicating the location of the device. For example, the location information of the device may include a name indicating a specific place in a house such as a room or a living room, or may include a name of a place such as a house or an office.

The device state information may be information indicating a current state of the device, including at least one of power on/off information and information on an operation currently being executed.

In one embodiment, the server 300 transmits the information on the target device type determined in step S530 and the user's account information to the IoT cloud server 400, and among a plurality of devices previously registered in relation to the user's account information Only information about a device corresponding to the determined target device type may be obtained.

In an embodiment, the server 300 may store device information on a plurality of devices of a user that are pre-registered for each user account. In this case, the server 300 does not request device information from the IoT cloud server 400, and retrieves device information about a plurality of devices of a user that is previously registered in relation to the user's account information stored in the server 300. Can be used. In addition, by using information on the determined target device type, only information on a device corresponding to the determined target device type among a plurality of devices of a pre-registered user may be obtained.

The server 300 may select a candidate device corresponding to the determined target device type from among a plurality of devices previously registered in relation to the user's account information. There may be one selected candidate device, but there may be a plurality of selected candidate devices. When there is only one selected candidate device, the selected candidate device may be determined as the target device.

In an embodiment, when there are a plurality of selected candidate devices, the server 300 may determine the target device in consideration of state information of the selected plurality of candidate devices. For example, when the target device type related to the first intent of "playing content" is determined as "TV", the server 300 is a living room that is currently powered on among candidate devices such as living room TV, home TV, and children's room TV. TV can be determined as a target device.

In another embodiment, when there are a plurality of selected devices, the server 300 may determine a target device in consideration of installation locations of the selected plurality of candidate devices. For example, when the target device type related to the first intent of "playing content" is determined to be "TV", the server 300 may be close to the current location of the user among candidate devices such as living room TV, home room TV, and child room TV. The home TV installed at the location can be determined as the target device. In the above-described examples, information on whether the power of the living room TV, the home TV, and the children's room TV is turned on and off and installation location information may be obtained from the IoT cloud server 400.

The server 300 may determine a target device based on the determined type and device information received from the IoT cloud server 400, but is not limited thereto. If there are a plurality of candidate devices corresponding to the type determined by the server 300 and the target device cannot be determined even though the device information received from the IoT cloud server 400 is used, the user selects a specific target device through a query message. You can also ask for a decision.

Although it has been described that the type of one target device is determined in step S530, the present invention is not limited thereto. In step S530, the server 300 may determine a plurality of target device types. When a plurality of target device types are determined, the server 300 may determine a target device based on the plurality of target device types. For example, when the first device type and the second device type are determined as the target device type, the server 300 may select a plurality of devices corresponding to the first device type among a plurality of devices previously registered in relation to the user's account information. Then, the first candidate device and the second candidate device may be determined based on state information and installation location information of the selected plurality of devices. In addition, the server 300 selects a plurality of devices corresponding to the second device type from among a plurality of previously registered devices, and based on status information and installation location information of the selected plurality of devices, the third candidate device and the fourth candidate Device can be determined. The server 300 may determine any one of a first candidate device, a second candidate device, a third candidate device, and a fourth candidate device as the target device.

In step S540, the server 300 selects a second natural language understanding model corresponding to the determined type of the target device from among the plurality of second natural language understanding models, and uses the selected second natural language understanding model to perform an operation to be performed by the target device. Obtain information about The second natural language understanding model is a model specialized for a specific type of device, which is related to a device corresponding to the type determined by the first natural language understanding model and is an artificial intelligence model that has been trained to acquire a second intent and parameter corresponding to the text. I can. The second intent is information indicating the user's speech intention included in the text, and may be used to determine an operation to be performed by a specific type of device.

In an embodiment, the second natural language understanding model may be a model trained to determine an operation of a device related to a user's voice input by analyzing text. An operation may mean at least one action performed by the device by executing a specific function in the device. The operation may represent at least one action performed by the device by executing an application on the device.

In an embodiment, the server 300 may analyze the text using a second natural language understanding model corresponding to the determined type of the target device. In this case, the text input to the second natural language understanding model may be at least a part of the text converted from the speech signal. For example, when the device name is included in the converted text, the server 300 may remove the device name from the text and input it into the second natural language understanding model. That is, the server 300 may convert and input the converted text into text in a format that can be analyzed by the second natural language understanding model. The format that the second natural language understanding model can analyze refers to a sentence structure corresponding to the text used as training data of the second natural language understanding model.

The server 300 parses the text in units of morphemes, words, or phrases using a second natural language understanding model, grasps the meaning of the parsed morphemes, words, or phrases through grammatical and semantic analysis, and identifies the identified meanings. The second intent and parameter may be determined by matching to a predefined word. The server 300 may determine a second intent from text by using the second natural language understanding model, and may determine an intent specialized for a device type rather than the first intent as the second intent. For example, when the device type is determined to be "TV", when the second natural language understanding model is used, the text "Play Bohemian Rhapsody~" may be analyzed to determine'play movie content' as the second intent. . In another example, when the device type is determined to be “speaker”, and when the second natural language understanding model is used, “play a song” may be determined as the second intent by analyzing the text “Play Bohemian Rhapsody”.

The server 300 may include a plurality of second natural language understanding models, and may select at least one second natural language understanding model corresponding to the type of the target device based on the type of the target device determined in S530. When at least one second natural language understanding model is selected, the server 300 may determine a second intent and parameters by analyzing the text using the selected second natural language understanding model.

The server 300 may obtain operation information on at least one detailed operation related to the second intent and the parameter by using an action plan management model in which action plans related to the determined target device type are stored. The action plan management model may manage detailed operations of a device for each device type and information about a relationship between detailed operations. The server 300 may plan detailed operations to be performed by the device and an execution order of detailed operations based on the second intent and parameters by using the action plan management model.

The server 300 may obtain operation information on a series of detailed operations to be performed by the target device by providing the determined identification information (device id information) of the target device and the identification value of the target device type to the action plan management model. have. The operation information may be information related to detailed operations to be performed by the device and an execution order of the detailed operations. The operation information may include detailed operations to be performed by the device, a relationship between each detailed operation and other detailed operations, and information related to an execution order. The operation information may include, for example, functions to be executed by the target device in order to perform a specific operation, an execution order of functions, an input value necessary to execute the functions, and an output value output as a result of execution of the functions. , Is not limited thereto.

In step S550, the server 300 transmits the obtained operation information to the IoT cloud server 400. The server 300 may transmit operation information and identification information (device ID) of the determined target device to the IoT cloud server 400.

The IoT cloud server 400 may obtain a control command for controlling detailed operations of the target device based on the operation information and device identification information received from the server 300. In an embodiment, the IoT cloud server 400 may select a control command for controlling detailed operations of the target device in operation information from among control commands of the target device previously stored in the database DB. The IoT cloud server 400 may include a database in which control commands and operation information for a plurality of devices are stored. The IoT cloud server 400 may select a database using device identification information received from the server 300 and search for a control command using operation information from the selected database.

The control command is information that can be read by the device and may include a command for the device to execute a function and perform detailed operations according to the operation information.

The IoT cloud server 400 may transmit a control command to the target device.

Conventionally, in order for a device to perform an operation by receiving a user's voice in a network environment composed of a plurality of devices, all of the plurality of devices must have a function to receive the user's voice input and operate according to the voice input. As a result, there is a problem in that the resources of the device become excessive. In other words, when a new device is added to the network environment, all functions have to be newly developed to control the new device.

The server 300 according to an embodiment of the present disclosure converts the user's voice input received by the client device 110 into text, analyzes the text using the first natural language understanding model, and determines the type of the target device. And, by using the second natural language understanding model corresponding to the device type determined by the first natural language understanding model, operation information related to the analyzed text is obtained, reducing unnecessary resources in the device and securing the universality of the system architecture. It works.

6 is a flowchart illustrating a specific method of determining a type of a target device through text converted from a voice input by the server 300 of the present disclosure. 6 is a diagram illustrating a specific embodiment of step S530 of FIG. 5.

In step S610, the server 300 parses the text converted in step S520. In one embodiment, the server 300 may parse the text in units of morphemes, words, or phrases using the first natural language understanding model. For example, the server 300 may parse the text of "Play Bohemian Rhapsody on TV~" as'TV','Bohemian Rhapsody','Play', and'Give me'.

In step S620, the server 300 determines whether the name of the device is included in the parsed text. In one embodiment, the server 300 analyzes linguistic features (eg, grammatical elements) of parsed morphemes, words, or phrases using the first natural language understanding model, and analyzes the meaning of words extracted from the parsed text. I can infer. The server 300 may determine a device name corresponding to the meaning of the inferred word by comparing the meaning of the inferred word with predefined device names provided in the first natural language understanding model. For example, the server 300 may determine'TV' as the device name by parsing the text "Play Bohemian Rhapsody on TV" and comparing the extracted word or phrase with predefined device names. In this case, the server 300 may determine that the text includes the device name.

For another example, if the word or phrase corresponding to the device name cannot be extracted even if the word or phrase extracted by parsing the text of "Play Bohemian Rhapsody~" is compared with the predefined device names, the server ( 300) may determine that the device name is not included in the text.

If it is determined that the device name is included in the text parsed in step S620 (YES), the server 300 determines the type of the target device based on the name of the target device included in the text (step S630). For example, when the server 300 parses the text “Play Bohemian Rhapsody on TV~” and extracts “TV” as the device name, “TV” may be determined as the target device type.

In one embodiment, the parsed text may include the generic name and installation location of the device. In this case, the server 300 may parse the text and match the extracted word or phrase with a predefined word or phrase to obtain a common name of the device and a word regarding the installation location. For example, if the text is "Play Bohemian Rhapsody on a living room TV~", the server 300 parses the text, and then, from the parsed text, a word about the common name (TV) of the device and the installation location (living room) Can be extracted. The server 300 may determine the type of the target device based on the extracted device name and the installation location.

If it is determined that the device name is not included in the text parsed in step S620 (NO), the server 300 recognizes the first intent by analyzing the text using the first natural language understanding model (step S640). In one embodiment, the server 300 may determine a user's first intent from the text by performing syntactic analysis or semantic analysis using the first natural language understanding model. have. The server 300 infers the meaning of the word extracted from the parsed text by analyzing the linguistic features (eg, grammatical elements) of morphemes, words, or phrases extracted from the parsed text using the first natural language understanding model. can do. The server 300 may determine a first intent corresponding to the meaning of the inferred word by comparing the meaning of the inferred word with predefined intents provided in the first natural language understanding model. For example, the server 300 parses the text of "Play Bohemian Rhapsody~" into words or phrases such as'Bohemian Rhapsody','Turn','Give~', and pre-define the parsed word or phrase. A'content play' for playing content having an identification value (eg, a movie title) of a Bohemian Rhapsody compared with the intent may be determined as the first intent.

In step S650, the server 300 acquires characteristic information of a plurality of devices. In one embodiment, the server 300 provides characteristic information of a plurality of devices previously registered in the IoT cloud server 400 in relation to the user's account information (eg, user ID) from the IoT cloud server 400. Can receive. The server 300 may request the IoT cloud server 400 for characteristic information about a plurality of devices that are previously registered in relation to the user's account information. The server 300 may receive characteristic information on a plurality of devices registered in relation to the user's account information from the IoT cloud server 400. The characteristic information may be information used by the server to determine the device type. The characteristic information may include some of the device information. The characteristic information may include, for example, information on an identification value (device id) of each of a plurality of devices, and function performance capability information (capability).

The server 300 may store characteristic information for a plurality of devices that are previously registered in relation to the user's account information, and in this case, the server 300 may use the stored characteristic information.

In step S660, the server 300 determines a target device type based on the first intent and characteristic information. In an embodiment, the server 300 may determine a target device type related to the first intent recognized from the text, based on a matching model capable of determining the relationship between the first intent and the target device type. The matching model may be predefined based on the first intent and characteristic information of the target device. In the matching model, for example, at least one function may be matched with respect to an operation indicated by the first intent. The server 300 may determine at least one device type by identifying a device capable of performing a function matched with the first intent based on the device characteristic information. The function performance capability information of the device may mean information on a function that the device can execute. For example, the mobile phone's ability to perform functions may be SNS, map, phone, Internet, etc., the TV's ability to play content, and the air conditioner's ability to control air temperature.

For example, in the matching model, the first intent of'playing content' may be defined as being matched with'TV' or'speaker' having a function performance capability capable of playing content such as a movie or music. As another example, in the matching model, the first intent of'temperature control' may be defined as being matched with'air conditioner', which is a device having the ability to raise or lower the temperature of air.

In an embodiment, a matching rule or matching pattern between the first intent and the target device type may be predefined in the matching model.

In the matching model, the degree of matching, that is, the degree of correlation between the first intent and the target device type may be calculated as a probability value. The server 300 obtains a plurality of probability values indicating the degree of matching between the first intent and a plurality of target device types by applying the matching model to the first intent determined from the text, and determines the maximum value among the obtained plurality of probability values. The target device type to have may be determined as the final target device type.

For example, a first probability value indicating the degree of matching between the first intent of the “content playback” calculated by applying the matching model and the “TV” is between different target device types, for example, between “refrigerator” and “content playback” It may be greater than the second probability value indicating the degree of matching. In this case, the server 300 may determine the TV as a target device type related to the first intent of content playback. In another example, when the first intent is "temperature control", a probability value representing the degree of matching between the first intent and "air conditioner" may be higher than a probability value representing the degree of matching between the first intent and "TV". . In this case, the server 300 may determine the air conditioner as a target device type related to the first intent of temperature control.

The server 300 may learn a matching model between the first intent and the target device type using, for example, a rule-based system, but is not limited thereto. The artificial intelligence model used by the server 300 is, for example, a neural network-based system (e.g., a convolution neural network; CNN, a recurrent neural network; RNN), and SVM (Support Vector Machine), linear regression, logistic regression, Naive Bayes, random forest, decision tree, or k-nearest neighbor algorithm. It could be a combination of one or another AI model.

7 is a flowchart illustrating a method of determining a target device based on a target device type determined from text and information on a plurality of devices by the server 300 of the present disclosure.

In step S710, the server 300 requests device information by transmitting the user's account information to the IoT cloud server 400. The server 300 transmits the user's account information (eg, user ID) acquired from the client device 110 to the IoT cloud server 400, and transmits the user's account information to the IoT cloud server 400 Device information on a plurality of devices of a user who has been registered may be requested.

In an embodiment, the server 300 may transmit information on the determined target device type to the IoT cloud server 400 together with the user's account information. In this case, the server 300 may request from the IoT cloud server 400 only device information regarding a device corresponding to the determined target device type among a plurality of devices previously registered in relation to the user's account information.

In one embodiment, the server 300 may store information on a plurality of devices in the memory 306, and retrieve information on a plurality of devices stored in the memory 306 using user account information. I can.

In step S720, the server 300 receives device information of a plurality of devices previously registered in relation to the user's account information from the IoT cloud server 400. The IoT cloud server 400 stores a list of a plurality of devices previously registered for each user's account information received from the server 300, and function performance information and location information for each of the plurality of devices included in the list Device information including at least one of, and state information may be stored. The status information may include, for example, at least one of power on/off information of a plurality of devices previously registered in relation to the user's account information, and operation information currently being executed. The server 300 may receive at least one of function performance information, location information, and status information of each of a plurality of devices from the IoT cloud server 400.

In one embodiment, when the server 300 stores information on a plurality of devices in the memory 306, among information stored in the memory 306, information about a plurality of devices previously registered in relation to the user's account information Information can be obtained.

In step S730, the server 300 selects at least one device corresponding to the determined type from among a plurality of devices. In an embodiment, when only one type of target device related to the first intent is determined, the server 300 may select at least one device corresponding to the determined type. In another embodiment, when the server 300 determines a plurality of types as target device types related to the first intent, the server 300 may select a plurality of devices corresponding to each of the plurality of target device types. For example, when a plurality of types including'TV' and'speaker' are determined as the target device type, the server 300 may select a plurality of devices corresponding to the TV among the plurality of devices previously registered in relation to the user's account information. Devices such as a living room TV, a home TV, and a children's room TV, and an artificial intelligence speaker and a Bluetooth audio speaker corresponding to the speaker may be selected, respectively.

In step S740, the server 300 determines whether there are a plurality of target device candidates by considering device information of a plurality of selected devices. In one embodiment, when the target device type is one, the server 300 determines a target device candidate based on device information of a device corresponding to the target device type, and determines whether there are a plurality of the determined target device candidates. I can. In another embodiment, when there are multiple types of target devices, the server 300 determines a target device candidate based on status information or installation location information of each of a plurality of devices corresponding to the plurality of types, and the determined target device candidate It can be determined whether or not is a plurality of.

In step S740, if one target device candidate is determined (NO), the server 300 determines one target device as the final target device (step S750).

In step S740, if there are a plurality of target device candidates (YES), the process proceeds to step ⓐ, which will be described with reference to FIG. 8.

8 is a flowchart illustrating a method of determining a target device based on a user's response input when the server 300 of the present disclosure cannot determine the target device even if the information of a plurality of devices is considered. 8 is a diagram showing a specific embodiment of the case of ⓐ shown in step S740 of FIG. 7.

In step S810, the server 300 generates a query message for selecting a target device using a natural language generation model (NLG). The server 300 may generate a query message for selecting any one target device from among a plurality of device candidates corresponding to a specific type selected in step S730 of FIG. 7. For example, if the type of the plurality of devices selected in step S730 of FIG. 7 is a TV, a plurality of TVs may exist in the network as'living room TV','home room TV', and'children's room TV'. In step S810, the server 300 may perform a disambiguation step of determining a target device among a plurality of selected devices. In an embodiment, the server 300 may generate a query message that induces a user's response as to which device among a plurality of device candidates to be determined as a target device using the natural language generation model.

In one embodiment, the query message includes a list listing a plurality of device candidates corresponding to a specific type, and may be a message requesting the user to select a target device from among a plurality of device candidates included in the list. . For example, the server 300 is a list listing a plurality of TVs in a preset order, such as "1. Living room TV, 2. home room TV, 3. children's room TV on which TV to play the Bohemian Rhapsody movie?" You can create a query message that provides to the user.

In another embodiment, the query message may be a message for selecting a plurality of device candidates according to the installation location. For example, the server 300 presents a plurality of TVs based on the location, such as "Which TV will play the movie Bohemian Rhapsody on any of the TVs in the living room, home room, and children's room?" You can create a query message asking the user to select a device.

The server 300 may convert a text query message into an audio signal by using a text to speech (TTS) model in the query message generated by the natural language generation model.

In step S820, the server 300 requests the client device 110 to output a query message by transmitting a query message to the client device 110. The server 300 may transmit a query message to the client device 110. The server 300 may transmit a query message converted to an audio signal to the client device 110. When the client device 110 is a device including a display, a text query message may be transmitted.

In step S830, the server 300 receives a user's response input from the client device 110. The client device 110 may receive a response input from a user selecting a specific device from among a plurality of devices included in the query message. For example, the query message is "1. Living room TV, 2. home room TV, 3. children's room TV on which TV to play the movie Bohemian Rhapsody?", a user who lists a plurality of TVs in a preset order. In the case of a message provided to the client, the client device may receive a response input for selecting a target device through a specific ordinal number, such as "first TV". For example, if the query message is a message asking the user to select multiple TVs based on location, such as "Which TV in the living room, master room, or children's room will play the movie Bohemian Rhapsody?" The device may receive a response input from a user selecting a target device through an installation location, such as "living room". The server 300 may receive a user's response input from a client device.

The user's response input may be transmitted from the client device 110 in the form of a voice signal.

In step S840, the server 300 determines a target device based on the user's response input. The server 300 converts the received response input into text through an ASR process, and analyzes the converted text using a natural language understanding model, thereby extracting a word for a target device selected by the user. The server 300 may analyze the meaning of the extracted word by using the natural language understanding model and determine a target device based on the extracted word.

In one embodiment, a text for a user's response input is obtained through an ASR process, and the meaning of the text may be determined by matching the obtained text with a predefined text. The ASR model and the natural language understanding model used at this time may be different from the models described in FIGS. 2A and 2B. Of course, the server 300 may use at least one of the models described in FIGS. 2A and 2B.

9 is a flowchart illustrating a method of obtaining, by the server 300 of the present disclosure, operation information regarding an operation to be performed by a target device using a second natural language understanding model. 9 is a diagram illustrating a specific embodiment of step S540 of FIG. 5.

In step S910, the server 300 selects a second natural language understanding model (NLU) specialized for the target device type by using the type of the target device determined in step S530. The second natural language understanding model is a model trained to interpret text in relation to a specific type of device. The second natural language understanding model may be a model trained to determine second intents and parameters related to specific types of devices by interpreting the text. There may be a plurality of second natural language understanding models, and each second natural language understanding model corresponds to each device type. In an embodiment, the server 300 may select a second natural language understanding model corresponding to the target device type from among the plurality of second natural language understanding models based on the determined identification information of the target device type.

In step S920, the server 300 obtains a second intent and a parameter by analyzing the text using the second natural language understanding model. In one embodiment, the server 300 parses the text into words or phrases using the second natural language understanding model, infers the meaning of the parsed word or phrase through grammatical analysis and semantic analysis, and inferred The second intent and parameter may be obtained by matching the meaning with the predefined intent and parameter. In an embodiment, the server 300 may obtain a second intent and a parameter by using the result of parsing the text performed in step S610.

The second intent is information indicating the user's speech intention included in the text, and may be used to determine an operation to be performed by a specific type of device. The second intent may be determined by analyzing the text using the second natural language understanding model. The parameter means variable information for determining detailed operations of the target device related to the second intent. The parameter is information corresponding to the second intent, and a plurality of types of parameters may correspond to one second intent. The parameter may include variable information for determining operation information of the target device, and a numerical value indicating a probability that the text is related to the variable information. When a plurality of variable information indicating a parameter is obtained as a result of analyzing the text using the second natural language understanding model, variable information having a maximum numerical value corresponding to each variable information may be determined as a parameter. For example, in the text of "Play Bohemian Rhapsody on Living Room TV~", the second intent is'play content' representing an operation related to TV, and parameters are'TV','Living Room TV', ' It can be decided by the movie' or the title of the movie,'Bohemian Rhapsody'.

Since the second natural language understanding model determines a second intent related to a specific device type by analyzing text, the second intent may be more specific data than the first intent. The first intent and the second intent will be described in detail with reference to FIG. 10.

In step S930, the server 300 provides a second intent, a parameter, and identification information of a target device to an action plan management model storing an action plan for each device. In an embodiment, the server 300 may provide the second intent and parameter obtained in step S920 and the determined device ID of the target device to the action plan management model. The action plan management model may be a model that manages information related to detailed operations of a device in order to create an action plan. The action plan management model may manage detailed operations of a device for each device type and information about a relationship between detailed operations. In an embodiment, the action plan management model may store information corresponding to a parameter value input for execution of detailed operations or a result value output by execution of the operations for each device type.

In an embodiment, step S930 may be performed by the IoT cloud server 400. In this case, the server 300 may transmit the second intent, parameter, and identification information of the target device acquired in step S920 to the IoT cloud server 400. The IoT cloud server 400 may include an action plan management model.

In step S940, the server 300 acquires operation information on detailed operations to be performed by the target device from the action plan management model. The action plan management model identifies the target device from the identification information of the target device obtained from the server 300, and obtains pre-stored'information indicating the relationship between detailed operations and detailed operations' related to the target device from the memory. I can.

The action plan management model 210 identifies actions related to a second intent and parameter among'information representing the relationship between detailed actions and detailed actions', and plans the execution order of the identified detailed actions and detailed actions. , Operation information to be performed by the target device may be generated. The operation information may be information related to detailed operations to be performed by the device, a correlation relationship between detailed operations, and an execution order of the detailed operations. The operation information may include, for example, functions to be executed by the target device to perform specific operations, an execution order of functions, an input value required to execute the functions, and an output value output as a result of execution of the functions. , Is not limited thereto. The action plan management model provides motion information to the server 300, and the server 300 may obtain motion information from the action plan management model.

In an embodiment, step S940 may be performed in the IoT cloud server 400. The IoT cloud server 400 receives a second intent, a parameter, and identification information of a target device, and obtains operation information including detailed operations to be performed by the target device and an execution order of the detailed operations.

In step S950, the server 300 transmits the acquired operation information to the IoT cloud server 400 so that a control command can be provided from the IoT cloud server 400 to the target device. The server 300 may transmit operation information and identification information of the determined target device to the IoT cloud server 400.

The IoT cloud server 400 may obtain a control command for controlling detailed operations of the target device based on the operation information and device identification information received from the server 300. In an embodiment, the IoT cloud server 400 may select a control command for controlling detailed operations of the target device in operation information from among control commands of the target device previously stored in the database DB. In an embodiment, the IoT cloud server 400 may select a database using device identification information received from the server 300 and search for a control command using operation information from the selected database. The control command is information that can be read by the device, and may include a command for the device to perform an operation by executing a function.

The IoT cloud server 400 may transmit a control command to the target device.

FIG. 10 is a diagram illustrating a method of determining a first intent and a second intent from text by the server 300 of the present disclosure and obtaining operation information from the action plan management model 210.

Referring to FIG. 10, the server 300 may include a first natural language understanding model 300a, a second natural language understanding model 300b, and an action plan management model 210. Here, the "natural language understanding model" is a model for interpreting text, and may be implemented as hardware or software, or a combination of hardware and software. In one embodiment, the action plan management model 210 may be a separate component from the server 300 and may be included in the IoT cloud server 400.

In the embodiment shown in FIG. 10, when the user utters "Play Bohemian Rhapsody~", the server 300 receives a voice signal from the client device 110 and performs ASR to convert the voice signal into text. Can be converted. The first natural language understanding model 300a is a model that is trained to obtain a first intent corresponding to the text by analyzing text. The first natural language understanding model 300a parses text in units of morphemes, words, or phrases, and determines linguistic features (eg, grammatical elements) of the parsed morphemes, words, or phrases. The meaning of the word extracted from the parsed text can be inferred. The first natural language understanding model 300a may determine a first intent corresponding to the inferred meaning of the word by comparing the meaning of the inferred word with predefined intents. The first intent refers to information indicating the user's speech intention included in the text.

For example, when the text is "Play Bohemian Rhapsody", the first natural language understanding model 300a subdivides the text into morphemes and words such as "Bohemian", "Rhapsody", and "Play". The first natural language understanding model 300a performs a process of tagging each morpheme or word with grammatical elements, that is, parts of speech, sentence structure, etc. for each subdivided morpheme or words. The word "bohemian" may be tagged with an adjective or modifier, the word "rhapsody" may be tagged with a noun or object, and the word "flick" may be tagged with a verb or predicate. The first natural language understanding model 300a derives a relationship between each word by using each word, the position/order of the word, and tagging information of the word. That is, in the first natural language understanding model 300a, "bohemian" is an adjective or modifier that modifies "rapsody," and the word "bohemian rhapsody" which is a combination of "bohemian" and "rhapsody" is a verb or It judges that it is a noun or object that is the target of the predicate. The first natural language understanding model 300a infers the meaning of a verb or predicate in the text using the result of analyzing the text using grammatical elements. For example, in the first natural language understanding model 300a, in the text "Play Bohemian Rhapsody",'play it' has a meaning of performing an action of'play something'. Can be deduced.

In the present specification, the first natural language understanding model determines a first intent corresponding to the meaning inferred above. The first natural language understanding model 300a includes information on a plurality of first intents, and corresponds to the inferred meaning by comparing the information on the meaning inferred above with information on the plurality of first intents. Determine the first intent to do. For example, the first natural language understanding model 300a responds to'play something' by comparing it with information about'play something' and a plurality of first intents. The first intent to be played may be determined as “content play”.

The first natural language understanding model 300a is a model that analyzes text without considering a specific device type, and may be a simplified model than the second natural language understanding model 300b. The first natural language understanding model 300a is a model that analyzes a word corresponding to a predicate or verb in the input text. In addition, the first natural language understanding model 300a may be configured to update the entire existing model with a new model through learning or the like in order to interpret a new text.

The second natural language understanding model 300b is a model trained to interpret text in relation to a specific type of device. The second natural language understanding model 300b may be applied to specific types of devices, and may be a model trained to determine second intents and parameters related to an operation intended by the user by analyzing text. The second natural language understanding model 300b parses text into morphemes, words, or phrases, infers the meaning of the parsed morphemes, words, or phrases through grammatical and semantic analysis, and defines the inferred meaning as a predefined word. The second intent and parameter may be obtained by matching with. The second intent is information indicating a user's speech intention included in the text, and may be used to determine an operation to be performed by a specific type of device. The parameter means variable information for determining detailed operations of the target device related to the second intent. The parameter is information corresponding to the second intent, and a plurality of types of parameters may correspond to one second intent.

For example, when the text is "Play Bohemian Rhapsody", the second natural language understanding model 300b subdivides the text into morphemes and words such as "Bohemian", "Rhapsody", and "Play". The second natural language understanding model 300b performs a process of tagging grammatical elements, that is, parts of speech, sentence structures, etc., to each morpheme or word for each subdivided morpheme or words. The word "bohemian" may be tagged with an adjective or modifier, the word "rhapsody" may be tagged with a noun or object, and the word "flick" may be tagged with a verb or predicate. The second natural language understanding model 300b derives a relationship between each word using each word, the position/order of the word, and tagging information of the word. That is, in the second natural language understanding model 300b, "Bohemian" is an adjective or modifier that modifies "Rhapsody", and the word "Bohemian Rhapsody" combined with "Bohemian" and "Rhapsody" is a verb of "Turn on" or It judges that it is a noun or object that is the target of the predicate. The second natural language understanding model 300b has the meaning that the text "Play Bohemian Rhapsody" performs an action of'play something', and some are the object of'Bohemian Rhapsody'. We can infer that it is (object).

Here, the second natural language understanding model 300b may obtain a result analyzed by the first natural language understanding model 300a and additionally perform some steps to determine a second intent and a parameter. For example, the second natural language understanding model 300b may receive relationship information of words included in the text analyzed by the first natural language understanding model 300a. For example, in the second natural language understanding model 300b, bohemian" is an adjective or modifier that modifies "rapsody", and the word "bohemian rhapsody" which combines "bohemian" and "rhapsody" is a verb of "turn it on" Alternatively, information on a noun or object that is the target of the predicate may be received from the first natural language understanding model 300a. The second natural language understanding model 300b uses the received information to use the text "Play Bohemian Rhapsody". Is intended to perform an action called'play something', and it can be inferred that the target of the action is'Bohemian Rhapsody'.

Also, the second natural language understanding model 300b may analyze at least a part of the converted text input to the first natural language understanding model 300a. For example, when a device name is included in the converted text, the second natural language understanding model 300b may receive and analyze the remaining text after the device name is deleted from the converted text. Deleting the device name from the converted text may be performed by the first natural language understanding model 300a, or may be performed by the second natural language understanding model 300b.

In the present specification, the second natural language understanding model 300b determines a second intent corresponding to the meaning inferred above. The second natural language understanding model 300b includes information on a plurality of second intents, and compares information on the inferred meaning with information on a plurality of second intents, and corresponds to the inferred meaning. Determine the second intent.

In addition, the second natural language understanding model 300b includes information on a plurality of parameters that may be targets of the second intent. The second natural language understanding model 300b determines a parameter related to the second intent by comparing information on an object called'Bohemian Rhapsody' with information on a plurality of parameters.

For example, when the device type is "TV", the second natural language understanding model 300b parses the text of "Play Bohemian Rhapsody~" and the inferred meaning of'play something' and the second natural language Comparing the plurality of second intent information included in the understanding model 300b, based on a numerical value indicating the degree of association between the plurality of second intent information and'play something','playing movie content (movie content) content play)' may be determined as the second intent. In the second natural language understanding model 300b, a numerical value indicating the degree of relevance is determined in consideration of the device type, and when considering that the device type is TV, the first numerical value indicating the degree of relevance between'play me' and movie content playback is It may be larger than the second numerical value indicating the degree of correlation between'play it' and music content playback. Accordingly, when the device type is TV, the second natural language understanding model 300b may determine “play movie content” as a second intent related to “play it”.

In another example, when the device type is a speaker, when the same text is interpreted through the second natural language understanding model 300b, the first numerical value indicating the degree of relevance between'play it up' and movie content playback is'play it up' and music The second numerical value indicating the degree of relevance between content reproduction may be larger. In this case, the second natural language understanding model 300b may determine “music content play” as a second intent related to “play it”.

Since the second natural language understanding model 300b determines the second intent by analyzing the text by reflecting the attribute according to the specific device type, the second intent may be more specific information than the first intent. The second natural language understanding model 300b is a model that analyzes not only a predicate or verb, but also an object or a noun, a modifier, or a word corresponding to an adjective in the input text. Accordingly, the second natural language understanding model 300b may interpret the text by using a longer time for text interpretation than the first natural language understanding model 300a or by using more information stored in the memory. In addition, the second natural language understanding model 300b may be configured to add a model corresponding to a new device to an existing model in order to analyze text related to a new device.

The second natural language understanding model 300b may determine a parameter by analyzing the text in consideration of the device type. In the second natural language understanding model 300b, when a plurality of variable information representing a parameter is obtained as a result of analyzing the text, variable information having a maximum numerical value corresponding to each variable information may be determined as a parameter. For example, in the second natural language understanding model 300b, when the device type is TV, the second natural language understanding model 300b includes'Bohemian Rhapsody' extracted by parsing the text of "Play Bohemian Rhapsody". A parameter related to the second intent may be determined by comparing information on a plurality of parameters that are present. At this time, the second natural language understanding model 300b compares the information of a plurality of parameters related to the second intent among the information on the plurality of parameters and the'Bohemian Rhapsody' extracted from the text, and compares the parameter corresponding to the Bohemian Rhapsody. You can decide. When the second intent is determined to be'playing movie content', the second natural language understanding model 300b determines a parameter corresponding to the Bohemian Rhapsody among a plurality of parameters related to the movie content. In another example, when the second intent is determined to be'music content playback', the second natural language understanding model determines a parameter corresponding to a bohemian rhapsody among a plurality of parameters related to music content.

The server 300 may provide the determined second intent and parameter to the action plan management model 210 and obtain operation information related to the second intent and parameter from the action plan management model 210. In an embodiment, the server 300 may provide identification information on the type of the target device as well as the second intent and parameter to the action plan management model 210.

The action plan management model 210 may manage detailed operations of a device and information on a relationship between detailed operations for each device type. The action plan management model 210 may identify a target device based on the acquired identification information of the target device, and obtain information indicating pre-stored detailed operations related to the target device and a relationship between the detailed operations from the memory. . The action plan management model 210 selects detailed actions related to a second intent and a parameter among detailed actions of the target device and information on the relationship between the detailed actions, and plans the execution order of the selected detailed actions, thereby operating information Can be obtained.

In the embodiment shown in FIG. 10, the action plan management model 210 acquires identification information about the target device type, that is, ID information of the TV type, and performs a plurality of detailed operations (operation A) for reproducing the movie Bohemian Rhapsody. To operation C), and planning the execution order of a plurality of detailed operations (operations A to C), operation information can be obtained. Here, the operation information may include, for example, functions to be executed by the target device to perform a specific operation, an execution order of functions, an input value required to execute the functions, and an output value output as a result of execution of the functions. However, it is not limited thereto.

The action plan management model 210 includes, for example, an action A of running a movie player application, an action B of retrieving a movie Bohemian Rhapsody within a network or local storage, and an action C of playing the retrieved Bohemian Rhapsody into a database. By selecting from and planning the execution order of the selected operations A to C, operation information may be obtained.

11 is a flowchart illustrating operations of the client device 110, the server 300, and the IoT cloud server 400 of the present disclosure.

Referring to FIG. 11, the server 300 may include a first assistant model 200a and a second assistant model 200b. In FIG. 11, the second assistant model 200b is illustrated as a component included in the server 300, but is not limited thereto. The second assistant model 200b may be a separate component from the server 300. In one embodiment, the second assistant model 200b may be included in the IoT cloud server 400.

The first assistant model 200a and the second assistant model 200b are components included in the server 300 and may be implemented as hardware or software, or a combination of hardware and software. In one embodiment, the first assistant model 200a and the second assistant model 200b are program code including instructions, an application, an algorithm, and a routine. , A set of instructions, or an artificial intelligence learning model, or at least two sets.

In step S1110, the client device 110 obtains a voice signal from the user. When the user speaks, the client device 110 may receive the user's voice input through the microphone. The client device 110 converts a sound received through a microphone into an acoustic signal and removes noise (eg, a non-speech component) from the acoustic signal to obtain a speech signal.

In step S1112, the client device 110 transmits a voice signal to the server 300.

In step S1120, the first assistant model 200a converts the speech signal into text by performing ASR using the Automatic Speech Recognition (ASR) model 202. In one embodiment, the ASR model 202 of the first assistant model 200a reads a speech signal by a computer using a predefined model such as an acoustic model (AM) or a language model (LM). You can perform ASR that converts to text as much as possible. When the first assistant model 200a receives an acoustic signal from which noise is not removed from the client device 110, the ASR model 202 obtains a speech signal by removing noise from the received acoustic signal, and ASR can be performed on.

In step S1130, the device dispatcher model 310 of the first assistant model 200a analyzes the text using the first natural language understanding model 300a, and determines a first intent based on the analysis result. The first natural language understanding model 300a may be a model trained to obtain a first intent corresponding to the text by analyzing the text. Here, the first intent may be information indicating the user's speech intention included in the text. The device dispatcher model 310 performs a syntactic analysis or semantic analysis using the first natural language understanding model 300a, so that the user's first intent from the converted text Can be determined. In one embodiment, the device dispatcher model 310 parses the converted text in units of morphemes, words, or phrases using the first natural language understanding model 300a, and The meaning of the word extracted from the parsed text can be inferred by using the parsed morphemes, words, or linguistic features (eg, grammatical elements) of the phrase. The device dispatcher model 310 determines a first intent corresponding to the meaning of the inferred word by comparing the meaning of the inferred word with predefined intents provided in the first natural language understanding model 300a. I can.

In step S1140, the device dispatcher model 310 determines a type of the target device based on the first intent. In an embodiment, the device dispatcher model 310 may determine the type of the target device by using the first intent obtained by using the first natural language understanding model 300a.

In an embodiment, the device dispatcher model 310 may determine a target device type related to the first intent recognized from the text, based on a matching model capable of determining the relationship between the first intent and the target device type. have. The relationship between the first intent and the target device type may be calculated as a predetermined numerical value. In an embodiment, the relationship between the first intent and the target device type may be calculated as a probability value.

The device dispatcher model 310 applies a matching model to the first intent acquired through the first natural language understanding model 300a to determine the type of the target device related to the first intent among the types of the plurality of target devices. You can decide. In one embodiment, the device dispatcher model 310 obtains a plurality of numerical values indicating the degree of correlation between the first intent and the plurality of target device types by applying a matching model to the first intent, and The target device type having the maximum value among the numerical values of may be determined as the final type.

In step S1150, the device dispatcher model 310 determines a target device based on the determined type and device information of a plurality of previously registered devices. In an embodiment, the device dispatcher model 310 may receive device information of a plurality of devices previously registered in relation to the user's account information from the IoT cloud server 400. The device dispatcher model 310 includes, for example, at least one of function performance information (capability), location information, and status information of a plurality of devices previously registered in relation to the user's account information, and the IoT cloud server 400 Can be received from Here, the state information may include information on whether a plurality of devices are powered on or off and an operation currently being executed.

The device dispatcher model 310 selects a plurality of devices corresponding to the type determined in step S1140 from among a plurality of devices previously registered in relation to the user's account information, and a target device based on status information about the selected plurality of devices. Can be determined. In an embodiment, the device dispatcher model 310 may determine a device in a location close to the user's location as the target device in consideration of installation location information of the selected plurality of devices. In another embodiment, the device dispatcher model 310 may determine a target device based on status information regarding whether the selected plurality of devices are powered on or off or an operation currently being executed.

In step S1160, the first assistant model 200a provides the parsed text and target device information to the second assistant model 200b. In an embodiment, the device dispatcher model 310 may provide the determined identification information (eg, device ID) of the target device to the second assistant model 200b together with the parsed text.

In step S1170, the second assistant model 200b analyzes the parsed text using the second natural language understanding model corresponding to the type of the target device determined in step S1140 among the plurality of second natural language understanding models 300b. 2 Determine the intent and parameters. In one embodiment, the second assistant model 200b infers the meaning of the word or phrase extracted from the parsed text using the second natural language understanding model, and matches the inferred meaning to predefined intents and parameters to provide a 2 Can determine intents and parameters. The second intent is information indicating the user's speech intention included in the text, and may be used to determine an operation to be performed by a specific type of device. The parameter means variable information for determining detailed operations of the target device related to the second intent. The parameter is information corresponding to the second intent, and a plurality of types of parameters may correspond to one second intent.

In step S1180, the action plan management model 210 of the second assistant model 200b plans operation information to be performed by the target device based on the second intent and parameters. The action plan management model 210 may identify the target device from the identification information of the target device obtained from the device dispatcher model 310 and analyze actions to be performed by the target device based on the second intent and parameters. . The action plan management model 210 may select detailed operations related to analyzed operations among previously stored device-specific operations, and may plan an execution order of the selected detailed operations. The action plan management model 210 may acquire operation information regarding a detailed operation to be performed by the target device by using the planning result. Here, the operation information may be information related to detailed operations to be performed by the device, a correlation relationship between the detailed operations, and an execution order of the detailed operations. The operation information may include, for example, functions to be executed by the target device in order to perform detailed operations, an execution order of functions, an input value required to execute the functions, and an output value output as a result of execution of the functions. , Is not limited thereto.

In step S1190, the second assistant model 200b provides the motion information acquired through the action plan management model 210 to the first assistant model 200a.

12 is a flowchart illustrating operations of the client device 110, the server 300, and the IoT cloud server 400 of the present disclosure. 12 is a diagram illustrating a specific embodiment of step S1150 of FIG. 11. In FIG. 12, for convenience of description, it is illustrated that the server 300 includes only the first assistant model 200a, but is not limited thereto. The server 300 may further include a second assistant model 200b as shown in FIG. 11. In FIG. 12, the first assistant model 200a may include an ASR model 202, an NLG model 204, a first natural language understanding model 300a, and a device dispatcher model 310.

In step S1210, the server 300 requests device information about a plurality of devices from the IoT cloud server 400. The server 300 may transmit, to the IoT cloud server 400, a signal for requesting device information about a plurality of devices previously registered in the IoT cloud server 400 in connection with the user's account information. The IoT cloud server 400 may be a server that stores device information about a plurality of devices of a user, which are previously registered for each user account. The IoT cloud server 400 is separate from the server 300 and may be connected to the server 300 through a network. In one embodiment, the server 300 transmits the user's account information (eg, user ID) acquired from the client device 110 to the IoT cloud server 400, and the user's account information to the IoT cloud server 400 In connection with, it is possible to request device information on a plurality of devices of a user who has been previously registered.

In an embodiment, the server 300 may transmit type information on the determined target device type to the IoT cloud server 400. In this case, the server 300 may request only information on a device corresponding to a determined type among a plurality of devices previously registered in the user's account information from the IoT cloud server 400.

In an embodiment, the server 300 may store device information on a plurality of devices of a user that are pre-registered for each user account. In this case, the server 300 does not request device information from the IoT cloud server 400, and retrieves device information about a plurality of devices of a user that is previously registered in relation to the user's account information stored in the server 300. Can be used. In addition, by using information on the determined target device type, only information on a device corresponding to the determined target device type among a plurality of devices of a pre-registered user may be obtained.

In step S1220, the IoT cloud server 400 transmits at least one of function performance information (capability), location information, and status information of a plurality of devices previously registered in relation to the user's account information to the server 300 do.

In one embodiment, when the IoT cloud server 400 obtains type information on the target device type from the server 300, the IoT cloud server 400 is among a plurality of devices previously registered in relation to the user's account information. Only information about a device corresponding to a specific type based on the type information may be transmitted to the server 300.

In step S1230, the server 300 determines whether there are a plurality of target device candidates based on the type determined using the device dispatcher model 310 of the first assistant model 200a and the obtained device information. In an embodiment, when the target device type is one, the device dispatcher model 310 determines a target device candidate based on state information or installation location information of a device corresponding to the target device type, and the determined target device candidate It can be determined whether or not is a plurality of.

In another embodiment, when there are multiple types of target devices, the device dispatcher model 310 determines a target device candidate based on state information or installation location information of each of a plurality of devices corresponding to the plurality of types, and the determined It may be determined whether there are a plurality of target device candidates.

In step S1230, when it is determined that the target device candidate is one (NO), the server 300 determines one target device as the final target device using the device dispatcher model 310 of the first assistant model 200a. (Step S1240).

In step S1230, if a plurality of target device candidates is determined (YES), the server 300 uses the NLG model 204 of the first assistant model 200a to determine a target device among the plurality of candidate devices. A query message is generated (step S1250). The server 300 may generate a query message for selecting any one target device from among a plurality of devices corresponding to a specific type by using the NLG model 204. In step S1250, the server 300 may perform a disambiguation step of determining a target device among a plurality of candidate devices. In one embodiment, the server 300 may generate a query message that induces a user's response as to which device among a plurality of devices to determine as the target device using the NLG model 204.

In an embodiment, the query message includes a list listing a plurality of devices corresponding to a specific type, and may be a message requesting the user to select a target device from among a plurality of devices included in the list. In another embodiment, the query message may be a message requesting to select a plurality of devices according to installation locations.

In step S1260, the server 300 transmits the generated query message to the client device 110. In an embodiment, the first assistant model 200a of the server 300 may convert a query message composed of text into an audio signal through text-to-speech (TTS). The server 300 may transmit a query message converted into an audio signal to the client device 110. However, the present invention is not limited thereto, and when the client device 110 is a device including a display unit, the server 300 may transmit a query message to the client device 110 in the form of text data.

In step S1270, the client device 110 receives a query message from the server 300 and outputs the received query message. In an embodiment, the client device 110 may output a query message converted into an audio signal through a speaker. In another embodiment, when the client device 110 is a device including a display unit, the client device 110 may display an inquiry message on the display unit.

In step S1272, the client device 110 receives a user's response input. In an embodiment, the client device 110 may receive a response input from a user selecting any one of a plurality of candidate devices. The name of the device selected for the user's response input may be, but is not limited thereto. In one embodiment, the user's response input may be an ordinal number (eg, a second device) in the order listed in a list including a plurality of candidate devices.

In step S1274, the client device 110 transmits the response input received from the user to the server 300. The user's response input may be transmitted from the client device 110 in the form of a voice signal.

In step S1280, the server 300 determines a target device based on the response input using the first assistant model 200a. In one embodiment, the server 300 converts the response input received using the ASR model 202 into text through an ASR process, and analyzes the converted text using the first natural language understanding model 300a, Words related to the target device selected by the user can be extracted. The server 300 may determine the meaning of the extracted word using the first natural language understanding model 300a and determine a target device based on the extracted word. In one embodiment, an ASR model 202 and a first natural language understanding model 300a for determining the meaning of the text by converting the user's response input into text through an ASR process and matching the converted text with a predefined text May be a model different from the model described in FIGS. 2A and 2B. However, the present invention is not limited thereto, and the server 300 may use at least one of the models described in FIGS. 2A and 2B.

In one embodiment, the device dispatcher model 310 of the server 300 selects a target device that is matched in a specific order based on a response input from a user who selects a specific order from a list of candidate devices in the query message. You can decide.

13 is a flowchart illustrating operations of the client device 110, the target device 120, the server 300, and the IoT cloud server 400 of the present disclosure. FIG. 13 is a diagram specifically showing steps performed by the client device 110, the target device 120, the server 300, and the IoT cloud server 400 after step S1190 of FIG. 11.

In step S1310, the second assistant model 200b provides motion information to the first assistant model 200a. Step S1310 is the same step as step S1190 of FIG. 11.

In step S1320, the first assistant model 200a transmits operation execution request information to the IoT cloud server 400. In an embodiment, the first assistant model 200a may transmit operation information and target device identification information (eg, device ID) to the IoT cloud server 400. In an embodiment, the first assistant model 200a may transmit conversation information to the IoT cloud server 400. Here, the conversation information is a text obtained by converting the user's voice input received through the client device 110 by the ASR model 202 of the server 300, and the natural language by using the NLG models 204 and 206. It may include at least one of a query message generated using the generation model and text converted from a user's response input.

In step S1330, the IoT cloud server 400 acquires, generates, or determines a control command based on identification information and operation information of the target device received from the first assistant model 200a. In an embodiment, the IoT cloud server 400 may generate a control command for causing the target device to perform detailed operations in the operation information based on the operation information. The generated control command is transmitted to the target device, and the target device can sequentially execute detailed operations in the operation information according to the execution order by reading the generated control command. However, the present invention is not limited thereto, and the IoT cloud server 400 may store operation information and a control command corresponding to identification information of a target device.

In step S1340, the IoT cloud server 400 transmits the control command acquired, generated, or determined in step S1330 to the target device 120 by using the identification information of the target information.

In step S1350, the target device 120 executes operations corresponding to the control command according to the received control command.

In step S1352, when the execution of the operation is successfully completed or the execution of the operation fails, the target device 120 transmits the execution result of the operation to the IoT cloud server 400. When the execution of the operation fails, the target device 120 transmits the execution result of the operation to the IoT cloud server 400 together with the cause of the failure of the operation.

In step S1354, the IoT cloud server 400 transmits the operation execution result to the first assistant model 200a.

In step S1356, the first assistant model 200a transmits the operation execution result to the second assistant model 200b. In an embodiment, the first assistant model 200a may transmit information about the type of the target device 120 together with the operation execution result.

In step S1360, the second assistant model 200b uses the NLG model 206 to generate, obtain, or determine a response message indicating the result of the operation execution. In an embodiment, the second assistant model 200b may generate, obtain, or determine a response message corresponding to an operation execution result based on the received information on the target device type. The response message may be a message indicating an execution result indicating that the operation has been executed by the target device 120.

For example, when the target device 120 is a TV and the TV executes an operation of playing a movie, the second assistant model 200b is based on the execution result of the movie playing operation received from the server 300, You can generate a response message stating "The movie was played on the TV."

In another example, when the target device 120 is a speaker and the speaker fails to perform the operation of increasing the volume, the second assistant model 200b is the result of executing the operation of increasing the volume received from the server 300 and Based on the cause of the failure, you can generate a response message stating "The volume increase has failed because the volume is in the highest state." In this case, the execution result received from the server 300 may be "fail", and the cause of the failure received together with the execution result may be "the current volume is in the highest state".

In step S1360, it is described that the second assistant model 200b generates, acquires, or determines a response message according to the result of the operation execution using the NLG model 206, but is not limited thereto. In an embodiment, the response message may be pre-stored together with an action plan for each device type in the action plan management model 210. The action plan management model 210 may identify a response message corresponding to a result of the received operation execution among the previously stored response messages, and obtain the identified response message.

In step S1362, the second assistant model 200b transmits a response message to the first assistant model 200a.

In step S1370, the first assistant model 200a generates notification content for notifying the result of the operation execution based on the response message. In an embodiment, the first assistant model 200a may determine at least one of text, audio, image, and video as the format of the notification content, and generate or obtain the notification content according to the determined format.

The first assistant model 200a may convert a text-formatted response message into an audio signal by performing a text-to-speech (TTS) function that converts text into speech. In an embodiment, the server 300 may generate an image based on a layout corresponding to a layout key received from the target device 120.

In step S1372, the first assistant model 200a of the server 300 transmits the notification content to the client device 110. In an embodiment, the server 300 may transmit notification content to the target device 120. The server 300 may determine a device to provide notification content according to a predefined rule. The predefined rule includes information on the format of the notification content corresponding to each operation. In addition, it may additionally include information on a device to output notification content.

For example, the movie playback operation is defined as outputting notification content in an audio format, and is defined as being provided by the client device 110 that has transmitted the user's voice input. If the notification content for the movie playback operation is output in an audio format and is defined as being provided by the client device, the server 300 may convert the notification content into an audio format and transmit it to the client device.

In step S1380, the client device 110 outputs notification content. When the notification content is in an audio format, the client device 110 may output the notification content through a speaker. When the notification content is an image, the client device 110 may display the notification content on the display unit.

14 is a block diagram illustrating a client device 110, a server 300, and an IoT cloud server 400 of the present disclosure.

The client device 110 may be configured to include at least a microphone 112, a processor 114, a memory 116, and a communication interface 118. The client device 110 may receive a voice input (eg, speech) from a user through the microphone 112 and obtain a voice signal from the user's voice input. The client device 110 may control the communication interface 118 through the processor 114 and transmit a voice signal to the server 300 through the communication interface 118. In one embodiment, the processor 114 of the client device 110 converts the sound received through the microphone 112 into an acoustic signal, and removes noise (for example, a non-speech component) from the acoustic signal. The signal can be acquired.

The memory 116 may pre-store identification information (eg, ID information of a client device) or user account information (eg, user ID information) of the client device 110. In one embodiment, when the client device 110 transmits a voice signal to the server 300 through the communication interface 118 under the control of the processor 114, the client device 110 stored in the memory 116 Identification information (eg, ID information of a client device) or user account information (eg, ID information of a user) may be transmitted to the server 300.

Server 300 may include a communication interface 302, a processor 304, and a memory 306.

The communication interface 302 may perform data communication with the client device 110 and the IoT cloud server 400. The communication interface 302 is, for example, wired LAN, wireless LAN, Wi-Fi, Bluetooth, zigbee, Wi-Fi Direct (WFD), infrared communication (IrDA, infrared Data Association), BLE (Bluetooth Low Energy), NFC (Near Field Communication), Wibro (Wireless Broadband Internet, Wibro), WiMAX (World Interoperability for Microwave Access, WiMAX), SWAP (Shared Wireless Access Protocol), Wiigg Data communication with the client device 110 or the IoT cloud server 400 may be performed by using at least one of a data communication method including (Wireless Gigabit Allicance, WiGig) and RF communication.

The processor 304 may execute one or more instructions of a program stored in the memory 306. The processor 304 may be composed of hardware components that perform arithmetic, logic and input/output operations and signal processing. The processor 304 is, for example, a central processing unit (Central Processing Unit), a microprocessor (microprocessor), a graphics processor (Graphic Processing Unit), ASICs (Application Specific Integrated Circuits), DSPs (Digital Signal Processors), DSPDs (Digital Signal Processing Devices), Programmable Logic Devices (PLDs), and Field Programmable Gate Arrays (FPGAs), but are not limited thereto.

A program including instructions for determining a target device to perform an operation may be stored in the memory 306. The memory 306 may store instructions and program code that the processor 304 can read. In the following embodiment, the processor 304 may be implemented by executing instructions or codes of a program stored in memory.

In addition, data corresponding to the first assistant model 200a and the second assistant model 200b may be stored in the memory 306. The data corresponding to the first assistant model 200a includes at least data for the ASR model 202, data for the NLG model 204, data for the first natural language understanding model 300a, and the device dispatcher model 310. ) Can be included. The data corresponding to the second assistant model 200b includes data for at least a plurality of second natural language understanding models 300b, data for the NLG model 206, and data for the action plan management model 310. do.

The memory 306 is, for example, a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (eg, SD or XD memory). Etc.), RAM (RAM, Random Access Memory) SRAM (Static Random Access Memory), ROM (ROM, Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), Magnetic It may include at least one type of storage medium among memory, magnetic disk, and optical disk.

The processor 304 is, for example, a central processing unit (Central Processing Unit), a microprocessor (microprocessor), a graphics processor (Graphic Processing Unit), ASICs (Application Specific Integrated Circuits), DSPs (Digital Signal Processors), DSPDs (Digital Signal Processing Devices), Programmable Logic Devices (PLDs), and Field Programmable Gate Arrays (FPGAs), but are not limited thereto.

The processor 304 can control the communication interface 302 to receive a voice signal from the client device 110. The processor 304 performs ASR (Automatic Speech Recognition) using the data for the ASR model 202 stored in the memory 306, and converts the speech signal received from the client device 110 into text. I can.

The processor 304 analyzes the converted text by using data on the first natural language understanding (NLU) 300a stored in the memory 306, and analyzes at least one target based on the analysis result of the text. The device type can be determined. In an embodiment, the processor 304 parses the text in units of morphemes, words, or phrases by using data on the first natural language understanding model 300a stored in the memory 306 ( parse), and the meaning of the word extracted from the parsed text can be inferred by using the linguistic features (eg, grammatical elements) of the morpheme, word, or phrase. The processor 304 may determine a first intent corresponding to the meaning of the inferred word by comparing the meaning of the inferred word with predefined intents provided in the first natural language understanding model 300a.

The processor 304 may determine a target device type related to the first intent recognized from the text, based on a matching model capable of determining the relationship between the first intent and the target device type. In one embodiment, the matching model is included in data on the device dispatcher model 310 stored in the memory 306, and may be obtained through learning through a rule-based system, but is not limited thereto. Does not. The device dispatcher model 310 may be configured separately from the first natural language understanding model 300a, but is not limited thereto. The device dispatcher model 310 may be configured to include a first natural language understanding model 300a.

In an embodiment, the processor 304 may control the communication interface 302 to receive information on a plurality of devices that are previously registered in the IoT cloud server in relation to the user's account information. The user's account information is information stored in the memory 116 of the client device 110, and the server 300 communicates between the communication interface 302 and the communication interface 118 of the client device 110. You can receive account information.

The IoT cloud server 400 may be a server that stores device information about a plurality of devices of a user, which are previously registered for each user account. The IoT cloud server 400 is separate from the server 300 and may be connected to the server 300 through a network. The IoT cloud server 400 may include at least a communication interface 402, a processor 404, and a memory 406. The memory 406 of the IoT cloud server 400 may store identification information, function performance capability information, location information, status information, device name information, and device-specific control commands for a plurality of devices previously registered for each user account. . The IoT cloud server 400 is connected to the server 300 or the target device 120 through a network through the communication interface 402 and may receive or transmit data. The IoT cloud server 400 may transmit data stored in the memory 406 to the server 300 or the target device 120 through the communication interface 402 under the control of the processor 404. In addition, the IoT cloud server 400 may receive data from the server 300 or the target device 120 through the communication interface 402 under the control of the processor 404.

The processor 304 controls the communication interface 302, and through the communication interface 302, identification information (eg, device ID) of a plurality of devices previously registered in relation to the user's account information, function performance capability information, Device information including at least one of location information and status information may be received from the IoT cloud server 400. Here, the state information may be information indicating a current state of the device, such as at least one of power on/off information of a plurality of devices and information about an operation currently being executed.

The processor 304 uses the data on the first assistant model 200a stored in the memory 306 to determine the target device based on the determined target device type and device information of a plurality of devices previously registered in the user's account. I can.

In one embodiment, the processor 304 uses the data about the first natural language understanding model 300a stored in the memory 306, from the text to a word or phrase related to the type of device and the installation location of the device. Is extracted, and the target device can be determined based on the normal name and installation location of the extracted device.

In one embodiment, the processor 304 may extract the name of the personalized device pre-registered by the user from the text. The processor 304 controls the communication interface 302 to store the extracted device names as the names of a plurality of devices and device identification information corresponding to the names of each device. 410 (refer to FIG. 15), the extracted device name may be transmitted. The processor 304 may control the communication interface 302 to receive device identification information corresponding to the device name transmitted from an external server. Here, the device name server 410 may be some components included in the IoT cloud server 400, but is not limited thereto. The device name server 410 may be configured as a server separate from the IoT cloud server 400. The names of the plurality of devices stored in the device name server 410 may include at least one of a name directly registered by the user and a synonym of the registered name, a similar word, and a slang word.

The processor 304 may determine a target device based on the received device identification information by using the data for the first assistant model 200a stored in the memory 302. In one embodiment, when a plurality of devices corresponding to the determined type exist, the processor 304 uses the NLG model 204 stored in the memory 302 to make a query for selecting a target device from among a plurality of candidate devices. You can create a message. The processor 304 may control the client device 110 so that the client device 110 outputs the generated query message.

The processor 304 uses a second natural language understanding model corresponding to the determined type of the target device among the plurality of second natural language understanding models 300b included in the second assistant model 200b stored in the memory 302, Acquires operation information on detailed operations to be performed by the device. In one embodiment, the processor 304 parses the text into morphemes, words, or phrases using the second natural language understanding model, and infers the meaning of the parsed morphemes, words, or phrases through grammatical and semantic analysis. , By matching the inferred meaning to a predefined word, a second intent and a parameter may be obtained. The processor 304 may determine a second intent from the text using the second natural language understanding model, and may determine an intent specialized for a device type rather than the first intent as the second intent.

The processor 304 may obtain operation information on at least one operation related to the second intent and parameter by using the action plan management model 210 of the second assistant model 200b stored in the memory 302. . The operation information may be information related to detailed operations to be performed by the device and an execution order of the detailed operations. The action plan management model 210 may manage detailed operations of a device for each device type and information about a relationship between detailed operations. The processor 304 may plan detailed operations to be performed by the device and an execution order of detailed operations based on the second intent and parameters by using the action plan management model 210.

The communication interface 302 transmits, to the IoT cloud server 400, identification information of the target device determined according to the control of the processor 304, detailed operation of the target device, and information on the execution order of the detailed operation. The IoT cloud server 400 uses the identification information of the target device received through the communication interface 402, the detailed operation of the target device, and the information on the execution order of the detailed operation, according to the control of the processor 404, the memory ( Among the control commands for each device stored in 402), corresponding control commands may be determined.

The communication interface 402 of the IoT cloud server 400 transmits control commands determined through the communication interface 402 to the target device under the control of the processor 404. In one embodiment, the processor 404 determines a device to transmit the determined control command using the identification information of the target device, and transmits the control commands to the determined device through the communication interface 402. In an embodiment, the processor 404 may receive a result of executing an operation according to a control command in the target device through the communication interface 402.

15 to 19 are diagrams illustrating operations of a program module executed by the server of the present disclosure.

FIG. 15 is a diagram illustrating an architecture of a program executed by the server 300 according to an embodiment of the present disclosure. The client device 110 and IoT cloud server 400 shown in FIG. 15 are the same components as the client device 110 and IoT cloud server 400 shown in FIG. 14. In FIG. 15, for convenience, some of the components of the first assistant model 200a are omitted, and illustration of components included in the second assistant model 200b is omitted. For an example of the components included in each of the first assistant model 200a and the second assistant model 200b, refer to FIG. 14.

Referring to FIG. 15, the server 300 may include a second assistant model 200b and a device dispatcher model 310. The device dispatcher model 310 includes a flow controller 320, a device type classifier 330, a device disambiguation unit 340, and an intelligent device resolver. ) 350 and a response message execution control unit (Reponse Execute Manager) 360.

IDS Flow Controller 320, Device Type Classifier 330, Conversational Device Disambiguation 340, Intelligent Device Resolver 350, and Response Message The Response Execute Manager 360 may be a software module composed of instructions or program codes of a program executed by the server 300. In FIG. 15, the software modules are illustrated as being separate configurations, but this is for convenience of description and is not limited as illustrated.

Although not shown in FIG. 15, the device dispatcher model 310 may further include a session manager. The session manager may store information on a session between the server 300 and a plurality of devices previously registered in the IoT cloud server 400 in relation to the user's account information. The session manager may also control transmission and reception of data and commands between the client device 110 and the device dispatcher model 310, and the second assistant model 200b and the device dispatcher model 310. In an embodiment, the session manager may store a policy regarding whether to maintain a conversation between a plurality of devices in an interaction with a user or to receive a user's voice input only through the client device 110. When receiving a voice input that does not specify the name or type of a specific device from the user, the session manager selects the device that performed the operation before the voice input is received, and relates to the target device determined in the second assistant model 200b. You can search operation information.

The flow control unit 320 may receive a voice signal from the client device 110. The flow control unit 320 may be a module that controls a data flow in the device dispatcher model 310. The flow control unit 320 is connected to the device type classification unit 330 and the device disambiguation unit 340, and determines the type of the target device from the voice signal between the device type classification unit 330 and the device disambiguation unit 340. You can control the transmission and reception of necessary data flows.

The device type classifier 330 may determine a target device type related to the voice signal. The device type classification unit 330 may convert a speech signal into a computer-readable text by performing ASR (Automatic Speech Recognition). The device type classifier 330 may analyze text using the first natural language understanding model and obtain a first intent that the user intends to perform in the target device based on the analysis result. In one embodiment, the device type classifier 330 learns a matching rule or a matching pattern between types of target devices related to the first intent using an artificial intelligence model, and the learned matching The type of target device according to the intent may be determined based on a rule or a matching pattern.

The device type classifier 330 may learn a matching rule or a matching pattern between an intent and a target device type using, for example, a rule-based system, but is not limited thereto. The artificial intelligence model used by the device type classification unit 330 is, for example, a neural network-based system (e.g., a convolution neural network; CNN, a recurrent neural network; RNN), SVM (Support Vector Machine), linear regression (linear regression), logistic regression (logistic regression), Naive Bayes classification (Naive Bayes), random forest (random forest), may be a decision tree, or a k-nearest neighbor algorithm. Alternatively, it may be a combination of the above or other artificial intelligence models.

In an embodiment, the device type classifier 330 may recognize a device name from text and determine a device type based on the recognized device name. The device type classification unit 330 parses the text into words or phrases using the first natural language understanding model, and associates the parsed word or phrase with a predefined device name, and the device name included in the text Can be obtained. For example, when receiving a voice signal "Play Bohemian Rhapsody on TV" from the flow control unit 320, the device type classification unit 330 converts the voice signal into text, parses the converted text, A device name of'TV' can be obtained.

In one embodiment, the device type classification unit 330 transmits a query to the device name server 410 and personalizes it from the device name server 410 when the device name obtained from the text is a personalized alias. Device identification information corresponding to the alias may be received. The device name server 410 may be an external server that stores alias information related to names of a plurality of previously registered devices. The device name server 410 may be included in the IoT cloud server 400, but is not limited thereto. The device name server 410 may be a server separate from the IoT cloud server 400. The device name server 410 may store, for example, a personalized vocabulary of at least one of synonyms, similar words, and slang words of names of a plurality of devices registered by the user.

When a user registers a device in account information, the device name server 410 may register and store a personalized word or vocabulary, and the personalized word or vocabulary may be added or deleted. For example, when the user stores the living room TV as'square' in the device name server 410, the device type classification unit 330 queries the device name server 410 for'square'. And receive identification information (id information) of the “living room TV” corresponding to the “square” from the device name server 410. In an embodiment, the device name server 410 may exist in the form of a cloud server, an external storage, or a database (DB).

The device type classification unit 330 may transmit information on the determined type of the target device to the flow controller 320.

The flow control unit 320 transmits information on the target device type to the device disambiguation unit 340.

When there are a plurality of devices corresponding to the type determined by the device type classification unit 330, the device disambiguation unit 340 may disambiguate the target device through a conversation with the user. The device disambiguation unit 340 may generate a query message for selecting a target device from among a plurality of target device candidates. The device disambiguation unit 340 may generate a query message using a natural language generation (NLG) model. The device disambiguation unit 340 may convert a text query message into an audio signal using a text to speech (TTS) model. In an embodiment, the device disambiguation unit 340 may transmit a query message to the flow controller 320 so that the generated query message is output by the client device 110 that receives a voice input from the user. The device disambiguation unit 340 may transmit a text query message to the flow control unit 320, but is not limited thereto. The device disambiguation unit 340 may transmit an inquiry message converted into an audio signal to the flow controller 320.

The flow control unit 320 may transmit an inquiry message to the client device 110.

When it is necessary to select any one of a plurality of target devices, the device disambiguation unit 340 may provide information on the target device previously selected by the user by the user's existing response input and the existing response input together with a query message. have.

The device disambiguation unit 340 may determine a target device based on a response input from a user selecting any one of a plurality of target device candidates included in the query message. The user's response input may be transmitted from the client device 110 in the form of a voice signal. In one embodiment, the device disambiguation unit 340 recognizes the order in which the plurality of target device candidates are listed in the general name, installation location, or query message of the device from the response input, and determines the target device based on the response input. have. The device disambiguation unit 340 converts the response input received from the client device 110 into text, parses the text into words or phrases, and the normal name, installation location, or ordinal number of the device included in the text (sequence order). , Example:'first device') can be acquired. For example, if the query message includes a list in which a plurality of target device candidates are listed in a preset order, the device disambiguation unit 340 may within the list based on the user's response input such as "first" or "second". A device corresponding to an order of a specific target device may be recognized, and the recognized device may be determined as a target device.

The device disambiguation unit 340 may transmit information on a plurality of target device candidates to the intelligent device resolver 350.

The intelligent device resolver 350 acquires identification information about a plurality of devices previously registered in the IoT cloud server 400 in relation to the user's account information, and selects a target device based on device information about each of the plurality of devices. I can. The device information may include function performance information, location information, and status information of each of a plurality of devices previously registered in relation to the user's account information. The state information may include, for example, information about whether a plurality of devices are powered on or off, and at least one of an operation currently being executed.

The intelligent device resolver 350 may transmit information on the selected target device to the device disambiguation unit 340. In one embodiment, the intelligent device resolver 350 transmits at least one of information on the identification information (id), name, alias, location, manufacturer, model id, function performance capability, and current operation state of the target device to the device disambiguation unit ( 340).

The flow controller 320 may transmit information about the target device to the session manager. The session manager may transmit the parsed text and information on the target device to the second assistant model 200b.

The action plan management model 210 of the second assistant model 200b may manage device operations for each device type and information about a relationship between the operations. The second assistant model 200b may determine a second intent and a parameter by analyzing the parsed text using a second natural language understanding model corresponding to the type of the target device among the plurality of second natural language understanding models. The second assistant model 200b generates operation information on the operation to be performed by the target device by planning detailed operations to be performed by the device and the execution order of the detailed operations based on the second intent and parameters. can do. The second assistant model 200b may generate motion information by performing action planning related to an action execution of a TV agent or a speaker agent, for example.

The second assistant model 200b may transmit operation information and identification information of the target device to the session manager. The session manager may transmit motion information obtained from the second assistant model 200b to the response message controller 360. The response message controller 360 may transmit operation information and target device identification information to the IoT cloud server 400. In an embodiment, the response message controller 360 may transmit conversation information to the IoT cloud server 400. The conversation information may include at least one of a text converted from a voice input received from a user, a query message generated by the server 300 using a natural language generation model, and a text converted from a response input for the query message. .

The IoT cloud server 400 may acquire, determine, or generate a control command capable of controlling a device based on the received identification information and operation information of the target device. In an embodiment, the IoT cloud server 400 may obtain, determine, or generate a control command for causing the target device to perform detailed operations in the operation information based on the operation information. The control command may mean a command that controls the target device to execute detailed operations according to the operation information. However, the present invention is not limited thereto, and the IoT cloud server 400 may store operation information and a control command corresponding to identification information of a target device.

The IoT cloud server 400 may transmit a control command to the target device. By reading the generated control command, the target device can sequentially execute detailed operations in the operation information according to the execution order. The IoT cloud server 400 may receive a signal related to an operation execution result, which is a result of executing a control command by the target device, from the target device.

The IoT cloud server 400 may transmit an operation execution result signal to the response message controller 360.

The response message controller 360 may generate notification content based on a result of the operation execution. In an embodiment, the response message controller 360 may generate notification content using a natural language generation (NLG) model. The notification content may be a message indicating a result that the operation has been executed by the target device.

The response message controller 360 may determine at least one of text, audio, image, and video as the format of the notification content, and generate or obtain the notification content according to the determined format. The response message controller 360 may perform a text-to-speech (TTS) function for converting text into speech, and may convert notification content in a text format into an audio signal.

The response message controller 360 may transmit notification content to the session manager.

The session manager may transmit notification content to the client device 110. The session manager may determine a device to provide notification content according to a predefined rule. The predefined rule may include information on the format of the notification content corresponding to each operation. In addition, the predefined rule may additionally include information on a device to output notification content.

The client device 110 may output the received notification content through a speaker.

In FIG. 15, the device dispatcher model 310 is the final device through a query message that asks the user which device to select as the target device when there are multiple devices corresponding to the type determined in relation to the voice input received from the user. Although an embodiment of determining a target device to perform an operation is illustrated, it is not limited thereto. In one embodiment, the device dispatcher model 310 is trained by applying a pair of a first intent recognized from a user's voice input and a target device finally selected by the user as inputs and outputs of an artificial neural network. ), and the target device may be selected using the learning network model generated through the learning result. Here, the device dispatcher model 310 is trained using a known deep neural network (DNN) such as a convolution neural network (CNN) or a recurrent neural network (RNN). It can be, but is not limited thereto. Through a method through learning, the accuracy of determining a target device according to the user's intention may be improved.

16 is a diagram illustrating a device type classifier 330 of the device dispatcher model 310 according to an embodiment of the present disclosure.

Referring to FIG. 16, the device dispatcher model 310 may include a flow control unit 320 and a device type classification unit 330.

The device type classification unit 330 includes a proxy 331, a dictionary manager 332, a device name dispatcher 333, a device type dispatcher 334, and performs It may include a capability dispatcher 335, a rule engine 336, a criterion handler 337, and a grammar 338.

The dictionary manager 332 transmits a query to the device name server 410 (refer to FIG. 15) when the device name obtained from the user's voice input received from the client device is a personalized alias, and transmits a query to the device name server. Identification information of the device corresponding to the personalized alias may be received from. The dictionary manager 332 may synchronize the personalized alias of the device included in the text converted from the voice input with at least one vocabulary of synonyms, similar words, and slang registered in an external device name server. For example, when recognizing the word'stupid box' from the text, the dictionary manager 332 transmits a query to the device name server and retrieves identification information of the'living room TV' corresponding to the'stupid box'. Can receive. The dictionary manager 332 may synchronize the received identification information of the living room TV with an alias called'stupid box'.

The device name dispatcher 333 may parse the text converted from the voice input and obtain a device name from the parsed text. In one embodiment, the device name dispatcher 333 performs a syntactic analysis or semantic analysis using a natural language understanding model, and extracts a word or phrase corresponding to the device name from the text. I can. In one embodiment, the device name dispatcher 333 transmits a query regarding the extracted device name to the dictionary manager 332 and compares the device name with a predefined device name to recognize device identification information. have. For example, when the text "Play Bohemian Rhapsody on TV" is received from the device type classifier proxy 331, the device name dispatcher 333 may obtain a device name of "TV" from the text. .

The device type dispatcher 334 may determine the type of device based on the text converted from the voice input. The device type dispatcher 334 obtains the first intent from the text using the first natural language understanding model, and the device type matches the obtained first intent based on a matching rule previously stored in the rule engine 336 Can be determined. The detailed method of determining the first intent from the text using the first natural language understanding model is the same as the method of determining the target device type from the first intent described in FIG. 5, so a duplicate description will be omitted. do.

The performance capability dispatcher 335 may determine a target device type based on a function execution capability related to an operation indicated by the first intent. For example, at least one function may be matched with an operation indicated by the first intent. The performance capability dispatcher 335 may determine a target device type by identifying a device capable of performing a function matched with the first intent based on the function performance capability information of the device. Here, the function performance capability information may mean information on a function that the device can execute. For example, the mobile phone's ability to perform functions may be SNS, map, phone, Internet, etc., the TV's ability to play content, and the air conditioner's ability to control air temperature.

When the device type dispatcher 334 and the performance capability dispatcher 335 are composed of program codes or algorithms, the above-described functions may be performed using the grammars 338 of the rule engine 336. When the device type dispatcher 334 and the capability dispatcher 335 are implemented as machine learning or artificial neural network models, the rule engine 336 and the grammar 338 may be omitted.

When there are a plurality of devices corresponding to the determined type, the criteria handler 337 may store the order in which the plurality of candidate devices are arranged and mapping information between the actual target devices. In one embodiment, the reference handler 337 stores information on the order in which the plurality of candidate devices are listed, and is mapped to the target device in a specific order selected by the user's response input based on the information on the listing order. Device can be determined. For example, when there are three TVs such as living room TV, home TV, and children's room TV as target device candidates, the reference handler 337 determines what actual TV is mapped to each of the first TV, the second TV, and the third TV. It may contain identification information about.

17 is a diagram illustrating a device disambiguation unit 340 of the device dispatcher model 310 according to an embodiment of the present disclosure.

Referring to FIG. 17, the device dispatcher model 310 may include a flow control unit 320, a device disambiguation unit 340, and an intelligent device resolver 350.

The device disambiguation unit 340 includes a device disambiguation service 341, a query message generation unit 342, a device state manager 343, an IDR connection unit 344, and a conversation state tracking unit. (Conversation State Tracker) 345, and a database 346.

The device disambiguation service model 341 recognizes whether there is one or more target device candidates from the flow control unit 320, and when there are multiple target device candidates, a query message is generated for a signal to clarify the target device. It can be transmitted to the unit 342.

The query message generator 342 may generate a query message for selecting a target device from among a plurality of target device candidates. In an embodiment, the query message generator 342 may generate a query message using a natural language generation (NLG) model. In an embodiment, the query message generator 342 may convert a text query message into an audio signal using a text to speech (TTS) model.

The conversation state manager 343 may control a state regarding a conversation between a user and a client device or a conversation between devices. For example, when the target device is recognized as "TV" from the user's voice input in the idle state, the conversation state manager 343 is a state in which disambiguation is performed as to which TV is to be operated. Can be changed to.

IDR connection unit 344, when there are a plurality of device candidates corresponding to the type determined based on the user's voice input, through the query (query) transmission and reception with the intelligent device resolver 1640 to correspond to the name or type of the target device It is possible to determine how many device candidates exist and which device is an optimal target device.

In an embodiment, when the conversation state manager 343 recognizes that there are a plurality of devices corresponding to a type determined from a voice input received from a user, the state may be changed to a disambiguation state. The device disambiguation service model 341 may transmit a signal to the query message generator 342 based on a signal relating to the disambiguation state. The query message generator 342 may generate a query message for inquiring which device from among a plurality of candidate device candidates to be selected as a target device.

The conversation status tracking unit 345 may track the conversation status by acquiring information about a conversation situation between a user and a client device or between devices. For example, the conversation status tracking unit 345 relates to a situation such as whether the current situation is a state in which a response input for clarifying the target device has been received, and whether a conversation should be resumed in accordance with the received response input. Information can be obtained.

18 is a diagram illustrating a response message controller 360 of a device dispatcher model 310 according to an embodiment of the present disclosure.

Referring to FIG. 18, the device dispatcher model 310 may include a response message controller 360.

The response message controller 360 may distribute a signal to perform an operation in a client device corresponding to a listener that receives a voice input from a user and a target device that is an executer executing an operation. The response message control unit 360 includes a determination unit 361, a response dispatcher 362, a layout handler 363, operation information 364, a layout DB 365, and a natural language generation DB. (366) may be included.

The determiner 361 may determine a property of feedback that provides a result of an operation performed by the target device to a user according to the operation information obtained from the second assistant model. In an embodiment, the determination unit 361 may determine whether the operation information requires only execution of the operation through the target device or includes feedback such as output of a response message through natural language generation.

Here, the response message may be a message indicating an execution result indicating that the operation has been executed by the target device or the operation execution has failed. For example, if the target device is a TV and the TV executes an operation to play a movie, the response message may be "The movie was played on TV" indicating the execution result of the movie play operation. In another example, when the target device is a speaker and the speaker fails to execute the operation of increasing the volume, the response message indicates “The volume is in the highest state, so the volume increase is It may have failed.” In this case, the execution result may be "fail", and the cause of the failure received with the execution result may be "the current volume is in the highest state".

The response message may be configured in at least one format of text, audio, image, and video. The determination unit 361 may determine the format of the response message when the operation information requires feedback.

When receiving a signal including operation information 364 and layout information from the response dispatcher 362, the decision unit 361 outputs a control command and layout for executing the operation from the target device, and responds You can separate the signals so that the messages are output from each client device. For example, the decision unit 361 may transmit operation information and GUI information for performing an operation to the IoT cloud server 400, and a response message such as "Played a movie on TV" to the session manager. have. The response message may be generated using a message previously stored in the natural language generation DB 366. Here, the GUI information may be stored in the layout DB 365.

The response dispatcher 362 may determine a device to which a response message is to be output. The response dispatcher 362 may determine a device to which a response message is to be output according to a predefined rule. The predefined rule may include information on the format of a response message corresponding to each operation. Also, the predefined rule may additionally include information on a device to output a response message.

For example, the movie playback operation is defined as outputting a response message in an audio format, and is defined as being provided by the client device 110 that transmits the user's voice input. If the response message for the movie playback operation is output in an audio format and is defined as being provided by the client device, the server 300 may convert the response message into an audio format and transmit it to the client device.

In an embodiment, the response dispatcher 362 may determine the output device of the generated response message as a client device that receives a voice input from a user. This is to prevent a problem in which the user may be confused as to which device to communicate with when a response message is output from a device other than the device being interacted with.

The layout handler 363 may determine a layout necessary for the target device to actually perform an operation from among the previously stored layouts. In one embodiment, the layout handler 363 may determine a layout to be delivered to the target device.

The layout DB 365 may store a layout used by the target device to perform an operation. The layout DB 365 may store, for example, data related to a GUI related to music playback, movie playback, and the like.

FIG. 19 is a diagram illustrating an intelligent device resolver (IDR) 350 of a device dispatcher model 310 according to an embodiment of the present disclosure.

Referring to FIG. 19, the device dispatcher model 310 may include a flow controller 320 and an intelligent device resolver 350. The IoT cloud server 400 shown in FIG. 19 is the same as the IoT cloud server 400 (see FIG. 15) shown in FIG. 15, and the device name server 410 is a device name server 410 shown in FIG. 15), the overlapping description will be omitted. In FIG. 19, the IoT cloud server 400 and the device name server 410 are illustrated as separate and separate servers, but are not limited thereto. The IoT cloud server 400 and the device name server 410 may be integrated into one server.

When there are a plurality of device candidates corresponding to the type determined in relation to the text converted from the user's voice input, the flow control unit 320 transmits/receives a query to/from the intelligent device resolver 350 to determine the name of the target device or It is possible to determine how many device candidates corresponding to the type exist and which of the device candidates is an optimal target device.

The intelligent device resolver 350 uses the device information of a plurality of devices previously registered in the IoT cloud server 400 in relation to the first intent obtained by interpreting the text through the first natural language understanding model and the user's account information. Based on the plurality of device candidates, an optimal target device may be determined. The device information includes, for example, location information of the client device 110, function performance information of each of a plurality of devices previously registered in the IoT cloud server 400, location information, power on/off status, and current operation information. It may include information on at least one of the.

The intelligent device resolver 350 may include a device resolver 351 and a sync manager 352.

The device resolver 351 may obtain a list of a plurality of devices previously registered in relation to the user's account information from the IoT cloud server 400. In an embodiment, the device resolver 346 may identify a plurality of devices installed around the client device 110 based on the location of the client device 110. In an embodiment, the device resolver 351 may determine a target device based on at least one of a type, a name, a common name, a location, an operation currently being performed, and whether power is turned on or off of each of the plurality of devices. In one embodiment, the device resolver 351 transmits a query to the IoT cloud server 400 for requesting information on a function performance capability according to the name or type of the device, and transmits a device from the IoT cloud server 400 It is possible to receive information on the function performance capability of The device resolver 351 may determine a target device based on the received information.

In an embodiment, the device resolver 351 stores at least one of information on identification information (id), name, alias, location, manufacturer, model id, function performance capability, and current operating state of the target device. , See FIG. 15).

The sync manager 352 may synchronize the device name server 410 with the device name. In one embodiment, the sync manager 352 may synchronize the device alias obtained from the text converted from the user's voice input with the alias of the device previously stored in the external device name server 410.

The device name server 410 may be an external server that stores at least one personalized vocabulary among synonyms, similar words, and slang words registered by a user in relation to the device name. When a user registers a device, the device name server 410 may register and store a personalized word or vocabulary, and the personalized word or vocabulary may be added or deleted. In an embodiment, the device name server 410 may exist in the form of a cloud server, an external storage, or a database (DB).

In an embodiment, the sync manager 352 may synchronize the device name stored in the device name server 410 and the device name stored in the Redis 420 with the personalized alias of the device. In this case, the sink manager 352 may operate like a cache server.

20 is a conceptual diagram illustrating an action plan management model 210 according to an embodiment of the present disclosure. The action plan management model 210 may be a component included in the server 300, but is not limited thereto. The action plan management model 210 may be a separate component from the server 300. In one embodiment, the action plan management model 210 may be included in the IoT cloud server 400. The action plan management model 210 may constitute a second assistant model 200b together with a plurality of second natural language understanding models 300b. In one embodiment, the action plan management model 210 may be configured to include a plurality of second natural language understanding models 300b.

The action plan management model 210 may be a model that manages operation information on a device to generate an action plan. The action plan management model 210 may be a server that stores operation information necessary for the target device determined by the server 300 to perform operations. The action plan management model 210 may include at least one of an artificial intelligence learning model, an artificial intelligence learning algorithm, a routine, a set of instructions, and a natural language understanding model. In one embodiment, the action plan management model 210 may include a natural language generation (NLG) model.

The action plan management model 210 may manage information about a plurality of detailed operations of a device and a relationship between a plurality of detailed operations for each device type. The association relationship between each detailed operation and another detailed operation among the plurality of detailed operations includes information on other detailed operations that must be executed before executing the detailed operation in order to execute one detailed operation.

The action plan management model 210 may determine a plurality of detailed operations to be performed by the device based on a second intent and a parameter determined from text through the second natural language understanding model. In an embodiment, the action plan management model 210 may determine an input parameter value required to execute a plurality of determined detailed operations or a result value output by execution of the detailed operations. Here, the input parameter value and the output result value may be defined as a concept of a designated format (or class). Accordingly, the action plan may include a plurality of detailed operations and a plurality of concepts determined in relation to the second intent and parameter.

The action plan management model 210 may determine a plurality of operations and a relationship between the plurality of concepts in stages (or hierarchical). For example, the action plan management model 210 may perform planning to determine an execution order of a plurality of detailed operations determined based on a second intent and a parameter, and may generate operation information as a result of the planning. That is, the action plan management model 210 plans an execution order of a plurality of detailed operations based on an input parameter value required for execution of a plurality of detailed operations and a result value output by execution of a plurality of detailed operations, Through this, motion information can be generated.

The operation information may be information related to detailed operations to be performed by the device, a correlation relationship between each detailed operation and another detailed operation, and an execution order of the detailed operations. The operation information may include, for example, functions to be executed by the target device in order to perform a specific operation, an execution order of functions, an input value necessary to execute the functions, and an output value output as a result of execution of the functions. , Is not limited thereto. The operation information may include an action plan generated by the action plan management model 210.

The action plan management model 210 may perform planning using information stored in a capsule database in which a set of relationships between concepts and actions is stored. The action plan management model 210 may store actions and concepts corresponding to the actions in a capsule-type database, a Concept Action Network (CAN). The action plan management model 210 may be configured as a capsule database that stores CAN for each device.

Referring to FIG. 20, the action plan management model 210 may include a speaker CAN 212, a mobile CAN 214, and a TV CAN 216. The speaker CAN 212 may include information on detailed operations including speaker control, media playback, weather, and TV control, and an action plan for storing concepts corresponding to each of the detailed operations in a capsule form. The mobile CAN 214 may include an action plan that stores information on detailed operations including SNS, mobile control, map, and QA, and concepts corresponding to each of the detailed operations in a capsule form. The TV CAN 216 may include information on detailed operations including shopping, media playback, education, and TV playback, and an action plan for storing concepts corresponding to each of the detailed operations in a capsule form. In one embodiment, a plurality of capsules included in each of the speaker CAN 212, mobile CAN 214, and TV CAN 216 are stored in a function registry, which is a component in the action plan management model 210 Can be.

In one embodiment, the action plan management model 210 determines detailed actions corresponding to the second intent and parameters determined by the server 300 interpreting the text converted from the speech input through the second natural language understanding model. It may contain the necessary strategy registry. When there are a plurality of action plans related to the text, the strategy registry may include reference information for determining one action plan. In one embodiment, the action plan management model 210 may include a follow up registry in which information of a follow-up operation for suggesting a follow-up operation to a user in a specified situation is stored. The subsequent operation may include, for example, a subsequent speech.

In one embodiment, the action plan management model 210 may include a layout registry that stores layout information output by the target device.

In one embodiment, the action plan management model 210 may include a vocabulary registry in which vocabulary information included in capsule information is stored. In an embodiment, the action plan management model 210 may include a dialog registry in which information about a conversation (or interaction) with a user is stored.

21 is a diagram illustrating a capsule database 220 stored in the action plan management model 210 according to an embodiment of the present disclosure.

Referring to FIG. 21, the capsule database 220 stores detailed operations and relationship information regarding concepts corresponding to the detailed operations. The capsule database 220 may be implemented in a CAN (Concept Action Network) form. Capsule database 220 may store a plurality of capsules (230, 240, 250). The capsule database 220 may store detailed operations for executing operations related to a user's voice input, and input parameter values and output result values required for detailed operations in the form of a concept action network (CAN).

The capsule database 220 may store operation related information for each device. In the embodiment shown in FIG. 21, the capsule database 220 may store a plurality of capsules 230, 240, 250 related to operations performed by a specific device, for example, a TV. In one embodiment, one capsule (eg, capsule A 230) may correspond to one application. One capsule may include at least one or more detailed operations and at least one or more concepts for performing a specified function. For example, the capsule A 230 includes a detailed operation 231a and a concept 231b corresponding to the detailed operation 231a, and the capsule B 240 includes a plurality of detailed operations 241a, 242a, and 243a. ), and a plurality of concepts 241b, 242b, and 243b corresponding to each of the plurality of detailed operations 241a, 242a, and 243a.

The action plan management model 210 may generate an action plan for performing an operation related to a user's voice input using a capsule stored in the capsule database 220. For example, the action plan management model 210 may generate an action plan using capsules stored in the capsule database 220. For example, the action plan management model 210 includes detailed operations 231a and concept 231b of capsule A 230, detailed operations 241a, 242a, 243a and concepts 241b of capsule B 240. , 242b, 243b), and the detailed operation 251a and concept 251b of the capsule C 250 may be used to generate the action plan 260.

The action plan management model 210 may provide the generated action plan 260 to the server 300.

22 is a block diagram illustrating components of the device 1000 of the present disclosure.

In FIGS. 2A to 19 of the present disclosure, it is described that embodiments of the present disclosure are performed by the server 300, but are not limited thereto. Embodiments of the present disclosure may be performed in the device 1000 as well as the server 300. The client device 110 or the plurality of devices 120 illustrated in the present disclosure may include components corresponding to components included in the device 1000 of FIG. 22. For example, the processor 1300 is the same as the processor 114 (see FIG. 14) of the client device 110, and the communication interface 1500 is the same as the communication interface 118 (see FIG. 14) of the client device 110. And, the microphone 1620 may be the same as the microphone 112 of the client device 110 (see FIG. 14), and the memory 1700 may be the same as the memory 116 of the client device 110 (see FIG. 14).

Referring to FIG. 22, a device 1000 according to an embodiment of the present disclosure includes a user input unit 1100, an output unit 1200, a processor 1300, a sensing unit 1400, a communication interface 1500, and A/ A V input unit 1600 and a memory 1700 may be included.

The user input unit 1100 refers to a means for a user to input data for controlling the device 1000. For example, the user input unit 1100 includes a key pad, a dome switch, a touch pad (contact type capacitance method, pressure type resistive film method, infrared detection method, surface ultrasonic conduction method, integral type). Tension measurement method, piezo effect method, etc.), a jog wheel, a jog switch, and the like, but are not limited thereto.

The user input unit 1100 may request a response input for a query message and receive a response input from a user.

The output unit 1200 may output an audio signal, a video signal, or a vibration signal, and the output unit 1200 may include a display unit 1210, an audio output unit 1220, and a vibration motor 1230. have.

The display 1210 displays and outputs information processed by the device 1000. For example, the display 1210 may receive notification content indicating an execution result of an operation from the server 300 and display the notification content. In an embodiment, the display 1210 may display text and GUI received from the server 300.

In an embodiment, the display 1210 may display content related to an image, such as playing a movie or playing a TV broadcast, based on a control command received from the IoT cloud server 400.

The sound output unit 1220 outputs audio data received from the communication interface 1500 or stored in the memory 1700. Also, the sound output unit 1220 outputs sound signals related to functions performed by the device 1000. When the notification content received from the server 300 is a voice signal, the sound output unit 1220 may output the notification content.

The processor 1300 typically controls the overall operation of the device 1000. For example, the processor 1300 executes programs stored in the memory 1700 and processes data stored in the memory 1700, so that the user input unit 1100, the output unit 1200, the sensing unit 1400, and communication The interface 1500, the A/V input unit 1600, and the like can be generally controlled. In addition, the processor 1300 may execute programs stored in the memory 1700 to process data stored in the memory 1700, thereby performing the function of the device 1000 illustrated in FIGS. 2 to 17.

The processor 1300 is, for example, a central processing unit (Central Processing Unit), a microprocessor (microprocessor), a graphics processor (Graphic Processing Unit), ASICs (Application Specific Integrated Circuits), DSPs (Digital Signal Processors), DSPDs (Digital Signal Processing Devices), Programmable Logic Devices (PLDs), and Field Programmable Gate Arrays (FPGAs), but are not limited thereto.

Specifically, the processor 1300 may control the communication interface 1500 to receive a user's voice input from the server 300 or another device connected through a network. The processor 1300 converts the received speech input into text by performing ASR using data on the ASR model 1712 stored in the memory 1700, and stores the first natural language understanding model 1714 in the memory 1700. ) May be used to determine a first intent related to text, and a type of a target device related to the first intent may be determined using data related to the device dispatcher model 1716.

In one embodiment, the processor 1300 includes a second natural language understanding model corresponding to the determined target device type among the plurality of second natural language understanding models 1722 stored in the memory 1700 and an action plan management model stored in the memory 1700. Through 1724, the target device may obtain operation information for executing operations related to the user's intention included in the text.

In an embodiment, the processor 1300 may control the communication interface 1500 to receive device information of a plurality of devices previously registered in the IoT cloud server 400 in relation to the user's account information. Device information of a plurality of previously registered devices may be stored in the memory 1700. In this case, the processor 1300 may obtain device information from the memory 1700. The processor 1300 may determine a target device based on device information of another device.

In one embodiment, the processor 1300 obtains the name of the device from the text using the first natural language understanding model 1714 stored in the memory 1700, and uses the device dispatcher model 1716 stored in the memory 1700. Using the related data, the type of the target device may be determined based on the device name and device information received from the IoT cloud server 400.

In one embodiment, the processor 1300 uses the first natural language understanding model 1714 to extract a common name related to the device type and a word or phrase related to the installation location of the device from the text, and the extracted common name and The type of target device may be determined based on the installation location.

In one embodiment, when extracting a personalized alias from the text, the processor 1300 transmits the personalized alias to an external server that stores at least one of synonyms, similar words, and slang words of a plurality of devices previously registered by the user. And, the communication interface 1500 may be controlled to receive device identification information corresponding to a personalized alias from an external server. The personalized alias and device identification information corresponding thereto may be stored in the memory 1700. In this case, the processor 1300 may obtain device identification information from the memory 1700. The processor 1300 may determine a target device based on the received or acquired device identification information.

In one embodiment, the processor 1300 is the name, type, installation location, and power on/off of each of a plurality of devices previously registered in relation to the user's account information, and during an operation currently being executed. The communication interface 1500 may be controlled to receive information including at least one from the external IoT cloud server 400.

In one embodiment, the processor 1300 generates a query message for selecting a target device from among a plurality of candidate devices using a natural language generation (NLG) model when a plurality of devices corresponding to the determined target device type exist. can do. The processor 1300 may control the client device to output the generated query message.

In addition, the processor 1300 may perform all operations performed by the server 300.

The sensing unit 1400 may detect a state of the device 1000 or a state around the device 1000 and transmit the sensed information to the processor 1300. The sensing unit 1400 may be used to generate location information of a user or device 1000.

The sensing unit 1400 includes a magnetic sensor 1410, an acceleration sensor 1420, a temperature/humidity sensor 1430, an infrared sensor 1440, a gyroscope sensor 1450, and a position sensor. (Eg, GPS) 1460, barometric pressure sensor 1470, proximity sensor 1480, and RGB sensor (illuminance sensor) 1490 may include at least one of, but is not limited thereto. Since the function of each sensor can be intuitively inferred by a person skilled in the art from its name, detailed description will be omitted.

For example, the device 1000 may obtain location information of the device 1000 through the location sensor 1460. For example, the location information may indicate a place where the device 1000 is currently located or location coordinates.

The communication interface 1500 may include one or more components through which the device 1000 communicates with other devices, the server 300, and the IoT cloud server 400. The communication interface 1500 is, for example, wired LAN, wireless LAN, Wi-Fi, Bluetooth, zigbee, Wi-Fi Direct (WFD), infrared communication (IrDA, infrared Data Association), BLE (Bluetooth Low Energy), NFC (Near Field Communication), Wibro (Wireless Broadband Internet, Wibro), WiMAX (World Interoperability for Microwave Access, WiMAX), SWAP (Shared Wireless Access Protocol), Wiigg Data communication with other devices, the server 300, and the IoT cloud server 400 may be performed using at least one of data communication methods including (Wireless Gigabit Allicance, WiGig) and RF communication.

For example, the communication interface 1500 may include a short-range communication unit 1510, a mobile communication unit 1520, and a broadcast reception unit 1530.

The short-range wireless communication unit 1510 includes a Bluetooth communication unit, a Bluetooth Low Energy (BLE) communication unit, a near field communication unit, a WLAN (Wi-Fi) communication unit, a Zigbee communication unit, and an infrared ( IrDA, infrared data association) communication unit, WFD (Wi-Fi Direct) communication unit, UWB (ultra wideband) communication unit, Ant+ communication unit, etc. may be included, but is not limited thereto.

In an embodiment, the device 1000 may obtain location information of the device 1000 through the short-range communication unit 1510. For example, the device 1000 may determine a location where the device 1000 is located through an NFC tag. Also, for example, the device 1000 may determine a location where the device 1000 is located through an identifier of Wi-Fi. For example, the device 1000 may determine the location where the device 1000 is located by checking the SSID of the Wi-Fi to which the device 1000 is connected.

The mobile communication unit 1520 transmits and receives a radio signal with at least one of a base station, an external terminal, and a server on a mobile communication network. Here, the wireless signal may include a voice call signal, a video call signal, or various types of data according to transmission/reception of text/multimedia messages.

The broadcast receiver 1530 receives a broadcast signal and/or broadcast-related information from outside through a broadcast channel. Broadcast channels may include satellite channels and terrestrial channels. Depending on the implementation example, the device 1000 may not include the broadcast receiver 1530.

The A/V (Audio/Video) input unit 1600 is for inputting an audio signal or a video signal, and may include a camera 1610 and a microphone 1620. The camera 1610 may obtain an image frame such as a still image or a video through an image sensor in a video call mode or a shooting mode. The image captured through the image sensor may be processed through the processor 1300 or a separate image processing unit (not shown).

The microphone 1620 receives an external sound signal and processes it as electrical voice data. For example, the microphone 1620 may receive a voice signal from a user. The microphone 1620 may receive a user's voice input. The microphone 1620 may use various noise removal algorithms to remove noise generated in the process of receiving an external sound signal.

The memory 1700 may store a program for processing and controlling the processor 1300 and may store data input to the device 1000 or output from the device 1000.

The memory 1700 is a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (for example, SD or XD memory), and RAM. (RAM, Random Access Memory) SRAM (Static Random Access Memory), ROM (ROM, Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, magnetic disk And at least one type of storage medium among optical disks.

In one embodiment, the memory 1700 may include a first assistant model 1710 and a second assistant model 1720, a UI module 1730, a touch screen module 1740, and a notification module 1750. . The first assistant model 1710 may store data related to the ASR model 1712, the first natural language understanding model 1714, and the device dispatcher model 1716. The second assistant model 1720 may store data about a plurality of second natural language understanding models 1722 and an action plan management model 1724. In an embodiment, the memory 1700 may store device information and device name information.

The UI module 1730 may provide a specialized UI, GUI, etc. that are interlocked with the device 1000 for each application.

The touch screen module 1740 may sense a user's touch gesture on a touch screen and transmit information about the touch gesture to the processor 1300. The touch screen module 1740 according to some embodiments may recognize and analyze a touch code. The touch screen module 1740 may be configured as separate hardware including a controller.

The notification module 1750 may generate a signal to notify the occurrence of an event of the device 1000. Examples of events occurring in the device 1000 include call signal reception, message reception, key signal input, and schedule notification. The notification module 1750 may output a notification signal in the form of a video signal through the display unit 1210, may output a notification signal in the form of an audio signal through the sound output unit 1220, and the vibration motor 1230 It is also possible to output a notification signal in the form of a vibration signal through.

The program executed by the server 300 described herein may be implemented as a hardware component, a software component, and/or a combination of a hardware component and a software component. A program can be executed by any system capable of executing computer-readable instructions.

Software may include a computer program, code, instructions, or a combination of one or more of them, and configure the processing unit to operate as desired or process it independently or collectively. You can command the device.

The software may be implemented as a computer program including instructions stored in a computer-readable storage media. The computer-readable recording medium includes, for example, a magnetic storage medium (e.g., read-only memory (ROM), random-access memory (RAM), floppy disk, hard disk, etc.) and optical reading medium (e.g., CD-ROM (CD-ROM) and DVD (Digital Versatile Disc)). The computer-readable recording medium is distributed over network-connected computer systems, so that computer-readable codes can be stored and executed in a distributed manner. The medium is readable by a computer, stored in memory, and executed on a processor.

The computer-readable storage medium may be provided in the form of a non-transitory storage medium. Here,'non-transient' means that the storage medium does not contain a signal and is tangible, but does not distinguish between semi-permanent or temporary storage of data in the storage medium.

In addition, a program according to the embodiments disclosed in the present specification may be included in a computer program product and provided. Computer program products can be traded between sellers and buyers as commodities.

The computer program product may include a software program and a computer-readable storage medium storing the software program. For example, a computer program product is a product (for example, a downloadable application) in the form of a software program that is electronically distributed through a device manufacturer or an electronic market (eg, Google Play Store, App Store). It may include. For electronic distribution, at least a part of the software program may be stored in a storage medium or may be temporarily generated. In this case, the storage medium may be a server of a manufacturer, a server of an electronic market, or a storage medium of a relay server temporarily storing a software program.

The computer program product may include a storage medium of a server or a storage medium of a device in a system composed of a server and a device. Alternatively, when there is a server or a third device (eg, a smartphone) that is communicatively connected to the device, the computer program product may include a storage medium of the third device. Alternatively, the computer program product may include a software program itself transmitted from a server to a device or a third device, or transmitted from a third device to the device.

In this case, one of the server, the device and the third device may execute the computer program product to perform the method according to the disclosed embodiments. Alternatively, two or more of a server, a device, and a third device may execute a computer program product to distribute and implement the method according to the disclosed embodiments.

For example, a server (eg, an IoT cloud server or an artificial intelligence server) may execute a computer program product stored in the server to control a device communicating with the server to perform the method according to the disclosed embodiments.

As another example, a third device may execute a computer program product, and a device connected in communication with the third device may be controlled to perform a method according to the disclosed embodiment.

When the third device executes the computer program product, the third device may download the computer program product from the server and execute the downloaded computer program product. Alternatively, the third device may execute the computer program product provided in a preloaded state to perform the method according to the disclosed embodiments.

As described above, although the embodiments have been described by the limited embodiments and drawings, various modifications and variations are possible from the above description by those of ordinary skill in the art. For example, the described techniques are performed in a different order from the described method, and/or components such as a computer system or module described are combined or combined in a form different from the described method, or other components or equivalents Even if substituted or substituted by, appropriate results can be achieved.

Claims (21)

In a network environment including a server and a plurality of devices, in a method for the server to control a device based on a voice input,
Receiving a user's voice input from a client device;
Converting the received speech input into text by performing Automatic Speech Recognition (ASR);
Determining a type of a target device related to the text by applying a first natural language understanding model to the text;
Selecting a second natural language understanding model corresponding to the determined type from among a plurality of second natural language understanding models;
Determining a target device based on the determined type and device information of the plurality of devices;
Analyzing the text by applying the selected second natural language understanding model to at least some of the texts, thereby obtaining operation information regarding an operation to be performed by the target device; And
Transmitting the obtained motion information to the target device;
Containing, method.
The method of claim 1,
The first natural language understanding model is a model configured to determine the target device type by analyzing the text,
The second natural language understanding model is a model configured to obtain the operation information to be performed by the determined target device by interpreting the text in relation to the specific type of device.
The method of claim 1,
Determining the type of the target device,
Parsing the text in units of words or phrases;
Obtaining a first intent by analyzing the parsed text using the first natural language understanding model;
Acquiring characteristic information of the plurality of devices from the IoT cloud server; And
Determining the target device type by using a matching model capable of determining a relationship between the first intent and characteristic information of the plurality of devices;
Containing, method.
The method of claim 3,
The step of determining the target device type,
Acquiring a plurality of numerical values indicating a degree of matching between the first intent and a plurality of device types by using the matching model; And
Determining a device type having a maximum numerical value among the obtained numerical values as the target device type;
Containing, method.
The method of claim 3,
The matching model comprises a matching rule obtained through learning through a rule-based system.
The method of claim 1,
Determining the type of the target device,
Parsing the text in units of words or phrases;
Extracting a device name from the parsed text by using the first natural language understanding model; And
Determining the type based on the extracted device name;
Containing, method.
The method of claim 1,
Acquiring device information of the plurality of devices; Including more,
Acquiring device information of the plurality of devices,
A method of acquiring from the IoT cloud server at least one of function performance information, location information, power on/off information, and information on an operation currently being performed of each of the plurality of devices.
The method of claim 1,
Generating a query message for selecting the target device from among a plurality of candidate devices using a Natural Language Generator (NLG) model when there are a plurality of devices corresponding to the determined type; And
The determining of the target device comprises determining the target device based on a user's response input to the generated query message.
The method of claim 1,
Obtaining the operation information,
Interpreting the text using the second natural language understanding model specialized for the type of the target device, thereby obtaining a second intent and a parameter;
Providing the second intent, the parameter, and identification information of the target device to an action plan management model that stores action plans related to each of the plurality of devices; And
Obtaining operation information regarding an operation to be performed by the target device from the action plan management model; Containing, method.
The method of claim 9,
The operation information is related to the second second intent and the parameter from information on the relationship between detailed operations and detailed operations of the target device previously stored in the action plan management model, according to the action plan management model. Operations are selected and obtained by planning an execution order of the selected detailed operations.
In the server for controlling a device based on a user's voice input,
A communication interface for performing data communication with a client device and a plurality of devices;
A memory storing a program including one or more instructions; And
A processor that executes one or more instructions of a program stored in the memory;
Including,
The processor controls the communication interface to receive a user's voice input from the client device,
By performing ASR (Automatic Speech Recognition), the received speech input is converted into text,
By applying a first natural language understanding model to the text, a type of a target device related to the text is determined, and
Selecting a second natural language understanding model corresponding to the determined type from among a plurality of second natural language understanding models,
Determine a target device based on the determined type and device information of the plurality of devices,
By analyzing the text by applying the selected second natural language understanding model to at least a portion of the text, operation information regarding an operation to be performed by the target device is obtained,
The server for controlling the communication interface to transmit the obtained operation information to the target device.
The method of claim 11,
The first natural language understanding model is a model configured to determine the target device type by analyzing the text,
The second natural language understanding model is a model configured to obtain the operation information to be performed by the determined target device by interpreting the text in relation to the specific type of device.
The method of claim 11,
The processor,
The text is parsed in units of words or phrases, the parsed text is analyzed using the first natural language understanding model to obtain a first intent, and the first intent and A server configured to determine a type of the target device by using a matching model capable of determining a relationship between characteristic information of the plurality of devices.
The method of claim 13,
The processor,
The server for controlling the communication interface to obtain characteristic information of the plurality of devices from the IoT cloud server.
The method of claim 13,
The processor,
Using the matching model, a plurality of numerical values indicating a matching degree between the first intent and a plurality of device types are obtained, and a device type having the largest numerical value among the obtained plurality of numerical values is selected as the target device. Server, determined by type.
The method of claim 13,
The matching model includes a matching rule obtained through learning through a rule-based system.
The method of claim 13,
The processor,
Parses the text in units of words or phrases, extracts a device name from the parsed text using the first natural language understanding model, and determines the type based on the extracted device name That, the server.
The method of claim 11,
The processor, from the IoT cloud server, includes at least one of capability information, installation location information, power on/off information, and information on an operation currently being performed of each of the plurality of devices. And controlling the communication interface to receive device information.
The method of claim 11,
The processor,
When a plurality of devices corresponding to the determined type exist, a query message for selecting the target device from among a plurality of candidate devices is generated using a natural language generator (NLG) model, and the generated query message Determining the target device based on the user's response input for the server.
The method of claim 11,
The processor,
By analyzing the text using the second natural language understanding model specialized for the type of the target device, the second intent and the parameter are obtained, and the action plan management model stores the action plans related to each of the plurality of devices. A server that provides a second intent, the parameter, and identification information of the target device, and obtains operation information about an operation to be performed by the target device from the action plan management model.
A computer-readable recording medium storing a program for executing the method of claim 11 on a computer.

Images