KR20200033140A - System and method for providing voice assistant service - Google Patents

Abstract

The present disclosure relates to an artificial intelligence (AI) system utilizing machine learning algorithm such as deep learning and the like, and an application thereof. The method for providing a voice assistant service by a device comprises the operations of: receiving a character specialized model; receiving a user voice input including a word for indicating a character and a sentence for requesting a response of a voice assistant program; determining the character specialized model according to the word indicating the character; generating a response message for the sentence, which requests the response of the voice assistant program, using the character specialized model; and outputting the generated response message.

Description

System and method for providing voice assistant service {SYSTEM AND METHOD FOR PROVIDING VOICE ASSISTANT SERVICE}

The present disclosure relates to a system and method for providing a voice assistant service, and more particularly, to a system and method for providing a voice assistant service specialized for a given character.

The Voice Assistant service processes speech recognition (ASR: Automated Speech Recognition) on audio signals, Natural Language Understanding (NLU) processing, Dialogue Manager (DM) processing, and Natural Language generation (NLG) processing. And a text to speech (TTS) process, providing a response to a user's speech.

In the past, the Voice Assistant service only provides answers in a predetermined style, and to change the style, it has to be changed through the setting menu. In addition, the voice assistant's ASR, NLU, DM, NLG, and TTS can be operated through rule-based algorithms other than artificial intelligence, and artificial intelligence can be grafted to improve the performance of the voice assistant.

Some embodiments may provide a system and method for providing a voice assistant service specialized for a character.

In addition, some embodiments may provide a system and method capable of providing a voice assistant service using an AI model specialized for a character.

In addition, some embodiments may provide a system and method for providing a voice assistant service specialized for various characters by using a voice assistant model and a character specialization model generated by another device of a 3rd party together.

As a technical means for achieving the above technical problem, a first aspect of the present disclosure includes an operation of receiving a character specialization model from an external device; Receiving a user voice input including a word indicating a character and a sentence requesting a response from the voice assistant program; Determining the character specialization model according to the word indicating the character; Generating a response message for the sentence requesting a response from the voice assistant program using the character-specific model; And outputting the generated response message. It is possible to provide a method for a device to provide a voice assistant service through a voice assistant program.

In addition, a second aspect of the present disclosure, a microphone for receiving a user voice input; Communication interface; A memory storing one or more instructions; And one or more processors executing a voice assistant program by executing the one or more instructions. Including, the processor receives a character-specific model to interwork with the voice assistant program from the server through the communication interface, and requests a response of the voice assistant program and a word indicating a character corresponding to the character-specific model. The user voice input including the sentence to be received is received through the microphone, and using the character-specific model interworking with the voice assistant program, a response message is generated for the sentence requesting a response from the voice assistant program, It is possible to provide a device that provides a voice assistant service that outputs the generated response message.

In addition, a third aspect of the present disclosure includes a communication interface; A memory storing one or more instructions; And one or more processors that execute the one or more instructions; wherein the processor acquires a character specialization model to interoperate with the voice assistant program of the device through the communication interface, and instructs a character corresponding to the character specialization model. When a user voice input including a word and a sentence requesting a response from the voice assistant program is received from the device through the communication interface, the character assistant model interworking with the voice assistant program is used to generate the voice assistant program. It is possible to provide a server that provides a voice assistant service, which generates a response message for a sentence requesting a response.

1 is a schematic diagram of a system for providing a voice assistant service according to some embodiments.
2 is a flowchart of a method in which a device provides a voice assistant service using a character-specific model according to some embodiments.
3 is a flow diagram of a method for a device to download a character-specific model from a server, according to some embodiments.
4 is a flow diagram of a method for downloading a character-specific model by a device approaching a predetermined product according to some embodiments.
5 is a flow diagram of a method for a device to change the response mode of a voice assistant according to some embodiments.
6 is a flowchart of a method in which a server provides a learning model generation tool to another device according to some embodiments, and provides a character-specific model generated by the other device to the device.
7 is a flowchart of a method for another device to generate and update a character-specific model using a learning model generation tool according to some embodiments.
8 is a flow diagram of a method for a device, a server, and other devices to provide a voice assistant service according to some embodiments.
9 is a block diagram of a learning model generation tool in accordance with some embodiments.
10 is a diagram illustrating an example in which a character-specific model is generated using a learning model generation tool according to some embodiments.
11 is a diagram illustrating an example in which another device generates a character specialization model according to some embodiments.
12 is a diagram illustrating an example in which a device according to some embodiments uses a character-specific model and a voice assistant learning model together.
13 is a diagram illustrating an example in which a device provides a voice assistant service using a character-specific model according to some embodiments.
14 is a diagram illustrating detailed operations in which a device provides a voice assistant service according to some embodiments.
15 is a diagram illustrating an example in which a device according to some embodiments downloads a character-specific model from a server.
16 is a diagram illustrating an example in which a device according to some embodiments acquires a character specialization model by approaching a character product.
17 is a diagram for a device in accordance with some embodiments, showing a voice assistant service dialog.
18 is a diagram illustrating an example of a response message according to a response mode according to some embodiments.
19 and 20 are block diagrams of devices in accordance with some embodiments.
21 is a block diagram of a server according to some embodiments.
22 is a block diagram of a processor in accordance with some embodiments.
23 is a block diagram of a data learning unit according to some embodiments.
24 is a block diagram of a data recognition unit according to some embodiments.
25 is a diagram illustrating an example of learning and recognizing data by interworking a device and a server according to some embodiments.

Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those skilled in the art to which the present disclosure pertains can easily implement them. However, the present disclosure can be implemented in many different forms and is not limited to the embodiments described herein. In addition, in order to clearly describe the present disclosure in the drawings, parts irrelevant to the description are omitted, and like reference numerals are assigned to similar parts throughout the specification.

Throughout the specification, when a part is "connected" to another part, this includes not only "directly connected" but also "electrically connected" with another element in between. . Also, when a part “includes” a certain component, this means that other components may be further included rather than excluding other components, unless otherwise specified.

Also, in the present specification, the voice assistant service may be a service that provides a conversation with a user. In the voice assistant service, a response message may be provided to the user as if a person directly communicates with the user in consideration of the user's situation and the device's situation. In addition, in the voice assistant service, information required by the user, such as the user's personal assistant, may be appropriately generated and provided to the user.

The artificial intelligence model is an artificial intelligence algorithm, and may be a model trained using at least one of machine learning, neural networks, genes, deep learning, and classification algorithms.

The model model of the voice assistant service may be an artificial intelligence model in which criteria and methods for providing a response message to a user in the voice assistant service are learned. The model of the voice assistant service includes, for example, a model for recognizing the user's input voice, a model for interpreting the user's input voice, a model for generating a response message to be provided to the user, and output characteristics of the response message. A model for determining may be included, but is not limited thereto. Models constituting the model of the voice assistant service may be an artificial intelligence model to which an artificial intelligence algorithm is applied.

The character specialization model is a model for the voice assistant service, and is an artificial intelligence model in which features are reflected in predetermined characters. The character-specific model may be a model in which standards and methods for providing a response message specific to a character to a user are learned, using, for example, character characteristics, characteristics of a user who prefers the character, and content information related to the character. You can. The character-specific model can be used independently, in parallel or auxiliary to the model of the voice assistant service. Models constituting the character-specific model may be an artificial intelligence model to which an artificial intelligence algorithm is applied.

Hereinafter, the present disclosure will be described in detail with reference to the accompanying drawings.

1 is a schematic diagram of a system for providing a voice assistant service according to some embodiments.

Referring to FIG. 1, a system for providing a voice assistant service may include a device 1000, a server 2000, and other devices 3000.

The device 1000 may output a response message for a user's voice input using a model of the voice assistant service in order to provide the voice assistant service. The model of the voice assistant service may be a basic model used by the voice assistant program installed by default on the device 1000. In addition, the device 1000 may receive and install a character-specific model for a given character in the device 1000, and use the character-specific model to provide a voice assistant service specialized for a given character.

The server 2000 provides a model creation tool for generating a character-specific model to be used in the voice assistant service to another device 3000 of the 3rd party, and the character-specific model generated by the other device 3000 is generated by another device ( 3000). In addition, the server 2000 may provide the received character specialization model to the device 1000. The model generation tool for generating a character-specific model may be, for example, an SDK (Software Develop Kit), and may provide a plurality of functions for generating a character-specific model. The server 2000 may be a server for providing a voice assistant service, but is not limited thereto. The server 2000 may include, for example, a server for providing an application to a device, and a server for providing theme data of an OS installed on the device to the device.

The other device 3000 may generate a character-specific model using a model generation tool for generating a character-specific model. The other device 3000 may receive the character specialization model from the server 2000 or obtain it from another source. The other device 3000 may be a 3rd party device or a general user device. The model generation tool may include a software development kit (SDK) that enables one or more models to be generated for functions required to provide a voice assistant service. The other device 3000 may provide a character-specific model to the device 1000 through the server 2000 or a character-specific model to the device 1000 through another server (not shown).

The device 1000 and other devices 3000 include a smart phone, a tablet PC, a PC, a smart TV, a mobile phone, a personal digital assistant (PDA), a laptop, a media player, a server, a micro server, a global positioning system (GPS) device, E-book terminals, digital broadcasting terminals, navigation, kiosks, MP3 players, digital cameras, home appliances, and other mobile or non-mobile computing devices, but are not limited thereto. However, the present invention is not limited thereto, and the device 1000 and other devices 3000 may include all types of devices that are connected to the server 2000 through a network and capable of transmitting and receiving data to and from each other.

The network includes a local area network (LAN), a wide area network (WAN), a value added network (VAN), a mobile radio communication network, a satellite communication network, and combinations thereof. In addition, each network component shown in Figure 1 is a comprehensive data communication network that enables smooth communication with each other, and may include wired Internet, wireless Internet, and mobile wireless communication networks.

2 is a flowchart of a method in which a device provides a voice assistant service using a character-specific model according to some embodiments.

In operation S200, the device 1000 may receive a character specialization model. The device 1000 may receive a character specialization model generated by another device 3000 from the server 2000. In this case, the device 1000 may display a list of character models provided by the server 2000 and download a character-specific model from the server 2000 based on a user's selection input.

Also, the device 1000 may receive a character-specific model generated by another device 3000 from a predetermined product (not shown). When the device 1000 is close to a product (not shown) including a short-range communication function and a data storage function, the device 1000 may form short-range communication with the product (not shown). In this case, the product (not shown) may be embodied as, for example, a character doll, a character toy, a key ring, a tag product, and an electronic product including a speaker, but is not limited thereto. Also, the device 1000 may receive a character-specific model stored in the product (not shown) from the product (not shown) through short-range communication. Also, link information for downloading a character-specific model may be stored in the product (not shown), and the device 1000 may receive link information stored in the product (not shown) from the product (not shown), and receive the received link information. You can download character-specific models using link information.

In operation S210, the device 1000 may access the character-specific model through the voice assistant program. The device 1000 may execute the voice assistant program installed in the device 1000 and access the character-specific model through the executed voice assistant program. The device 1000 may select one of a plurality of character-specific models installed in the device 1000 based on a user input, and may access the selected character-specific model. The user input for selecting one of the plurality of character-specific models may be a user's voice input. For example, the device 1000 may receive a voice input of a user who generates a call name corresponding to a predetermined character. The voice assistant program may be implemented as an application running on an operating system (OS), or may be implemented as one software module constituting middleware.

In operation S220, the device 1000 may receive a user's voice input. The device 1000 may receive a user's voice input in order to provide a voice assistant service. The operations S210 and S220 may be performed as separate operations, but may also be performed as a seamless operation. For example, in operation S210, a user input for selecting one of the character-specific models, such as “You are now Pororo”, is received, and after selecting one of the character-specific models, in operation S220, “How is the weather today?” As described above, a voice input of a user for requesting a voice assistant service may be received. Also, the operation S210 and the operation S220 are continuously operated continuously, “Pororo, how is the weather today?” When an input for selecting one of the character-specific models and an input for requesting a voice assistant service are received together, the device 1000 analyzes this, selects one of the character-specific models, and provides a service to be provided through the voice assistant program You can specify the contents of.

The contents of the service to be provided through the voice assistant program can be obtained by analyzing the user's speech through NLU (Natural Language Understanding). The device 1000 may acquire the intent and slot information related to the intent in the user's speech, and specify the content of the service. For example, when the utterance “What is the weather today?” Is obtained through the automatic response system (ASR), the device 1000 may identify that the Intent is “weather search” and the Slot is “today weather” through the NLU. You can. NLU is a model trained with a sample sentence and an Intent / Slot that each sample sentence means, and artificial intelligence technology can be applied. When the Intent and Slot are acquired through the NLU, the voice assistant program can be linked with an application that can provide a service corresponding to the acquired Intent. For example, when the acquired intent is "weather search", the voice assistant program may operate in conjunction with a weather application. Intent called 'weather search' and slot called 'today's weather' are converted into API data to enable 'search for weather today and obtain results' according to the API of the weather application, and the API data can be delivered to the weather application . API data may have a different shape for each application, and data conversion to API data may be performed through a separate program that converts Intent / Slot into API data of a specific application.

The weather application can receive API data, retrieve information about today's weather, and pass the search results to the voice assistant program. The search result delivered to the voice assistant program may be API data of the weather application, and the voice assistant program may convert the delivered search result into natural language data. The data conversion operation may be performed by the voice assistant program, but may be performed through a separately prepared program for data conversion. Although an example in which the voice assistant service is provided in connection with the weather application has been described, the voice assistant service may be provided in the same or similar manner to other applications.

In operation S230, the device 1000 may generate a response message using the character-specific model and the voice assistant model. The device 1000 may analyze a user's voice input and generate a response message for the user's voice input using a character-specific model and a voice assistant model.

The device 1000 may use data obtained from an application corresponding to the content of the service to generate a response message. The device 1000 may change the data obtained from the application to data in a form that the voice assistant program can process, and then generate a response message using the changed data. Generation of the response message can be performed by Dialog Management (DM) and Natural Language Generation (NLG), and when the DM determines the type of the response message, the NLG can generate a response message in a form suitable for the type of the response message. . DM may be a model trained using a sample conversation composed of questions and answers, and artificial intelligence technology may be incorporated. The NLG may be a model trained by using data to be included in a response and sample data composed of natural language sentences corresponding thereto, and artificial intelligence technology may be incorporated. For example, the device 1000 may generate a response message using the search result transmitted from the weather application. When the weather application delivers search results (eg, sunny) for today's weather to the voice assistant program, the DM of the voice assistant program may output the result that the type of response to the weather search request is the delivery of the weather search result. In addition, the NLG of the voice assistant program may receive the output result and generate a sentence for delivering the weather search result. As a result of sentence generation, the sentence with the most suitable form for the delivery of the weather search result is determined, and the response sentence (eg, today's weather is sunny) is reflected in the determined sentence by reflecting the search results for today's weather obtained from the weather application. Can be created.

The device 1000 may obtain a response message output from the character-specific model and the model of the voice assistant program by applying the user's voice input or the text converted from the user's voice input to the character-specific model and the voice assistant program model. have. When the voice assistant's NLG generates the sentence “Today's weather is sunny”, the voice assistant's general response message, the character-specific model's NLG can receive the generated sentence. In addition, the character-specific model can change the input sentence to "the sunny weather that is good to go out with friends", which is a response message specialized for the character.

Of course, the device 1000 does not use the voice assistant's NLG, but only the character-specific model's NLG, and can immediately generate a response message specialized for the character, “It is a sunny weather that is good to go outside and play with friends”. In addition, the character NLG model of the voice assistant is trained to reflect the character's characteristics and is changed to a character-specific NLG model, and a character-specific response message can be generated through the voice assistant's NLG model that is changed to reflect the character's characteristics.

In addition, the voice assistant's TTS (Text To Speech) can determine the output characteristics of the response message from the voice assistant's general voice characteristics, and the TTS of the character-specific model can receive this characteristic and determine the output characteristics that reflect the character's characteristics. have. Of course, the TTS of the character-specific model may receive a response message specialized to the character generated by the NLG of the character-specific model, determine the output characteristic reflecting the character characteristic, and output a response message so that the character characteristic is reflected.

In this case, the response message for the user's voice input may reflect, for example, character characteristics, characteristics of the user who prefers the character, and content information related to the character. In addition, the response message for the user's voice input may include an expression according to the tone of the character. In addition, when the response message is output as sound, the response message may include information on intonation so that the response message can be output according to the accent of the character.

The voice assistant service model includes, for example, an ASR model for recognizing a user's input voice, an NLU model for interpreting a user's input voice, a DM model for determining the type of response message, and a response message for generating a response message. An NLG model and a TTS model for determining output characteristics of a response message may be included, but are not limited thereto.

In addition, the character specialization model includes, for example, an ASR model for recognizing a user's input voice, an NLU model for interpreting a user's input voice, a DM model for determining the type of response message, and a response message for generating An NLG model and a TTS model for determining output characteristics of a response message may be included, but are not limited thereto. The character-specific model can be used independently, in parallel or auxiliary to the model of the voice assistant service.

In operation S240, the device 1000 may output the generated response message. The device 1000 may display the text of the response message on the screen of the device 1000. The device 1000 may output the sound of the response message through the speaker of the device 1000.

3 is a flow diagram of a method for a device to download a character-specific model from a server, according to some embodiments.

In operation S300, the device 1000 may access the server 2000 that provides a character-specific model. The device 1000 may connect to the server 2000 that provides a character-specific model by selecting a predetermined object in the execution screen of the voice assistant. In addition, the device 1000 may connect to the server 2000 that provides a character-specific model by selecting a predetermined object in a screen for changing the theme of an operating system (OS) installed in the device 1000.

The server 2000 that provides a character-specific model may be a server that provides a voice assistant service. Further, the device 1000 may access an application providing server that provides an application, such as an app market, to obtain a character-specific model.

In operation S310, the device 1000 may display a list of character-specific models. When a user input for selecting a predetermined object in the execution screen of the voice assistant is received, the device 1000 may display a list of character-specific models provided by the server 2000. Alternatively, when a predetermined object in the screen for changing the theme of the OS (Operating System) installed on the device 1000 is selected, a list of character-specific models may be displayed together with a list of themes of the OS (Operating System). In addition, a character-specific model may be included in the theme of the operating system (OS). The list of character-specific models may be, for example, a list including a character name, but is not limited thereto.

In operation S320, the device 1000 may select a character-specific model. The device 1000 may receive a user input for selecting a predetermined character-specific model from a list of character-specific models, and may select a character-specific model according to the received user input.

In operation S330, the device 1000 may download and install the selected character-specific model. The device 1000 may request the selected character specialization model from the server 2000 and receive the selected character specialization model from the server 2000. In addition, the device 1000 may install the received character-specific model in the device 1000. The device 1000 may install the character-specific model in the device 1000 so that the character-specific model can interoperate with the model of the voice assistant in the device 1000. Details that work in conjunction will be described later.

4 is a flow diagram of a method for downloading a character-specific model by a device approaching a predetermined product according to some embodiments.

In operation S400, the device 1000 may access a product that provides a character-specific model. A product (not shown) that provides a character-specific model may include a device including a short-range communication function and a data storage function. Products (not shown) that provide character-specific models may be implemented as, for example, character dolls, character toys, key chains, tag products, and electronic products including speakers, but are not limited thereto. As the device 1000 approaches within a communication range of a product (not shown) that provides a character-specific model, the device 1000 may form short-range communication with a product (not shown) that provides a character-specific model.

In operation S410, the device 1000 may receive link information for downloading a character-specific model from a product (not shown). A product (not shown) that provides a character-specific model may store link information of a source that provides a character-specific model, and the device 1000 may receive link information through short-range communication formed with the product (not shown). .

In operation S420, the device 1000 may download and install the character-specific model using the received link information. The device 1000 may access the server 2000 that provides the character-specific model using the received link information, and receive the character-specific model provided from the server 2000. In addition, the device 1000 may install the received character-specific model in the device 1000. The device 1000 may install the character-specific model in the device 1000 so that the character-specific model can interoperate with the model of the voice assistant in the device 1000.

On the other hand, in the above, the device 1000 has been described as receiving link information of a character-specific model from a product (not shown) that provides a character-specific model, but is not limited thereto. The device 1000 may directly receive a character-specific model from a product (not shown) that provides a character-specific model. In this case, the device 1000 may update the character-specific model installed on the device through another device 3000 of the 3rd party that generated the character-specific model.

5 is a flow diagram of a method for a device to change the response mode of a voice assistant according to some embodiments.

In operation S500, the device 1000 may receive a user voice input for changing the response mode of the voice assistant. The device 1000 may install a character-specific model and set a call name for calling or executing the installed character-specific model. Also, the device 1000 may receive a voice input of a user who calls a set call name. For example, the device 1000 may set “Pororo” as a call name for calling or executing a character-specific model for the character “Pororo”. When it is determined that a voice input of a user who calls 'pororo' is received, a character-specific model for the pororo character may be called or executed.

In operation S510, the device 1000 may select a character-specific model to be used for the voice assistant based on the user's voice input. The device 1000 may compare a user's voice input with a call name of a character-specific model set in the device 1000. For example, the device 1000 may convert a user's voice input into text, and compare the converted text with a call name of a character-specific model set in the device 1000. The device 1000 may select a character specialization model corresponding to the user's voice input based on the comparison result. In particular, when a plurality of character-specific models are installed, a call name may be used to select one of the plurality of character-specific models. For example, when a voice input of a user who calls “Pororo” is received, the device 1000 may select a character-specific model for Pororo from among a plurality of character-specific models.

In operation S520, the device 1000 may change the response mode of the assistant to the character response mode according to the selected character specialization model. When the character-specific model for Pororo is selected according to the voice input of the user who calls 'Pororo', the device 1000 may change the assistant's response mode to Pororo's response mode.

6 is a flowchart of a method in which the server 2000 provides a model generation tool to another device 3000 according to some embodiments and provides a character-specific model generated by the other device 3000 to the device 1000. .

In operation S600, the server 2000 may store a model generation tool for generating a character-specific model. The model generation tool may be a learning model generation tool for generating a character-specific model. The model generation tool may be, for example, an SDK (Software Develop Kit), and a plurality of functions for generating a character-specific model. Can provide. The model generation tool includes, for example, a function of generating an ASR model for recognizing a user's voice input, a function of generating an NLU model for interpreting a user's voice input, and a DM model for determining the type of response message. It may include, but is not limited to, a function to generate, a function to generate an NLG model to generate a response message, and a function to generate a TTS model to determine a voice characteristic of the response message.

In operation S610, the server 2000 may provide a model generation tool for generating a character-specific model to another device 3000 of the 3rd party. The user of the other device 3000 can generate a model specialized for the character and provide a service related to the character, and the other device 3000 may access the server 2000 and request the server 2000 a model generation tool. .

In operation S620, the server 2000 may receive a character specialization model generated by another device 3000. The other device 3000 may generate a character-specific model using a model generation tool, and upload the generated character-specific model to the server 2000. Character-specific models include, for example, an ASR model for recognizing a user's voice input, an NLU model for interpreting a user's voice input, a DM model for determining the type of response message, and an NLG model for generating a response message , And a TTS model for determining a voice characteristic of the response message, but is not limited thereto. The server 2000 may store the received character specialization model in the DB for each 3rd party.

In operation S630, the server 2000 may provide a list of character-specific models to the device 1000, and in operation S640, the server 2000 may device the character-specific model selected by the device 1000 from the list of character-specific models. (1000).

7 is a flowchart of a method in which another device generates and updates a character-specific model using a model generation tool according to some embodiments.

In operation S700, the other device 3000 may obtain a model generation tool for generating a character-specific model. The other device 3000 may request the model generating tool for generating the character-specific model from the server 2000 and receive the model generating tool from the server 2000. Alternatively, the other device 3000 may receive the model generation tool from another source (not shown) that provides a model generation tool for generating a character-specific model.

In operation S710, the other device 3000 may generate a character-specific model using a model generation tool. The model generation tool includes, for example, a function of generating an ASR model for recognizing a user's voice input, a function of generating an NLU model for interpreting a user's voice input, and a DM model for determining the type of response message. It may include, but is not limited to, a function to generate, a function to generate an NLG model to generate a response message, and a function to generate a TTS model to determine voice characteristics of the response message.

The other device 3000 may generate a character-specific model by training the character-specific model using learning data related to the character using the model generation tool. For example, the other device 3000 uses the tone data of the character, the accent data of the character, the conversation script about the character, the content information related to the character, the QNA data related to the character, and the related information related to the character. ASR model for recognizing voice input, NLU model for interpreting user's voice input, DM model for determining the type of response message, NLG model for generating response message, and for determining voice characteristics of response message By training the TTS model, a character-specific model can be generated.

The character specialization model may include call name information and a call name recognition model for calling a corresponding character specialization model. The call name information and call name recognition model for calling the corresponding character specialization model may be included in the model of the voice assistant service.

In operation S720, the other device 3000 may provide the generated character-specific model to the server 2000. Another device 3000 may register the generated character-specific model with the server 2000.

In operation S730, the other device 3000 may receive response result data using the character-specific model from the device 1000. The character-specific model may be provided to the device 1000 from the server 2000, and the device 1000 may use the character-specific model through a voice assistant application in the device 1000. Also, the device 1000 may provide a voice assistant service based on the character-specific model to the user. In this case, the device 1000 may store the content of conversations exchanged with the user through the voice assistant service and the operation history of the device 1000. In addition, the device 1000 may provide response result data including the conversation content between the user and the device 1000 and the operation history of the device 1000 to other devices 3000 and the server 2000. The device 1000 may provide response result data to other devices 3000 and the server 2000 in the form of big data. For example, the device 1000 may provide response result data to other devices 3000 and the server 2000 except for data related to personal privacy such as a user ID and a phone number.

In operation S740, the other device 3000 may update the character specialization model using the received response result data. The other device 3000 may update the character specialization model by further learning the character specialization model using the received response result data. For example, another device 3000 may include an ASR model for recognizing a user's voice input, an NLU model for interpreting a user's voice input, a DM model for determining the type of response message, and an NLG for generating a response message At least one of a model and a TTS model for determining a voice characteristic of a response message may be retrained using response result data.

8 is a flow diagram of a method for a device, a server, and other devices to provide a voice assistant service according to some embodiments.

In operation S800, the server 2000 may store the model generation tool. The server 2000 may store a model generation tool used to generate a character-specific model. The character-specific model may include a plurality of sub-models for data analysis and generation required to provide a character-specific voice assistant service. In addition, the model generation tool may provide functions necessary to generate a character-specific model including a plurality of sub-models.

In operation S805, the server 2000 may provide a model generation tool to other devices 3000. The server 2000 may provide a service for distributing a model generation tool to other devices 3000. The other device 3000 may access the server 2000 and download a model generation tool.

In operation S810, the other device 3000 may generate the character-specific model using the model generation tool, and in operation S815, the other device 3000 may transmit the character-specific model to the server 2000.

In operation S820, the server 2000 may register the character-specific model. The server 2000 can operate a separate DB (not shown) for managing the character-specific model. Also, the server 2000 may register and manage a character-specific model in a DB (not shown) for each device 3000 or for each character.

In operation S825, the device 1000 may execute the voice assistant application. The voice assistant application may be an assistant program installed on the device by the manufacturer of the device 1000, but is not limited thereto.

In operation S830, the server 2000 may transmit a list of character-specific models to the device 1000. In operation S835, the device 1000 may select a character from the list, and in operation S840, the server 2000 may transmit the device 1000 to the device 1000. ) To the selected character-specific model.

The server 2000 may provide a list of the character-specific models to the device 1000 and transmit the character-specific models selected by the device 1000 from the list of character-specific models to the device 1000. Also, a character-specific model may be included in the theme of the OS of the device 1000, and the server 2000 may provide a list of themes of the OS to the device 1000. In this case, the device 1000 may select a predetermined theme from the theme list, and the server 2000 may provide the device 1000 with a character-specific model in the theme by transmitting the selected theme to the device 1000. .

In operation S845, the device 1000 may change the response mode of the voice assistant. The device 1000 may change the response mode of the voice assistant based on the user input. For example, when a voice input of a user calling a predetermined call name is input, the device 1000 may change the voice assistant's response mode to a response mode of a character corresponding to the call name by recognizing the call name of the voice input. have.

For example, when a call name “pororo” is voice input to the device 1000, the device 1000 may change the response mode of the voice assistant to the pororo response mode. If a character-specific model is provided along with the theme of the OS of the device 1000, the device 1000 may change the theme of the OS to the Pororo theme while changing the response mode of the Voice Assistant to the Pororo response mode.

In operation S850, the device 1000 may access the character-specific model through the voice assistant application. The voice assistant application may provide a voice assistant service by using a character-specific model and a model of the voice assistant together.

In operation S855, the device 1000 may receive a user's voice input. The user may input a voice to the device 1000 to receive a voice assistant service.

In operation S860, the device 1000 may generate a response message using the character-specific model and the voice assistant model. The device 1000 may use a character-specific model and a voice assistant model together to interpret the meaning of the user's voice input, generate a response message to be answered by the user, and determine a voice characteristic of the response message. Also, the device 1000 may determine a response operation of the device 1000 by using a character-specific model and a voice assistant model together.

In operation S865, the device 1000 may output the generated response message. The device 1000 may display a text format response message on the screen of the device 1000. In addition, the device 1000 may output a response message in a sound format through the speaker of the device 1000. Also, the operation of the device 1000 may be controlled according to the determined response operation of the device 1000.

In operation S870, the device 1000 provides the response result data of the character-specific model to the server 2000, and in operation S875, the server 2000 provides the response result data of the character-specific model to the other device 3000. . The response result data may include, for example, data related to a conversation between a user and the device 1000 and a response operation of the device 1000. Alternatively, the device 1000 may transmit the response result data of the character-specific model to the other device 3000.

In operation S880, the other device 3000 may update the character-specific model. The other device 3000 may update the character specialization model by further learning the character specialization model using the received response result data. The other device 3000 may update the character-specific model for each user or device 1000 and manage the updated character-specific model.

Meanwhile, the device 1000 may update the character-specific model in the device 1000 using the response result data of the character-specific model.

9 is a block diagram of a model generation tool in accordance with some embodiments.

Referring to FIG. 9, the model generation tool 90 may include a model generation module 92, an assistant configuration change module 97, and an update module 98.

The model generation module 92 may provide functions for generating a character-specific model. Modules for generating a plurality of sub-models included in the character-specific model may be included. The model generation module 92 may include an NLU model generation module 93, an NLG model generation module 94, and a TTS model generation module 95.

The NLU model generation module 93 may generate a model for interpreting a user's voice input in consideration of character characteristics. The model for interpreting the user's voice input may be a learning model that analyzes the meaning of the user's input voice, or may be a model incorporating an artificial intelligence algorithm. The other device 3000 may input, for example, a dialogue script related to a character, a tone data of a character, intonation data of a character, content information related to the character, and QNA data related to the character into the NLU model generation module 93 , A learning model for interpreting a user's voice input can be generated and trained.

The NLG model generation module 94 may generate a model for generating a response message for a user's voice input. The model for generating a response message for a user's voice input may be a learning model for generating a response message for a user's voice input and determining a response operation of the device 1000. The other device 3000 may input, for example, a dialogue script related to a character, a tone data of a character, intonation data of a character, content information related to the character, and QNA data related to the character into the NLG model generation module 94, A model for generating a response message for a user's voice input can be generated and trained. Further, the other device 3000 inputs user information of the device 1000 and context information related to the device 1000 to the NLG model generation module 94, thereby generating a model for generating a response message for a user's voice input. Can be generated by learning. Accordingly, the model for generating the response message may generate a response message to be provided to the user by reflecting the user's situation, the device 1000's situation, and the characteristics of the character. In this case, the response message may be generated by reflecting the tone of the character.

The TTS model generation module 95 may generate a model for determining voice characteristics of the response message. The model for determining the voice characteristics of the response message may be used to determine the voice characteristics of the response message by reflecting the voice characteristics of the character. The other device 3000 may input, for example, a dialogue script related to a character, a tone data of a character, intonation data of a character, content information related to the character, and QNA data related to the character into the TTS model generation module 95, The model for determining the voice characteristics of the response message by reflecting the voice characteristics of the character can be generated by training. Accordingly, the model for determining the voice characteristics of the response message may cause the sound of the response message to be output through the speaker of the device 1000 according to the intonation and tone of the character.

Meanwhile, the model generation tool 92 may further include an ASR model generation module (not shown) and a DM model generation module (not shown). The ASR model generation module (not shown) may generate an artificial intelligence model for recognizing a user's voice input in consideration of character characteristics. The other device 3000 inputs, for example, a dialogue script related to a character, a tone data of a character, intonation data of a character, content information related to a character, and QNA data related to a character into an ASR model generation module (not shown) By doing so, it is possible to generate and train a learning model for recognizing a user's voice input. The DM model generation module (not shown) may generate an artificial intelligence model for determining the type of response message in consideration of character characteristics. The other device 3000 inputs, for example, a dialogue script related to a character, a tone data of a character, intonation data of a character, content information related to a character, and QNA data related to a character into a DM model generation module (not shown) By doing so, a learning model for determining the type of response message can be generated and trained.

In the above, in the model generation module 92, NLU model generation module 93, NLG model generation module 94, TTS model generation module 95, ASR model generation module (not shown) and DM model generation module (not shown) ) Has been described as operating individually, but is not limited thereto. For example, in the model generation module 92, NLU model generation module 93, NLG model generation module 94, TTS model generation module 95, ASR model generation module (not shown) and DM model generation module A module in which at least two or more (not shown) are integrated may be included.

The assistant setting change module 97 may provide a function for changing a setting value set in the voice assistant. The other device 3000 may use the assistant setting change module 97 to change the call name for calling the character-specific model, setting values related to the response mode of the character-specific model, and the like. The response mode of the character specialization model may include, for example, a conversation mode, a secretary mode, a learning mode, and a play mode, but is not limited thereto. The conversation mode may be a mode that provides a general conversation with the user, the secretary mode may be a mode for managing the user's schedule, the learning mode may be a mode for the user's study, and the play mode may play a game with the user. It may be a mode for providing.

The update module 98 may provide a function for updating the character-specific model. Other devices 98 may update the character-specific model using the update module 98.

10 is a diagram illustrating an example in which a character-specific model is generated using a model generation tool according to some embodiments.

Referring to FIG. 10, another device 3000 may generate a character-specific model 100 by inputting various learning data into the model creation tool 90. For example, the other device 3000 inputs the character's conversation script, character's tone data, character's accent data, character's content information, and character's QNA data into the model creation tool 90, thereby The specialized model 100 can be generated through learning.

The dialogue script associated with the character may be data including the dialogue contents of the character. The dialogue script related to the character may include, for example, the dialogue contents of 'Pororo' in the animation, the dialogue contents of the singer 'Psy' in the broadcast, and the like. The dialog script associated with the character may include, for example, the contents of the conversation in which the device 1000 chats with the user using a character specialization model.

The tone data of the character may be data including information related to the tone of the character, and the accent data of the character may be data including information related to the accent of the character.

Content information related to a character may include image data and audio data related to the character. For example, the content information related to a character includes animation video of 'Pororo', voice data of 'Pororo', news data about 'Pororo', interview voice data of singer Psy, sound source data of singer Psy, News data about the singer 'Psy' may be included, but is not limited thereto.

The QNA data related to the character may include data regarding the QNA exchanged with the customer in the service related to the character.

Also, the other device 3000 may generate the character specialized model 100 through learning by inputting user information of the device 1000 and context information related to the device 1000 into the model creation tool 90. The context information may include, for example, at least one of surrounding environment information of the device 1000, status information of the device 1000, user status information, user's device 1000 usage history information, and user schedule information. However, it is not limited thereto. Ambient environment information of the device 1000 means environmental information within a predetermined radius from the device 1000, and includes, for example, weather information, temperature information, humidity information, illuminance information, noise information, sound information, and the like. However, it is not limited thereto. The status information of the device 1000 includes the mode information of the device 1000 (eg, sound mode, vibration mode, silent mode, power saving mode, blocking mode, multi-window mode, automatic rotation mode, etc.), and the location of the device 1000 Information, time information, activation information of the communication module (eg, Wi-Fi ON / Bluetooth OFF / GPS ON / NFC ON, etc.), network connection status information of the device 1000, application information executed on the device 1000 (eg , Application identification information, application type, application usage time, application usage period, and the like), but is not limited thereto. The user's status information is information about the user's movement, life pattern, and the like, and may include information about the user's walking state, exercising state, driving state, sleeping state, and user mood state, but is not limited thereto. . The user's device 1000 usage history information is information about the user's history using the device 1000, and includes an application execution history, a history of functions executed in the application, a user's call history, and a user's text history. It may include, but is not limited to.

In addition, the other device 3000 may input the assistant setting value through the GUI provided by the assistant setting changing module 97. Accordingly, the character specialization model 100 may include an assistant setting value. The assistant setting value may be provided to the device 1000 together with the character-specific model as data incidental to the character-specific model.

11 is a diagram illustrating an example in which another device generates a character specialization model according to some embodiments.

Referring to FIG. 11, another device 3000 of the developer of the character specialization model may generate a character specialization model of the singer “Psy” using various learning data.

The other device 3000 may set the call name to call the character special model of the singer 'Psy' as 'Psy'. In addition, the other device 3000 may input Psy's broadcast voice data to the model generation tool as a voice example. In addition, the other device 3000 may input Psy's broadcast script as a dialog script to the model generation tool. Further, the other device 3000 may input a Psy video as a model information to the model generation tool. In addition, the other device 3000 may input Psy's Social Network System (SNS) data as model information into the model generation tool.

In addition, the other device 3000 may distribute the generated character-specific model of Psy and register the character-specific model of Psy in the OS theme of the device 1000.

12 is a diagram illustrating an example in which a device according to some embodiments uses a character-specific model and a voice assistant model together.

Referring to FIG. 12, the device 1000 may provide a voice assistant service specialized for a character by interworking with a voice assistant model and a character-specific model.

For example, the device 1000 may link the first NLU model 121 in the model of the voice assistant with the second NLU model 126 in the character specialization model. The first NLU model 121 and the second NLU model 126 may be learning models for interpreting a user's voice input.

 For example, when the user's voice input is recognized and the recognized data is input to the first NLU model 121, data that primarily analyzes the user's voice input may be output from the first NLU model 121. . In addition, by outputting data output from the first NLU model 121 to the second NLU model 126, data that is secondarily interpreted by a user's voice input may be output from the second NLU model 126.

When the character is not selected, data interpreting the user's voice input is output from the first NLU model 121, and when the character is subsequently selected, data interpreted by considering the user's voice input in consideration of the character is the second NLU. It can be output through the model 126.

Further, for example, the device 1000 may link the first NLG model 122 in the voice assistant's learning model with the second NLG model 127 in the character specialization model. The first NLG model 122 and the second NLG model 127 may be learning models for generating a response message to be provided to the user. The first NLG model 122 and the second NLG model 127 may determine a response operation of the device 1000. In particular, data output from the first NLG model 122 may be input to the second NLG model 127 to generate and output a final response message. In addition, when a character is not selected, a response message is generated and output through the first NLG model 122, and when a character is selected, a response message is generated and output through the second NLG model 127.

Also, for example, the device 1000 may link the first TTS model 123 in the model of the voice assistant with the second TTS model 128 in the character specialization model. The first TTS model 123 and the second TTS model 128 may be learning models for determining voice characteristics of the sound of the response message when outputting the sound of the response message. In particular, when the character is not selected, the sound of the response message is output using the first TTS model 123, and when the character is selected, the sound of the response message may be output through the second TTS model 128. In this case, the first TTS model 123 receives a response message output from the first NLG model, and outputs a response message in sound corresponding to the first TTS model 123. In addition, the second TTS model 128 receives a response message output from the second NLG model, and outputs a response message in a sound corresponding to the second TTS model 128.

In FIG. 12, although an example in which the voice assistant model and the character-specific model operate in cooperation has been described, among the voice assistant models, the first NLU model 121, the first NLG model 122, and the first TTS model 123 Changed to the second NLU model 126, the second NLG model 127, and the second TTS model 128 of this character-specific model, it can operate in conjunction with the rest of the voice assistant models. In this case, the second NLU model 126, the second NLG model 127, and the second TTS model 128 of the character-specific model are the first NLU model 121, the first NLG model 122, and the second TTS. The model 123 may be a model generated as a result of re-learning using the learning data for the character. In addition, the second NLU model 126, the second NLG model 127, and the second TTS model 128 of the character-specific model are the first NLU model 121, the first NLG model 122, and the second TTS model. Independently from (123), it may be a separately generated model.

Meanwhile, the voice assistant model of FIG. 12 further includes a first ASR model (not shown) and a first DM model (not shown), and the character-specific model includes a second ASR model (not shown) and a second DM model. (Not shown) may further include. In this case, the user's voice input may be recognized by at least one of the first ASR model (not shown) or the second ASR model (not shown). Further, the type of the response message to the user's voice input may be determined by at least one of the first DM model (not shown) or the second DM model (not shown).

13 is a diagram illustrating an example in which a device provides a voice assistant service using a character-specific model according to some embodiments.

Referring to FIG. 13, when a user of the device 1000 inputs a voice saying “Hello Poroya”, in operation 130, the device 1000 converts the user's voice into text according to the user's voice input and responds to the voice assistant You can change the mode to Pororo mode. For example, the device 1000 may recognize the user's voice input using the ASR model of the voice assistant, and may identify whether the recognized user's voice input includes a call name for changing to the Pororo mode. In addition, when the recognized user's voice input includes a call name for changing to the Pororo mode, the device 1000 may change the response mode of the Voice Assistant to the Pororo mode.

In addition, the device 1000 may analyze the voice input by using the first NLU model of the voice assistant and the second NLU model of the character specialization model together. The device 1000 may input the user's voice input to the first NLU model of the voice assistant and the second NLU model of the character specialization model, and obtain analysis data interpreting the user's voice input as an output value. In this case, the device 1000 may input data in FIG. 10 together in the first NLU model of the voice assistant and the second NLU model of the character specialization model.

In addition, in operation 132, the device 1000 may generate a response message for voice input. The device 1000 may generate a response message for voice input by using the first NLG model of the voice assistant and the second NLG model of the character specialization model together. The device 1000 may input the output value in operation 130 to the first NLG model of the voice assistant and the second NLG model of the character specialization model, and obtain a response message to be provided to the user as an output value. In this case, the device 1000 may input data in FIG. 10 together in the first NLG model of the voice assistant and the second NLG model of the character specialization model. The response message may be generated by reflecting various information related to Pororo and the tone of Pororo.

Meanwhile, the device 1000 may use the first DM model and the second DM model of the character specialization model to generate a response message for voice input. Before generating a response message using the first NLG model of the voice assistant and the second NLG model of the character specialization model, the device 1000 may use the first DM model of the voice assistant and the second DM model of the character specialization model. Use to determine the type of response message. In this case, the device 1000 may input output values of operation 130 to the first DM model of the voice assistant and the second DM model of the character specialization model, and output data indicating the type of response message to be provided to the user Can be obtained by value. In addition, the device 1000 may input data indicating the type of the response message to the first NLG model of the voice assistant and the second NLG model of the character specialization model together with the output value of operation 130.

Also, in operation 134, the device 1000 may generate a sound of a response message. The device 1000 may generate a response message for voice input by using the first TTS model of the voice assistant and the second TTS model of the character specialization model together. The device 1000 may input output values of operation 132 to the first TTS model of the voice assistant and the second TTS model of the character specialization model, and may determine a voice characteristic of the response message to be provided to the user. In this case, the device 1000 may input data in FIG. 10 together in the first NLG model of the voice assistant and the second NLG model of the character specialization model. Also, the device 1000 may generate a sound of a voice message by reflecting the determined voice characteristics.

Further, in operation 136, the device 1000 may display the text of the response message in the message window of the voice assistant, and change the icon representing the voice assistant to the Pororo icon. In addition, the device 1000 may output a sound of a voice message reflecting the voice characteristics of Pororo.

14 is a diagram illustrating detailed operations in which a device provides a voice assistant service according to some embodiments.

Referring to FIG. 14, in operation 140, the voice assistant of the device 1000 may recognize the user's voice by converting the user's voice to text, and refer to the call name of the voice assistant by referring to the call name set in the character specialization model. Can be changed. The voice assistant may use the ASR model to recognize the user's voice input and generate text representing the user's recognized voice.

In operation 141, the voice assistant of the device 1000 may analyze the recognized voice of the user using a character-specific model. By using the NLU model of the device 1000, it is possible to analyze the user's recognized speech. The voice assistant may input text representing the user's recognized voice into the NLU model, and obtain analysis data that interprets the user's recognized voice as an output value. The output analysis data may include, for example, Intent information and Slot information related to Intent information. In this case, the voice assistant may input data in FIG. 10 together in the NLU model.

In operation 142, the voice assistant of the device 1000 may determine the response to the user, and change the response mode of the voice assistant by referring to the setting value in the character-specific model. The voice assistant can determine whether to respond to the user by using the DM model. When the response to the user is determined, the voice assistant can change the response mode to the character response mode. By changing the response mode, the voice assistant can provide a conversation with the user according to the changed response style.

In operation 143, the voice assistant of the device 1000 may determine a response operation of the device 1000 using the character-specific model, and in operation 144, the voice assistant of the device 1000 using the character-specific model, You can create a response message and change the tone of the response message. The voice assistant of the device 1000 may determine a response operation and generate a response message in conjunction with a service related to the character.

The voice assistant may determine a response action using an action planner (AP) model. The AP (Action Planner) model may be an artificial intelligence model for determining an action to respond to a user. The voice assistant may determine a response operation by inputting data output from the NLU model to the AP model. In this case, the voice assistant may determine a response operation by interworking with a predetermined service. The voice assistant may determine a response operation related to the service by inputting information about the service to the AP model together with data output from the NLU model. The AP model may output a sequence of response actions based on INTENT information output from the NLU model. Information about the service may include, for example, information about the type of service, functions provided by the service, information provided by the service, and the like, but is not limited thereto.

In addition, the voice assistant can generate a response message and change the tone of the response message using the NLG model. The response message may be data in text format. The NLG model may be an AI model trained based on a conversation script related to a character, a tone data of a character, content information related to a character, and QNA data related to a character.

Also, in operation 145, the voice assistant of the device 1000 may generate a sound of a response message. The voice assistant can generate a sound of a response message by reflecting the character's accent using a character-specific model. The voice assistant can use the TTS model to change the intonation of the sound to be generated from the response message to the intonation of the character.

15 is a diagram illustrating an example in which a device according to some embodiments downloads a character-specific model from a server.

Referring to FIG. 15, the device 1000 may display a list of character-specific models. For example, the device 1000 may display a list of character-specific models in a GUI for downloading the theme of the OS of the device 1000. Also, the device 1000 may select a character-specific model from a list of character-specific models, and may download the selected character-specific model. For example, when the device 1000 selects 'Pororo Bixby' 152, the device 1000 may download a character-specific model for Pororo. In this case, the device 1000 may download a character-specific model for Pororo along with the OS theme for Pororo.

16 is a diagram illustrating an example in which a device according to some embodiments acquires a character specialization model by approaching a character product.

Referring to FIG. 16, when the device 1000 approaches the Pororo doll 160, the device 1000 may form short-range communication with the Pororo doll 160. Also, the device 1000 may receive a character-specific model from the Pororo doll 160 through short-range communication. The character-specific model may be a character-specific model for Pororo.

In addition, when the user inputs a voice of 'who are you' to the device 1000, the device 1000 may generate response data using a character-specific model, and the generated response data to the Pororo doll 160 Can transmit. The Pororo doll 160 may receive response data and output response data through a speaker of the Pororo doll 160.

17 is a diagram for a device in accordance with some embodiments, showing a voice assistant service dialog.

Referring to FIG. 17, while the voice assistant is in a conversation with the user, when the user inputs the voice 'You are now Pororo' to the device 1000, the voice assistant recognizes 'Pororo' in the user's voice input , You can change the voice assistant's response mode to Pororo mode. Thereafter, the voice assistant can conduct a conversation with the user using Pororo's character-specific model.

18 is a diagram illustrating an example of a response message according to a response mode according to some embodiments.

Referring to FIG. 18, the device 1000 may use a character specialization model corresponding to the response mode according to the response mode of the voice assistant. Further, the device 1000 may generate response messages of different content and different tone according to the response mode of the voice assistant.

19 and 20 are block diagrams of devices in accordance with some embodiments.

As illustrated in FIG. 19, the device 1000 according to some embodiments may include a communication unit 1500, an output unit 1200, a memory 1700, and a processor 1300. However, not all components shown in FIG. 19 are essential components of the device 1000. The device 1000 may be implemented by more components than those illustrated in FIG. 19, or the device 1000 may be implemented by fewer components than those illustrated in FIG. 19.

For example, as illustrated in FIG. 20, the device 1000 according to some embodiments includes a user input unit 1100 in addition to the communication unit 1500, the output unit 1200, the memory 1700, and the processor 1300. , A sensing unit 1400, and an A / V input unit 1600 may be further included.

The user input unit 1100 refers to a means for a user to input data for controlling the device 1000. For example, the user input unit 1100 includes a key pad, a dome switch, and a touch pad (contact capacitive type, pressure resistive film type, infrared sensing type, surface ultrasonic conduction type, integral type) Tension measurement method, piezo effect method, etc.), a jog wheel, a jog switch, and the like, but are not limited thereto.

The user input unit 1100 may receive a user input for a voice assistant service. The user input unit 1100 may receive a user input for executing the voice assistant service and changing the response mode of the voice assistant service.

The output unit 1200 may output an audio signal, a video signal, or a vibration signal, and the output unit 1200 may include a display unit 1210, an audio output unit 1220, and a vibration motor 1230. have.

The display unit 1210 displays and outputs information processed by the device 1000. For example, the display unit 1210 may display a GUI for the voice assistant service.

Meanwhile, when the display unit 1210 and the touch pad are configured as a touch screen by forming a layer structure, the display unit 1210 may be used as an input device in addition to an output device. Depending on the implementation form of the device 1000, the device 1000 may include two or more display units 1210.

The audio output unit 1220 outputs audio data received from the communication unit 1500 or stored in the memory 1700. Further, the sound output unit 1220 outputs sound signals related to functions (for example, call signal reception sound, message reception sound, and notification sound) performed by the device 1000. The sound output unit 1220 may include a speaker, a buzzer, and the like. The audio output unit 1220 may output the sound of the response message of the voice assistant service.

The vibration motor 1230 may output a vibration signal. For example, the vibration motor 1230 may output a vibration signal corresponding to the output of audio data or video data (eg, call signal reception sound, message reception sound, etc.).

The processor 1300 typically controls the overall operation of the device 1000. For example, the processor 1300 executes programs stored in the memory 1700, thereby allowing the user input unit 1100, the output unit 1200, the sensing unit 1400, the communication unit 1500, and the A / V input unit 1600. ) Etc. can be controlled overall. The processor 1300 may control operations of the device 1000 in the present specification by executing programs stored in the memory 1700.

Specifically, the processor 1300 may receive a character specialization model through the communication unit 1500. The processor 1300 may receive a character specialization model generated by another device 3000 from the server 2000. In this case, the processor 1300 may display a list of character-specific models provided by the server 2000 and download the character-specific models from the server 2000 based on a user's selection input.

Also, the processor 1300 may receive a character-specific model generated by another device 3000 from a predetermined product (not shown). When the device 1000 is close to a product (not shown) including a short-range communication function and a data storage function, the device 1000 may form short-range communication with the product (not shown). Also, the processor 1300 may receive a character-specific model stored in the product (not shown) from the product (not shown) through short-range communication. Also, link information for downloading a character-specific model may be stored in the product (not shown), and the device 1000 may receive link information stored in the product (not shown) from the product (not shown), and receive the received link information. You can download character-specific models using link information.

The processor 1300 can access the character-specific model through the voice assistant program. The processor 1300 may execute the voice assistant program installed in the device 1000 and access the character-specific model through the executed voice assistant program. At this time, the voice assistant program may be implemented as an application running on an operating system (OS), or may be implemented as one software module constituting middleware.

The processor 1300 may select one of a plurality of character-specific models installed in the device 1000 based on a user input, and may access the selected character-specific model. The user input for selecting one of a plurality of character-specific models may be a user's voice input including information indicating a character.

A voice assistant program running on a processor or an application operating in conjunction with a voice assistant program can receive a user's voice input for selecting one of character-specific models through the user input unit, and provide a GUI through the output unit to provide this. A user input for selecting one of the character-specific models may be received through the GUI.

The processor 1300 may receive a user's voice input. The processor 1300 may receive a user's voice input to provide a voice assistant service, and may generate a response message for the user's voice input using a character-specific model and a voice assistant model. The voice assistant program operating in the processor 1300 may analyze a user's voice input and generate a response message for the user's voice input using a character-specific model and a voice assistant model.

The processor 1300 applies the text converted from the user's voice input or the user's voice input to the character-specific model and the voice assistant's model through a voice assistant program operating in the processor, thereby allowing the character-specific model and the voice assistant to A response message output from the model can be obtained. In this case, the response message for the user's voice input may be generated in consideration of, for example, the characteristics of the character, the characteristics of the user who prefers the character, and content information related to the character. In addition, the response message for the user's voice input may include an expression according to the tone of the character. In addition, when the response message is output as sound, the response message may include information on intonation so that the response message can be output according to the accent of the character.

The processor 1300 may analyze a user's voice input, select one of character-specific models, and determine the content of a service to be provided through a voice assistant program. The contents of the service to be provided through the voice assistant program can be obtained by analyzing the user's speech through NLU (Natural Language Understanding). The processor 1300 may acquire the intent and the slot information related to the intent in the user's speech to specify the content of the service.

 The processor 1300 may use data obtained from an application corresponding to the content of the service to generate a response message. The processor 1300 may change the data obtained from the application to data in a form that the voice assistant program can process, and then generate a response message using the changed data. Generation of the response message can be performed by Dialog Management (DM) and Natural Language Generation (NLG), and when the DM determines the type of the response message, the NLG can generate a response message in a form suitable for the type of the response message. . Also, the processor 1300 may determine a response operation using the AP model before generating a response message.

The processor 1300 may output a response message generated through the output unit. A voice assistant program or an application operating in conjunction with the voice assistant program may display the text of the response message on the screen of the device 1000. Also, the processor 1300 may output the sound of the response message through the speaker of the device 1000.

Meanwhile, the processor 1300 may receive a user voice input for changing a response mode of the voice assistant through a voice assistant program or an application that works in conjunction with the voice assistant program. The processor 1300 may install a character-specific model through a voice assistant program and set a call name for calling the installed character-specific model. In addition, the processor 1300 may receive a voice input of a user who calls a set call name through a voice assistant program or an application that works in conjunction with the voice assistant program. The processor 1300 may select a character-specific model to be used for the voice assistant based on the user's voice input. The processor 1300 may change the response mode of the assistant to the character response mode according to the selected character specialization model. For example, if a character-specific model for Pororo is selected according to a voice input of a user who calls “Poro,” the device 1000 may change the assistant's response mode to Pororo's response mode.

The sensing unit 1400 may detect a state of the device 1000 or a state around the device 1000 and transmit the sensed information to the processor 1300.

The sensing unit 1400 includes a magnetic sensor 1410, an acceleration sensor 1420, a temperature / humidity sensor 1430, an infrared sensor 1440, a gyroscope sensor 1450, and a position sensor. (Eg, GPS) 1460, a barometric pressure sensor 1470, a proximity sensor 1480, and an RGB sensor (illuminance sensor) 1490, but may include at least one, but is not limited thereto. The function of each sensor can be intuitively inferred by those skilled in the art from the name, and thus detailed description will be omitted.

The communication unit 1500 may include one or more components for communication with the server 2000 and other devices 3000. For example, the communication unit 1500 may include a short-range communication unit 1510, a mobile communication unit 1520, and a broadcast reception unit 1530.

The short-range wireless communication unit 151 includes a Bluetooth communication unit, a Bluetooth Low Energy (BLE) communication unit, a Near Field Communication unit, a WLAN (Wi-Fi) communication unit, a Zigbee communication unit, and an infrared ray ( IrDA, an infrared data association (WDA) communication unit, a WFD (Wi-Fi Direct) communication unit, a UWB (ultra wideband) communication unit, an Ant + communication unit, and the like, but are not limited thereto.

The mobile communication unit 1520 transmits and receives wireless signals to and from at least one of a base station, an external terminal, and a server on a mobile communication network. Here, the wireless signal may include various types of data according to transmission and reception of a voice call signal, a video call signal, or a text / multimedia message.

The broadcast receiving unit 1530 receives a broadcast signal and / or broadcast-related information from the outside through a broadcast channel. The broadcast channel may include a satellite channel and a terrestrial channel. Depending on the implementation example, the device 1000 may not include the broadcast receiving unit 1530.

In addition, the communication unit 1500 may transmit and receive information necessary to provide a voice assistant service specialized for the character with the server 2000 and other devices 3000.

The A / V (Audio / Video) input unit 1600 is for inputting an audio signal or a video signal, which may include a camera 1610 and a microphone 1620. The camera 1610 may obtain a video frame such as a still image or a video through an image sensor in a video call mode or a shooting mode. The image captured through the image sensor may be processed through the processor 1300 or a separate image processing unit (not shown).

The microphone 1620 receives external sound signals and processes them as electrical voice data. For example, the microphone 1620 may receive an acoustic signal from an external device or speaker. The microphone 1620 may use various noise removal algorithms to remove noise generated in the process of receiving an external sound signal.

The memory 1700 may store a program for processing and controlling the processor 1300, and may store data input to or output from the device 1000.

The memory 1700 is a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (for example, SD or XD memory, etc.), RAM (RAM, Random Access Memory) SRAM (Static Random Access Memory), ROM (ROM, Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, magnetic disk , It may include at least one type of storage medium of the optical disk.

Programs stored in the memory 1700 may be classified into a plurality of modules according to their functions. For example, the programs stored in the memory 1700 may be classified into a UI module 1710, a touch screen module 1720, and a notification module 1730. .

The UI module 1710 may provide specialized UIs, GUIs, and the like that interwork with the device 1000 for each application. The touch screen module 1720 may detect a touch gesture on the user's touch screen and transmit information regarding the touch gesture to the processor 1300. The touch screen module 1720 according to some embodiments may recognize and analyze a touch code. The touch screen module 1720 may be configured with separate hardware including a controller.

Various sensors may be provided inside or near the touch screen to sense a touch or proximity touch of the touch screen. In addition, a proximity sensor is an example of a sensor for sensing a touch of a touch screen. The user's touch gesture may include tap, touch & hold, double tap, drag, panning, flick, drag and drop, swipe, and the like.

The notification module 1730 may generate a signal for notifying the occurrence of an event of the device 1000.

21 is a block diagram of a server according to some embodiments.

Referring to FIG. 21, the server 2000 according to some embodiments may include a communication unit 2100, a DB 2200, and a processor 2300.

The communication unit 2100 may include one or more components for communication with the device 1000 and other devices 3000.

The DB 2200 may store a program for processing and control of the processor 2300, and may store data input to or output from the server 2000.

DB 2200 includes flash memory type, hard disk type, multimedia card micro type, card type memory (for example, SD or XD memory, etc.), RAM (RAM, Random Access Memory) SRAM (Static Random Access Memory), ROM (ROM, Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, magnetic disk , It may include at least one type of storage medium of the optical disk, but is not limited thereto.

Programs stored in the DB 2200 may be classified into a plurality of modules according to their functions.

The processor 2300 typically controls the overall operation of the server 2000. For example, the processor 2300 may control the communication unit 2100, the DB 2200, and the like by executing programs stored in the DB 2200. The processor 1300 may control operations of the server 2000 in the present specification by executing programs stored in the DB 2200. The processor 2300 may manage the creation and use of the character-specific model so that the character-specific model can be used in the voice assistant service.

Specifically, the processor 2300 may store a model generation tool for generating a character-specific model. The model generation tool may be, for example, a model development tool (SDK) for generating a character-specific model, and may provide a plurality of functions for generating a character-specific model. The model generation tool includes, for example, a function for generating a learning model for interpreting a user's voice input, a function for generating a learning model for generating a response message for the user's voice input, and voice characteristics of the response message. It may include a function for generating a learning model for determining, but is not limited thereto.

The processor 2300 may provide a model generation tool for generating a character-specific model to another device 3000 of the 3rd party. The user of the other device 3000 may generate a learning model specialized for the character and provide a service related to the character, and the other device 3000 may access the server 2000 and request the server 2000 a model generation tool. have.

The processor 2300 may receive a character specialization model generated by another device 3000. The other device 3000 may generate a character-specific model using a model generation tool, and upload the generated character-specific model to the server 2000. The character specialization model may include, for example, a learning model for interpreting a user's voice input, a learning model for generating a response message for the user's voice input, and a learning model for determining a voice characteristic of the response message. However, it is not limited thereto. For example, the learning model for interpreting a user's voice input may include an ASR model and an NLU model, and the learning model for generating a response message for the user's voice input includes an AP model, a DM model, and an NLG model. A learning model for determining voice characteristics of a response message may include a TTS model, but is not limited thereto. The processor 2300 may store the received character specialization model in the DB for each 3rd party.

The processor 2300 may provide the device 1000 with a list of character-specific models, and the processor 2300 may provide the device 1000 with a character-specific model selected by the device 1000 in the list of character-specific models. Can provide.

Meanwhile, the other device 3000 may be implemented as a component of the device 1000 or a component of the server 2000, and a processor (not shown) in the other device 3000 is another device 3000 in the present specification. You can control the operation.

22 is a block diagram of a processor in accordance with some embodiments.

For convenience of description, the processor 1300 of FIG. 22 is described as being the processor 1300 in the device 1000, but is not limited thereto. The operation of the processor 1300 in FIG. 22 may be implemented by a processor 2300 in the server 2000 or a processor (not shown) in another device 3000.

Referring to FIG. 22, the processor 1300 according to some embodiments may include a data learning unit 1310 and a data recognition unit 1320.

The data learning unit 1310 may learn criteria for providing a voice assistant service specialized for the character. The data learning unit 1310 may learn criteria on which data to use to provide a voice assistant service specialized for the character, and how to provide a voice assistant service specialized for the character using the data. The data learning unit 1310 may learn criteria for interpreting a user's voice input, generating a response message, and outputting a response message. The data learning unit 1310 acquires data to be used for learning, and applies the obtained data to a character-specific model, which will be described later, to learn criteria for providing a voice assistant service specialized for the character.

The data recognition unit 1320 may provide output data for providing a voice assistant service specialized for the character based on the input data. The data recognition unit 1320 may analyze a user's voice input from predetermined data, generate a response message, and determine an accent of the response message using the learned character specialization model. The data recognition unit 1320 provides a voice assistant service specialized for a character based on the predetermined data by acquiring predetermined data according to a preset criterion by learning and using the character-specific model using the obtained data as an input value It can provide the output data to do. In addition, the result value output by the character-specific model using the obtained data as an input value may be used to update the character-specific model. The character specialization model may include, for example, a learning model for interpreting a user's voice input, a learning model for generating a response message for the user's voice input, and a learning model for determining a voice characteristic of the response message. However, it is not limited thereto. For example, the learning model for interpreting a user's voice input may include an ASR model and an NLU model, and the learning model for generating a response message for the user's voice input includes an AP model, a DM model, and an NLG model. A learning model for determining voice characteristics of a response message may include a TTS model, but is not limited thereto.

At least one of the data learning unit 1310 and the data recognition unit 1320 may be manufactured in the form of at least one hardware chip and mounted on an electronic device. For example, at least one of the data learning unit 1310 and the data recognition unit 1320 may be manufactured in the form of a dedicated hardware chip for artificial intelligence (AI), or an existing general-purpose processor (for example, a CPU) Alternatively, it may be manufactured as a part of an application processor or a graphics-only processor (for example, a GPU) and mounted on various electronic devices described above.

In this case, the data learning unit 1310 and the data recognition unit 1320 may be mounted on one electronic device or may be mounted on separate electronic devices. For example, one of the data learning unit 1310 and the data recognition unit 1320 may be included in the electronic device, and the other may be included in the server. In addition, the data learning unit 1310 and the data recognition unit 1320 may provide the model information constructed by the data learning unit 1310 to the data recognition unit 1320 through wired or wireless communication. 1320) may be provided to the data learning unit 1310 as additional learning data.

Meanwhile, at least one of the data learning unit 1310 and the data recognition unit 1320 may be implemented as a software module. When at least one of the data learning unit 1310 and the data recognition unit 1320 is implemented as a software module (or a program module including an instruction), the software module is a computer-readable, non-transitory readable It may be stored in a readable media (non-transitory computer readable media). Also, in this case, at least one software module may be provided by an operating system (OS) or may be provided by a predetermined application. Alternatively, some of the at least one software module may be provided by an operating system (OS), and the other may be provided by a predetermined application.

23 is a block diagram of a data learning unit according to some embodiments.

Referring to FIG. 23, the data learning unit 1310 according to some embodiments includes a data acquisition unit 1310-1, a pre-processing unit 1310-2, a learning data selection unit 1310-3, and a model learning unit 1310. -4) and a model evaluation unit 1310-5.

The data acquisition unit 1310-1 may acquire data necessary to provide a voice assistant service specialized for the character. The data acquisition unit 1310-1 may acquire data necessary for learning to provide a voice assistant service specialized for the character. For example, the data acquisition unit 1310-1 may acquire the data of FIG. 10, but is not limited thereto.

The pre-processing unit 1310-2 may pre-process the acquired data so that the acquired data can be used for learning to provide a voice assistant service specialized for the character. The pre-processing unit 1310-2 processes the acquired data in a preset format so that the model learning unit 1310-4, which will be described later, can use the acquired data for learning to provide a character-specific voice assistant service. can do.

The learning data selector 1310-3 may select data necessary for learning from pre-processed data. The selected data may be provided to the model learning unit 1310-4. The learning data selector 1310-3 may select data necessary for learning from pre-processed data according to preset criteria. Further, the learning data selector 1310-3 may select data according to a preset criterion by learning by the model learning unit 1310-4 described later.

The model learning unit 1310-4 may learn criteria for providing a voice assistant service specialized for the character based on the learning data. In addition, the model learning unit 1310-4 may also learn the criteria for what training data to use.

In addition, the model learning unit 1310-4 may train a character-specific model used to provide a voice assistant service specialized for the character. In this case, the character-specific model may be a pre-built model. For example, the character-specific model may be a pre-built model that receives basic learning data.

Alternatively, the model learning unit 1310-4 may generate a character-specific model and train the generated character-specific model. Further, the character-specific model may include a plurality of learning models. For example, the character specialization model may include a learning model for interpreting a user's input voice, a learning model for generating a response message to be provided to the user, and a learning model for determining an output characteristic of the response message. , But is not limited to this.

The character-specific model may be constructed in consideration of the application field of the recognition model, the purpose of learning, or the computer performance of the device. The character-specific model may be, for example, a model based on a neural network. For example, a model such as a deep neural network (DNN), a recurrent neural network (RNN), or a bidirectional recurrent deep neural network (BRDNN) may be used as a character-specific model, but is not limited thereto.

According to various embodiments, the model learning unit 1310-4 is a character-specific model for learning a character-specific model having a high relationship between input learning data and basic learning data when a plurality of pre-built character-specific models exist. Can decide. In this case, the basic learning data may be pre-classified for each data type, and the character specialization model may be pre-built for each data type. For example, the basic training data is classified into various criteria such as the region where the training data is generated, the time when the training data is generated, the size of the training data, the genre of the training data, the creator of the training data, and the type of object in the training data. It may be.

In addition, the model learning unit 1310-4 may train a character-specific model using, for example, a learning algorithm including an error back-propagation or a gradient descent method. .

In addition, the model learning unit 1310-4 may train a character-specific model, for example, through supervised learning using learning data as an input value. In addition, the model learning unit 1310-4, for example, through unsupervised learning to discover the criteria for determining the situation by learning the type of data necessary for situation determination without much guidance, for example, You can train character-specific models. In addition, the model learning unit 1310-4 may, for example, train a character-specific model through reinforcement learning using feedback on whether a result of situation determination according to learning is correct.

In addition, when the character-specific model is learned, the model learning unit 1310-4 may store the learned character-specific model. In this case, the model learning unit 1310-4 may store the trained character-specific model in the memory of the electronic device including the data recognition unit 1320. Alternatively, the model learning unit 1310-4 may store the learned character-specific model in the memory of the electronic device including the data recognition unit 1320 to be described later. Alternatively, the model learning unit 1310-4 may store the trained character-specific model in the memory of a server connected to the electronic device through a wired or wireless network.

In this case, the memory in which the learned character specialization model is stored may store, for example, instructions or data related to at least one other component of the electronic device. Also, the memory may store software and / or programs. The program may include, for example, a kernel, middleware, application programming interface (API) and / or application program (or "application"), and the like.

The model evaluation unit 1310-5 may input evaluation data into the character-specific model and, if the recognition result output from the evaluation data does not satisfy a predetermined criterion, may cause the model learning unit 1310-4 to learn again. have. In this case, the evaluation data may be preset data for evaluating the character-specific model.

For example, the model evaluator 1310-5, among the recognition results of the learned character-specific model for the evaluation data, sets a predetermined criterion when the number or ratio of the evaluation data whose recognition result is not accurate exceeds a preset threshold. It can be evaluated as unsatisfactory. For example, when a predetermined criterion is defined as a ratio of 2%, when the trained character-specific model outputs an incorrect recognition result with respect to evaluation data exceeding 20 out of a total of 1000 evaluation data, the model evaluation unit 1310-5 Can evaluate that the trained character-specific model is not suitable.

On the other hand, when there are a plurality of learned character specialization models, the model evaluator 1310-5 evaluates whether or not the predetermined criteria are satisfied for each trained character specialization model, and the final character specialization of the model that satisfies the predetermined criteria. You can decide as a model. In this case, when there are a plurality of models satisfying a predetermined criterion, the model evaluator 1310-5 may determine any one or a predetermined number of models preset in order of highest evaluation score as the final character-specific model.

Meanwhile, the data acquisition unit 1310-1, the pre-processing unit 1310-2, the learning data selection unit 1310-3, the model learning unit 1310-4, and the model evaluation unit 1310 in the data learning unit 1310 At least one of -5) may be manufactured in the form of at least one hardware chip and mounted on an electronic device. For example, at least one of the data acquisition unit 1310-1, the pre-processing unit 1310-2, the training data selection unit 1310-3, the model learning unit 1310-4, and the model evaluation unit 1310-5. One may be manufactured in the form of a dedicated hardware chip for artificial intelligence (AI), or as part of an existing general-purpose processor (eg, CPU or application processor) or graphics-only processor (eg, GPU). It may be mounted on various electronic devices.

In addition, the data acquisition unit 1310-1, the pre-processing unit 1310-2, the learning data selection unit 1310-3, the model learning unit 1310-4 and the model evaluation unit 1310-5 are one electronic It may be mounted on the device, or may be mounted on separate electronic devices, respectively. For example, some of the data acquisition unit 1310-1, the pre-processing unit 1310-2, the training data selection unit 1310-3, the model learning unit 1310-4, and the model evaluation unit 1310-5 Is included in the electronic device, and the other part may be included in the server.

In addition, at least one of the data acquisition unit 1310-1, the pre-processing unit 1310-2, the learning data selection unit 1310-3, the model learning unit 1310-4, and the model evaluation unit 1310-5 It can be implemented as a software module. At least one of the data acquisition unit 1310-1, the pre-processing unit 1310-2, the learning data selection unit 1310-3, the model learning unit 1310-4, and the model evaluation unit 1310-5 is a software module. When implemented as (or a program module including an instruction), the software module may be stored in a computer-readable non-transitory computer readable media. Also, in this case, at least one software module may be provided by an operating system (OS) or may be provided by a predetermined application. Alternatively, some of the at least one software module may be provided by an operating system (OS), and the other may be provided by a predetermined application.

24 is a block diagram of a data recognition unit according to some embodiments.

Referring to FIG. 24, the data recognition unit 1320 according to some embodiments includes a data acquisition unit 1320-1, a pre-processing unit 1320-2, a recognition data selection unit 1320-3, and a recognition result providing unit ( 1320-4) and a model update unit 1320-5.

The data acquisition unit 1320-1 may acquire data necessary to provide a voice assistant service specialized for the character, and the pre-processing unit 1320-2 may preprocess the acquired data. The pre-processing unit 1320-2 may process the acquired data in a preset format so that the recognition result providing unit 1320-4, which will be described later, can use the acquired data to provide a voice assistant service specialized for the character. . The data acquisition unit 1320-1 may acquire data in FIG. 10, for example, but is not limited thereto.

The recognition data selection unit 1320-3 may select necessary data from pre-processed data. The selected data may be provided to the recognition result providing unit 1320-4. The recognition data selection unit 1320-3 may select some or all of the pre-processed data according to a preset criterion. Further, the recognition data selection unit 1320-3 may select data according to a preset criterion by learning by the model learning unit 1310-4, which will be described later.

The recognition result providing unit 1320-4 may apply the selected data to the character-specific model and output data for providing a voice assistant service specialized for the character. The recognition result providing unit 1320-4 may provide output data according to the recognition purpose of the data. The recognition result providing unit 1320-4 may apply the selected data to the character-specific model by using the data selected by the recognition data selection unit 1320-3 as an input value. Also, the recognition result may be determined by a character-specific model.

For example, the character specialization model may include a learning model for interpreting a user's input voice, a learning model for generating a response message to be provided to the user, and a learning model for determining output characteristics of the response message. . In addition, for example, the character specialization model may output a result of the analysis of the user's voice input, a response message to be provided to the user, and a voice characteristic value of the response message.

The model updating unit 1320-5 may allow the character-specific model to be updated based on the evaluation of the recognition result provided by the recognition result providing unit 1320-4. For example, the model updating unit 1320-5 provides the model learning unit 1310-4 with the recognition result provided by the recognition result providing unit 1320-4, so that the model learning unit 1310-4 Character-specific models can be updated.

Meanwhile, the data acquisition unit 1320-1, the pre-processing unit 1320-2, the recognition data selection unit 1320-3, the recognition result providing unit 1320-4, and the model update unit in the data recognition unit 1320 ( At least one of 1320-5) may be manufactured in the form of at least one hardware chip and mounted on an electronic device. For example, among the data acquisition unit 1320-1, preprocessing unit 1320-2, recognition data selection unit 1320-3, recognition result providing unit 1320-4, and model update unit 1320-5 At least one may be manufactured in the form of a dedicated hardware chip for artificial intelligence (AI), or it may be manufactured as part of an existing general purpose processor (for example, CPU or application processor) or graphics only processor (for example, GPU), It may be mounted on various electronic devices.

In addition, the data acquisition unit 1320-1, the pre-processing unit 1320-2, the recognition data selection unit 1320-3, the recognition result providing unit 1320-4 and the model update unit 1320-5 are one It may be mounted on an electronic device, or may be mounted on separate electronic devices, respectively. For example, among the data acquisition unit 1320-1, preprocessing unit 1320-2, recognition data selection unit 1320-3, recognition result providing unit 1320-4, and model update unit 1320-5 Some may be included in the electronic device, and others may be included in the server.

In addition, at least one of the data acquisition unit 1320-1, the pre-processing unit 1320-2, the recognition data selection unit 1320-3, the recognition result providing unit 1320-4 and the model update unit 1320-5 Can be implemented as a software module. At least one of the data acquisition unit 1320-1, the pre-processing unit 1320-2, the recognition data selection unit 1320-3, the recognition result providing unit 1320-4, and the model update unit 1320-5 is software. When implemented as a module (or a program module including an instruction), the software module may be stored in a computer-readable non-transitory computer readable media. Also, in this case, at least one software module may be provided by an operating system (OS) or may be provided by a predetermined application. Alternatively, some of the at least one software module may be provided by an operating system (OS), and the other may be provided by a predetermined application.

25 is a diagram illustrating an example of learning and recognizing data by interworking a device and a server according to some embodiments.

Referring to FIG. 25, the server 2000 may learn a criterion for providing a character-specific voice assistant service, and the device 1000 may use a character-specific voice assistant service based on the learning result by the server 2000 Can provide.

In this case, the model learning unit 2340 of the server 2000 may perform the function of the data learning unit 1310 illustrated in FIG. 23. The model learning unit 2340 of the server 2000 may learn criteria for providing a voice assistant service specialized for the character. The model learning unit 2340 may provide a voice assistant service specialized for a character by acquiring data to be used for training and applying the acquired data to a character-specific model to be described later.

In addition, the recognition result providing unit 1320-4 of the device 1000 may determine the situation by applying the data selected by the recognition data selection unit 1320-3 to the character-specific model generated by the server 2000. have. For example, the recognition result providing unit 1320-4 transmits data selected by the recognition data selection unit 1320-3 to the server 2000, and the server 2000 recognizes the data selection unit 1320-3. By applying the data selected by to the recognition model, it is possible to request to provide a voice assistant service specialized for the character. In addition, the recognition result providing unit 1320-4 may receive a voice assistant service specialized for the character from the server 2000.

Alternatively, the recognition result providing unit 1320-4 of the device 1000 receives the recognition model generated by the server 2000 from the server 2000, and uses the received recognition model to provide a voice assistant service specialized for the character. Can provide. In this case, the recognition result providing unit 1320-4 of the device 1000 may apply the data selected by the recognition data selection unit 1320-3 to the character-specific model received from the server 2000.

On the other hand, in FIG. 25, it has been described that the device 1000 and the server 2000 provide a voice assistant service specialized for a character by interworking, but is not limited thereto. When the device 1000, the server 2000, and other devices 3000 work together, a voice assistant service specialized for a character may be provided. In this case, the server 2000 manages a general learning model for the voice assistant service, and the other device 3000 manages a character-specific model.

Some embodiments may also be embodied in the form of a recording medium comprising instructions executable by a computer, such as program modules, being executed by a computer. Computer readable media can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media. In addition, computer readable media may include computer storage media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.

In addition, in this specification, “part” may be a hardware component such as a processor or circuit, and / or a software component executed by a hardware component such as a processor.

The foregoing description of the present disclosure is for illustration only, and those skilled in the art to which the present disclosure pertains can understand that the present invention can be easily modified into other specific forms without changing the technical spirit or essential features of the present disclosure. will be. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as distributed may be implemented in a combined form.

The scope of the present disclosure is indicated by the following claims rather than the above detailed description, and it should be interpreted that all changes or modifications derived from the meaning and scope of the claims and equivalent concepts thereof are included in the scope of the present disclosure. do.

Claims (20)

A method for a device to provide a voice assistant service through a voice assistant program,
Receiving a character specialization model from an external device;
Receiving a user voice input including a word indicating a character and a sentence requesting a response from the voice assistant program;
Determining the character specialization model according to the word indicating the character;
Generating a response message for the sentence requesting a response from the voice assistant program using the character-specific model; And
Outputting the generated response message; How to include. According to claim 1,
The character specialization model includes a first natural language understanding (NLU) model for speech analysis, a first DM (Dialog Management) model for determining the type of response message, and a first NLG (Natural) for generating the response message Language Generation) method or a first text to speech (TTS) model for determining an output characteristic of the response message. According to claim 1,
The operation of generating the response message is to generate the response message by using the model of the voice assistant program together with the character-specific model. According to claim 3,
The model of the voice assistant program includes a second natural language understanding (NLU) model for speech analysis, a second DM (Dialog Management) model for determining the type of the response message, and a second NLG for generating the response message (Natural Language Generation) model and at least one of a second text to speech (TTS) model for determining output characteristics of the response message. According to claim 2,
The character-specific model is learned by the other device using a model generation tool provided to another device, and the character-specific model is an artificial intelligence algorithm, and is at least one of machine learning, neural networks, genes, deep learning, and classification algorithms. It is a learning model learned using one, the method. The method of claim 5,
The model generation tool provided to the other device is used by the other device to generate at least one of the first NLU model, the first NLG model, and the first TTS model. According to claim 4,
The operation of generating a response message for the voice input includes: using the first NLU model and the second NLU model together; Using the first NLG model and the second NLG model together; Or using the first TTS model and the second TTS model together; The method comprising at least one of. According to claim 1,
Accessing a server that provides a list of character-specific models;
Displaying a list of the character-specific models; And
Selecting the character-specific model from the list of the character-specific models;
Further comprising,
The operation of receiving the character-specific model is to receive the selected character-specific model from the server. According to claim 1,
Forming a short-range communication with a product providing link information for downloading the character-specific model; And
Downloading link information for downloading the character-specific model from the product;
Further comprising,
The operation of receiving the character specialization model is to download the character specialization model using the link information. According to claim 1,
Accessing the received character specialization model through the voice assistant program;
The method, which further comprises. In the device providing a voice assistant service,
A microphone receiving a user voice input;
Communication interface;
A memory storing one or more instructions; And
One or more processors to execute a voice assistant program by executing the one or more instructions; It includes,
The processor, through the communication interface, receives a character-specific model to interwork with the voice assistant program from a server, and includes a word indicating a character corresponding to the character-specific model and a sentence requesting a response from the voice assistant program The user's voice input is received through the microphone, and a response message for the sentence requesting a response from the voice assistant program is generated using the character-specific model interworking with the voice assistant program, and the generated response A device that outputs a message. The method of claim 11,
The character specialization model may include a first NLU model for speech analysis, a first DM model for determining the type of the response message, a first NLG model for generating the response message, or an output characteristic of the response message The device comprising at least one of the first TTS model for. The method of claim 11,
And the processor generates the response message by executing the one or more instructions, using the model of the voice assistant program together with the character-specific model. The method of claim 13,
The model of the voice assistant program includes a second NLU model for speech analysis, a second DM model for determining the type of the response message, a second NLG model for generating the response message, and output characteristics of the response message. And at least one of the second TTS models for determining. The method of claim 12,
The character-specific model is learned by the other device using a model generation tool provided to another device, and the character-specific model is an artificial intelligence algorithm, and is at least one of machine learning, neural networks, genes, deep learning, and classification algorithms. A device that is a learning model learned using one. The method of claim 15,
The model generating tool provided to the other device is used by the other device to generate at least one of the first NLU model, the first NLG model, and the first TTS model. The method of claim 14,
The processor using the first NLU model and the second NLU model together by executing the one or more instructions; Using the first NLG model and the second NLG model together; Or using the first TTS model and the second TTS model together; Device. The method of claim 11,
The processor, by executing the one or more instructions, accesses a server providing a list of character-specific models, displays a list of the character-specific models, selects the character-specific model from the list of character-specific models, and And receiving the selected character-specific model from the server. The method of claim 11,
The processor forms a short-range communication with a product that provides link information for downloading the character-specific model by executing the one or more instructions, downloads link information for downloading the character-specific model from the product, and the A device that downloads the character-specific model using link information. In the server providing the voice assistant service,
Communication interface;
A memory storing one or more instructions; And
And one or more processors executing the one or more instructions.
The processor, through the communication interface, obtains a character-specific model to interwork with the voice assistant program of the device, and includes a word indicating a character corresponding to the character-specific model and a sentence requesting a response from the voice assistant program When a user voice input is received from the device through the communication interface, the server generates a response message for a sentence requesting a response from the voice assistant program by using the character-specific model interworking with the voice assistant program.

Images