RU2792288C1 - Electronic device and its control method - Google Patents

Abstract

FIELD: voice technology.

SUBSTANCE: invention relates to voice input means. An electronic device for performing an application operation using a natural language understanding model includes: a storage device for storing at least one instruction and a processor for executing at least one instruction for receiving usage information for an application installed in an electronic device, receiving a natural language understanding model from a variety of natural language understanding models corresponding to the application, based on usage information, performing natural language understanding for user voice input, associated with the application, based on the fact that the natural language understanding model corresponds to the application, and performing the application operation based on a pre-formed understanding of the natural language. At the same time, the natural language understanding model corresponding to the application is received from an external server based on the number of application performances included in the usage information, or the execution time of the application.

EFFECT: increase in the efficiency of performing operations of applications of an electronic device based on voice input.

13 cl, 9 dwg, 3 tbl

Description

The field of technology to which the invention belongs

[1] The invention relates to an electronic device for obtaining a natural language understanding model and to a method for controlling it.

State of the art

[2] The natural language understanding model is a model that analyzes the user's voice and finds out the user's intent included in the user's voice. Such a natural language understanding model may exist separately for each of the various applications.

[3] However, in order to maintain the natural language understanding model for all applications installed on an electronic device, such as a smartphone, it may be necessary to increase the storage capacity of the electronic device.

[4] In addition, when the natural language understanding model is received from the server and used as needed, some time must pass for the natural language understanding model to be received. In addition, the natural language understanding model is not adopted in a situation in which there is no network connection.

Disclosure of invention

Technical task

[5] The invention provides an electronic device for obtaining a natural language understanding model corresponding to an operation of an application frequently used by a user, and a method for controlling it.

[6] Additional aspects of the embodiments are set forth in part in the description below and in part are obvious from the description or may be ascertained by practice of the presented embodiments.

The solution of the problem

[7] According to an aspect of the invention, an electronic device includes: a storage device configured to store at least one instruction; and a processor configured to execute at least one instruction for: obtaining usage information for an application installed in the electronic device, obtaining a natural language comprehension model from a plurality of natural language comprehension models corresponding to the application, based on the usage information, performing natural language comprehension for user voice input associated with the application based on the natural language understanding model corresponding to the application, and performing an application operation based on the pre-generated natural language understanding.

[8] In addition, the electronic device further comprises a communication interface, wherein the processor is further configured to execute at least one instruction for: obtaining usage information including at least one of the number of application executions and application execution time, control a communication interface for transmitting the received usage information to the external server, receiving the natural language understanding model corresponding to the application from the plurality of natural language understanding models corresponding to the plurality of applications from the external server via the communication interface, and storing the natural language understanding model in a storage device.

[9] In addition, wherein the natural language understanding model corresponding to the application is received from the external server based on that the number of executions of the application included in the usage information is equal to or greater than the threshold number, or on the basis that the execution time of the application is or more than the threshold time.

[10] In addition, the application includes resource information that matches information about a given voice utterance and operation information corresponding to a given voice utterance; and the processor is further configured to execute, based on that the usage information for the application satisfies the predetermined condition, at least one instruction for generating a natural language understanding model corresponding to the application according to the information regarding the predetermined voice utterance and the operation information corresponding to the predetermined voice utterance. .

[11] In addition, the processor is further configured to execute at least one instruction to request a first user voice input corresponding to an application operation that is determined, based on the usage information, to be executed a number of times greater than or equal to a threshold number of times, or for a time greater than or equal to the threshold time.

[12] In addition, the processor is further configured to execute at least one instruction for:

- determining whether the first user voice input conflicts with the user voice input stored in the storage device, based on the first user voice input entered according to the request; requesting a second user voice input different from the first user voice input based on determining that the first user voice input conflicts with the stored user voice input; and storing in the storage device the first user voice input according to the application operation based on the determination that the first user voice input does not conflict with the stored user voice input.

[13] In addition, the processor is further configured to execute at least one instruction to request from an external server a natural language understanding model corresponding to the application based on the fact that the application is installed in the electronic device.

[14] Wherein, the natural language understanding model includes at least one of a personal model generated based on the user's voice utterance using an electronic device and a public model generated based on the public user's voice utterance; and the processor is further configured to execute at least one instruction for: performing natural language comprehension for the user's voice input in the personal model, performing natural language comprehension for the user's voice input in the public model, based on that the natural language comprehension for the user's voice input in the personal model was not successful, and transmitting the user voice input to an external server to perform natural language comprehension for the user voice input based on the fact that natural language understanding for the user voice input in the public model was not successful.

[15] In addition, the processor is further configured to execute at least one instruction to delete the obtained natural language understanding model based on the fact that the user voice input is not entered a number of times greater than or equal to a threshold number of times, or for a time greater than or equal to the threshold time.

[16] According to another aspect of the invention, a method for controlling an electronic device includes: obtaining usage information for an application installed in the electronic device; deriving, based on the usage information, a natural language understanding model from a plurality of natural language understanding models corresponding to the application; performing natural language understanding for the user voice input associated with the application based on the natural language understanding model corresponding to the application; and performing an application operation based on the natural language understanding performed.

[17] In accordance with another aspect of the invention, a persistent computer-readable recording medium has instructions recorded thereon, executable by at least one processor for implementing a method for controlling an electronic device, the method of controlling including: obtaining usage information for an application installed in the electronic device ; deriving, based on the usage information, a natural language understanding model from a plurality of natural language understanding models corresponding to the application; performing natural language understanding for the user voice input associated with the application based on the natural language understanding model corresponding to the application; and performing an application operation based on the natural language understanding performed.

Benefits of the Invention

[18] According to various embodiments, when a natural language understanding model is obtained on the electronic device 100, an operation corresponding to the user's voice may be performed by the obtained natural language understanding model.

Brief description of the drawings

[19] The above and other exemplary aspects, features and advantages of particular embodiments of the present invention will become more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which:

[20] FIG. 1 is a diagram for describing the configuration of an electronic system according to the embodiment;

[21] FIG. 2 is a diagram illustrating a flowchart of a process for obtaining a natural language comprehension model for performing natural language comprehension according to an embodiment;

[22] FIG. 3A is a flowchart of a case in which a natural language understanding model is received from a server according to an embodiment;

[23] FIG. 3B is a diagram for describing a process in which a natural language understanding model is received from a server according to the embodiment;

[24] FIG. 4A is a flowchart of a case in which a natural language understanding model is generated according to another embodiment;

[25] FIG. 4B is a diagram for describing a process in which a natural language understanding model is generated by an electronic device according to another embodiment;

[26] FIG. 5 is a block diagram illustrating an electronic device and a server according to an embodiment;

[27] FIG. 6 is a view for describing the sequence in which the natural language understanding model operates according to the embodiment; And

[28] FIG. 7 is a flowchart for describing an example in which an electronic device and a server work together to perform an application function according to a user's voice, according to an embodiment.

Implementation of the invention

[29] The invention will now be described in detail with reference to the accompanying drawings.

[30] FIG. 1 is a diagram for describing the configuration of the electronic system 1000 according to the embodiment.

[31] Referring to FIG. 1, an electronic system 1000 according to an embodiment includes an electronic device 100 and a server 200.

[32] The electronic device 100 may be implemented or configured to perform natural language comprehension of the user's voice using an artificial intelligence based model (or a neural network model, a trained or trained network model, etc.) and perform an application function based on the result of the natural language understanding being performed. Here, understanding natural language means finding out or understanding the intent of the user included in natural language, which is usually used without following only the rules in which phraseology is specified, as opposed to an artificial language such as a programming language. An AI-based model performing such natural language understanding is collectively referred to as a "natural language understanding model".

[33] The artificial intelligence related functions of an embodiment operate through a processor (eg, at least one processor) and a storage device (eg, at least one storage device). The processor may be configured as one or multiple processors. At this time, one or more processors may be a general purpose processor such as a central processing unit (CPU), an application processor (AP), a digital signal processor (DSP), a dedicated graphics processor such as a graphics processing unit (GPU), or a machine vision processor (VPU) or a specialized artificial intelligence processor such as a neural processor (NPU). One or a plurality of processors are controlled to process input data according to predetermined rules of operation or artificial intelligence models stored in a storage device. Alternatively, when one or more of the processors is a dedicated AI processor, the dedicated AI processor may be designed in a hardware architecture specifically designed to process a particular AI model.

[34] Predefined rules of operation or models based on artificial intelligence are distinguished by creation (or updating) through learning (or training). Here, creation by training means that the base model based on artificial intelligence is trained (or trained) by using a set of training data (or training data) by a training algorithm (or training algorithm), thereby creating a predetermined set of operation rules or models. based on artificial intelligence in order to fulfill the required characteristic (or goal). Such training may be performed within the device itself in which the artificial intelligence according to the embodiment is performed, or may also be performed via a separate server and/or system. An example of the learning algorithm includes supervised learning, unsupervised learning, semi-supervised learning, or stimulated learning, but the learning algorithm is not limited to the examples described above.

[35] An AI-based model may include a plurality of neural network layers. Each of the plurality of neural network layers has a plurality of weight values and performs a neural network operation through an operation between the operation result of the previous layer and the plurality of weight coefficients. The plurality of weight values of the plurality of neural network layers may be optimized by the learning outcome of the AI model. For example, the set of weight values may be updated such that the loss value or cost value derived from the AI model is reduced or minimized during the learning process. Artificial Neural Network may include Deep Neural Network (DNN), Convolutional Neural Network (CNN), Recurrent Neural Network (RNN), Restricted Boltzmann Machine (RBM), Deep Belief Network (DBN), Bidirectional Recurrent Deep Neural Network (BRDNN) or deep Q-nets, but is not limited to these.

[36] According to an embodiment, as long as the AI-based model is a natural language understanding model, the electronic device 100 includes a natural language understanding model that recognizes the user's intent, and can further provide a function (or operation) with it. ) virtual secretary. The electronic device 100 may be implemented in various forms such as a smartphone, tablet personal computer (PC), mobile phone, camera phone, e-book reader, desktop personal computer (PC), laptop personal computer (PC), netbook, workstation, server, personal digital assistant (PDA), portable media player (PMP), MP3 player, medical device, camera, wearable device, etc.

[37] Such a natural language understanding model may exist separately according to an application, and one or a plurality of natural language understanding models corresponding to one application may exist. Since the operation performed for each of the various applications may be different, the input content of the user's voice utterance for each application may also be different. Accordingly, the natural language understanding model may be a model tailored to the respective application. In addition, the natural language understanding model can be divided into a personal model and a public model. Here, the private model is a model generated or trained based on the voice utterance of the user of the electronic device 100, and may be a model personalized for the user of the electronic device 100. In addition, the public model may be a model generated or trained based on the voice utterances sets of users (for example, sets of public users).

[38] The server 200 is a device for managing multiple natural language understanding models and can be implemented as a central server or a cloud server.

[39] According to an embodiment, the server 200 may transmit an appropriate natural language understanding model to the electronic device 100 based on a request from the electronic device 100. In particular, the server 200 may transmit a pre-trained public model to the electronic device 100 based on public user data, and may also transfer the personal model to the electronic device 100.

[40] According to another embodiment, a natural language comprehension model may be generated in the electronic device 100 based on resource information included in an application installed on the electronic device 100. Here, the resource information may include information regarding a given voice utterance, operation information corresponding to the predetermined voice utterance, condition information for generating a natural language comprehension model, application function information corresponding to the predetermined voice utterance, and the like.

[41] The electronic device 100 may generate or train a natural language understanding model based on the resource information received from the server 200. In addition, the electronic device 100 may receive the operation list and the slot list from the server 200 and train the public model as a private model. This is described in detail below with reference to the drawings.

[42] According to various embodiments, when (or based on) a natural language understanding model is obtained on the electronic device 100, an operation corresponding to the user's voice may be performed based on the obtained natural language understanding model.

[43] In addition, according to various embodiments, since the electronic device 100 obtains (or obtains) the natural language understanding model corresponding to the operation of the application frequently used by the user, it is not necessary to store all natural language understanding models in the electronic device 100. Therefore, even in the electronic device having a small storage capacity, natural language understanding of the user's voice can be efficiently performed.

[44] In addition, since the natural language comprehension model is obtained in advance in the electronic device 100 according to the usage frequency of the application, it is not necessary to download the natural language comprehension model from an external server in real time, which makes it possible to efficiently perform natural language comprehension for the user's voice, even when there is no network connection.

[45] FIG. 2 is a diagram illustrating a flowchart of a process for obtaining a natural language understanding model to perform an application function (or operation) according to an embodiment.

[46] Referring to FIG. 2, the electronic device 100 can obtain usage information for an application installed in the electronic device 100 (step S210). Here, the usage information may include at least one of the user input information associated with the application and the application usage information according to the user input. For example, the usage information may include at least one of information regarding the number of user inputs to execute the application, information regarding operations performed in the application, time information when the application is executed, and so on.

[47] According to an embodiment, user input may include stylus or user touch input. For example, the user input may be a user touch input for an application icon or an icon that performs a function included in the application to perform an application or a function included in the application.

[48] According to another embodiment, the user input may include a user input that is input via a virtual assistant. As an example, according to a user input by a user's voice, a user's touch, a pen touch, or the like, while the virtual secretary service is being performed, an operation corresponding to the user's input can be performed.

[49] Here, the virtual secretary service means a service for performing voice recognition for an inputted user voice using an artificial intelligence model, recognizing (or determining) the user's intention, and outputting response information corresponding to the user's voice as a result. In addition, the virtual secretary service may be provided as a dedicated program other than other software (SW), and may be implemented to execute the AI model through a separate dedicated AI processor. When the virtual secretary service (or virtual assistant service) uses a plurality of artificial intelligence models, some of the artificial intelligence models may be executable via a dedicated AI processor.

[50] As another example, the electronic device 100 may receive information regarding an application operation performed according to user input. At this time, information regarding the application operation may include information regarding the type of application operation, information regarding the execution time, information regarding the number of executions, and the like.

[51] The usage information tracking unit included in the electronic device 100 can track the usage information of the user input and store the obtained usage information in a storage device.

[52] The electronic device 100 may obtain a natural language understanding model corresponding to the installed application from a plurality of natural language understanding models based on the usage information (step S220). In particular, if the number of application executions is greater than or equal to a threshold number (e.g., a preset or predetermined threshold number), and/or if the execution time is greater than or equal to a threshold time (e.g., a predetermined or predetermined threshold time), the electronic device 100 may receive the natural language understanding model appropriate for the installed application. In other words, the electronic device 100 can obtain a natural language understanding model for an application frequently used by the user, and can efficiently process the user's voice input.

[53] According to an embodiment, the electronic device 100 can receive and obtain a natural language understanding model corresponding to the application identified based on the usage information from the server 200. This is described in detail with reference to FIG. 3b.

[54] According to another embodiment, the electronic device 100 can also generate a natural language understanding model corresponding to an application identified based on the usage information on the electronic device 100. This is described in detail with reference to FIG. 4b.

[55] In addition, the electronic device 100 may obtain resource information corresponding to an application operation based on the number of executions or execution time of the application operation, as well as the number of executions or execution time of the application, and may obtain a natural language understanding model trained or trained based on received resource information. In particular, if the number of executions of an application operation is greater than or equal to a threshold number, and/or if the execution time is greater than or equal to a threshold time, the electronic device 100 can obtain resource information corresponding to the operation of the application being executed, and obtain a natural language understanding model trained based on the received resource information.

[56] When (or based on) the user's voice associated with the application is input, the electronic device 100 may perform natural language understanding for the user's voice based on the natural language understanding model corresponding to the application (step S230). Here, the natural language understanding model corresponding to the application may be the model obtained in step S220.

[57] In particular, the natural language understanding module may ascertain the intent of the user by performing parsing or semantic analysis on the user's voice. Accordingly, the natural language understanding model may receive the domain, intent, or parameter (or time slice) required to represent the intent of the user input. Here, the subject area may mean a category classified according to the subject of the user's voice or the type of control command corresponding to the user's voice, and, for example, may represent a kind or type of application that is targeted by the user's voice. Such a subject area may be identified by a model based on a subject area classifier. As an example, if the user's voice is input, the domain classifier model can identify the application corresponding to the user's voice. For example, when "Enable camera app A self-portrait capture mode" is entered by the user's voice, the model can identify "camera app A" from the user's voice input as a domain representing the subject area based on the domain classifier. In other words, the model, based on the domain classifier, can identify the domain of the corresponding user voice as "camera application A". As another example, if the user's voice is input, the domain classifier model can identify a wide range of categories associated with the user's intent. For example, a model based on a domain classifier can identify whether a category associated with a user intent is associated with a schedule, message, or reminder.

[58] Thereafter, the natural language understanding model may perform natural language understanding included in the user's voice based on the identified subject area.

[59] The electronic device 100 may perform an application function based on the result of the natural language understanding being performed (step S240). For example, when (or based on) the result of the natural language understanding model is to execute camera application "A", electronic device 100 may execute camera application A.

[60] FIG. 3A is a flowchart of a case in which a natural language understanding model is received from the server 200 according to the embodiment.

[61] Electronic device 100 may include a storage device 110 (eg, at least one storage device), a communication interface 120 (eg, at least one communication interface), and a processor 130 (eg, at least one processor).

[62] Memory 110 may be electrically coupled to processor 130 and store data used in various embodiments (eg, instructions executed by processor 130).

[63] According to an embodiment, the storage device 110 may store a usage information tracking module (eg, instructions) and a voice conflict confirmation module (eg, instructions). The storage device 110 may also store a natural language understanding model transmitted by the server 200. In addition, the storage device 110 may also include a model based on a domain classifier.

[64] Here, the usage information tracking module may be a module that tracks user input information associated with an application and application usage information according to user input. In addition, the usage information tracking module can track usage information for domain and intent information related to the user's voice obtained through the AI model of the virtual assistant service.

[65] According to an example, the usage information tracking module can track usage information for an application. Specifically, the usage information tracking module can track the number of times a corresponding operation is performed and the time an application is executed by touching an application icon or an icon performing a function included in the application with a stylus or body (e.g., finger) of the user.

[66] According to another example, the usage information tracking module may track usage information regarding the virtual secretary. As an example, if the user's voice is input, the voice recognition model of the virtual secretary service AI models can convert the user's voice into text. In addition, if the user's voice is converted to text, at least one subject area associated with the user's voice can be identified through the model based on the subject area classifier included in the natural language understanding model from the artificial intelligence based models of the virtual assistant service. Here, the subject area may be information associated with the application. For example, the subject area can mean the application itself or the category of the application. In other words, the subject area of application A related to music may be the application A itself or the music category.

[67] The usage information tracking module may track the number of times a particular subject area is identified through the AI model of the virtual secretary service, and may transmit the identification information on the corresponding subject area to the server 200 when (or based on) the number times when one subject area is identified, is greater than or equal to the threshold number. Domain identification information may mean information that can identify one subject area, which must be different from other subject areas, and may include at least one of a domain name (title), identification information (identifier), identifier, etc. .P. Accordingly, the electronic device 100 may receive a natural language understanding model corresponding to the appropriate domain from the server 200. Alternatively, the server 200 may request the electronic device 100 to transmit information regarding the number of domain identifications with a given period. In this case, the electronic device 100 may transmit information regarding the number of domain identifications to the server 200 even when the number of domain identifications is less than a threshold number, and the server 200 may accumulate information regarding the number of domain identifications periodically received to determine whether the number of identifications is subject area equal to or greater than the threshold number. When it is determined (at one time, in response to or based on) that the number of identifications of a particular domain is a threshold number or more, the server 200 may transmit a natural language understanding model corresponding to the appropriate domain to the electronic device 100.

[68] As another example, after at least one subject area associated with the user's voice is identified through the model based on the domain classifier, natural language understanding of the text corresponding to the user's voice can be performed through the natural language understanding model of virtual secretary service AI models to obtain user intent information corresponding to the user's voice.

[69] The usage information tracking module may track the number of times specific intent information is identified through the AI model of the virtual secretary service, and may transmit identification information on the corresponding intent information to the server 200 when the number of times one intent information is identified, greater than or equal to the threshold number. The intent information identification information may mean information that can identify one intent information to be different from other intent information, and may include at least one of an intent information name (header), an identification information (identifier), an identifier, etc. .P. Accordingly, the electronic device 100 may receive a natural language understanding model corresponding to a proper intent from the server 200. Alternatively, the server 200 may request the electronic device 100 to transmit usage information for the intent information with a predetermined period or based on a predetermined event. In this case, the electronic device 100 may transmit the intent information identification information to the server 200 even when the number of intent information identifications is less than a threshold number, and the server 200 may accumulate the intent information identification information periodically received to determine whether the number of intent information identifications is intentions greater than or equal to the threshold number. If it is determined that the number of intent information identifications is greater than or equal to a threshold number, the server 200 may transmit a natural language understanding model corresponding to the appropriate intent information to the electronic device 100.

[70] In other words, the usage information tracking module can obtain usage information for domain and intent information identified in the virtual secretary service even before the application is executed. Since the usage information for the domain and intent information is transmitted to the server 200, a natural language understanding model corresponding to the domain information and intent can be received from the server 200 regardless of the execution of the application.

[71] As another example, the usage information tracking module may track the number of executions of an application via the virtual secretary or the number of executions of a function included in the application. For example, the virtual assistant-based model may store usage information based on user input in the storage device 110 or a separate storage device connected to the virtual assistant, and the usage information tracking module may obtain the stored usage information.

[72] Meanwhile, in the above embodiment, the usage information tracking module is described as performing tracking based on domain information or intent information, but this is just an example. As another example, the usage information tracking module may track application transaction information. In other words, the usage information tracking module can keep track of the number of times the operation information of the application corresponding to the user input is identified, and can transmit the identification information of the corresponding operation information and the information identification information of the application performing the corresponding operation to the server 200 when the number of times one transaction information is identified is greater than or equal to the threshold number. Transaction information identification information may mean information that can identify one operation to be different from other transactions, and may include at least one of the operation information name (header), identification information (identifier), time slot information included in the operation, etc. Accordingly, the electronic device 100 may receive resource information and application information corresponding to the appropriate operation from the server 200, and may train or generate a natural language understanding model based on the resource information and application information received from the server 200. Alternatively, the server 200 may train or generate a natural language understanding model based on the operation identification information and the application identification information received from the electronic device 100, and transmit the trained or generated natural language understanding model to the electronic device 100.

[73] Alternatively, the server 200 may query the electronic device 100 to transmit usage information for transaction information with a predetermined period or based on a predetermined event. In this case, the electronic device 100 may transmit the transaction information identification information to the server 200 even when the number of transaction information identifications is less than a threshold number, and the server 200 may accumulate the transaction identification information periodically received to determine whether the number of information identifications is by operation greater than or equal to the threshold number. If it is determined that the number of transaction information identifications is greater than or equal to a threshold number, the server 200 may transmit resource information or a natural language understanding model corresponding to the appropriate transaction information to the electronic device 100.

[74] Meanwhile, the usage information tracking module is described above as being included in the electronic device 100, but the usage information tracking module may also be included in the server 200. In particular, when natural language understanding for the user's voice is performed on the server 200, the server 200 also can track usage information for an application through the usage information tracking module.

[75] Specifically, when the user's voice is input to the electronic device 100, the electronic device 100 can obtain text corresponding to the user's voice through the model based on the voice recognition. At this time, the text corresponding to the user's voice may be obtained through the model based on the voice recognition existing in the electronic device 100, but this is only an example, and the text may be obtained through the model based on the voice recognition of another server.

[76] In addition, the electronic device 100 may transmit the text obtained for natural language understanding to the server 200. The server 200 may use the received text as input to a model based on a domain classifier included in the natural language understanding model. The server 200 may receive domain or intent information associated with the text obtained through the model based on the domain classifier. However, the server 200 is not limited to this, and can receive the user's voice from the electronic device 100 and obtain text corresponding to the user's voice through the voice recognition model provided in the server 200 as described above. Meanwhile, the electronic device 100 transmits the user's voice or text corresponding to the user's voice to the server 200 as follows.

[77] As an example, it is assumed that the model based on the domain classifier of the electronic device 100 does not identify the subject area from the text corresponding to the user's voice input to the electronic device 100. In this case, the electronic device 100 may transmit the user's voice or text, corresponding to the user's voice to the server 200. When a subject area for the user's voice or text received from the electronic device 100 is identified through the model based on the subject area classifier of the server 200, the usage information tracking module of the server 200 may track the number of identifications for each subject area, t .e. domain usage information for each domain. In addition, when the intent information for the user's voice or text after the domain identification is identified, the usage information tracking module of the server 200 may keep track of the number of identifications for each intent information, i. e. usage information for intent information for each intent information.

[78] As another example, when a domain is identified through a model based on the domain classifier of the electronic device 100 from the text corresponding to the user's voice input to the electronic device 100, but the intent information is not identified, the electronic device 100 can transmit the identified domain information and the user's voice or text to the server 200. The server 200 may identify intent information on the user's voice based on the received domain information and the user's voice or text. In addition, the usage information tracking module of the server 200 can track the number of identities for each intent information, i. e. usage information for intent information for each intent information.

[79] As described above, when domain or intent information is identified at the server 200, the usage information tracking module of the server 200 can obtain usage information for the domain or intent information. In this case, the usage information tracking module of the server 200 can summarize the usage information previously received from the electronic device 100 and the usage information received from the server 200. Accordingly, the server 200 can identify a natural language understanding model corresponding to the summarized usage information, and transmit the identified natural language understanding model into the electronic device 100.

[80] Because the natural language understanding model associated with the user's voice or text corresponding to the user's voice is not included in the electronic device 100, the electronic device 100 may query the server 200 to identify domain or intent information associated with the user's voice (or text). ) as described above. In addition, the electronic device 100 may include a natural language understanding model corresponding to the user's voice (text) received from the server 200. In this case, when the user's voice (or text) associated with the natural language understanding model received from the server 200 , is input to the electronic device 100, the electronic device 100 may identify the domain or intent information associated with the user's voice, and may therefore not separately request the domain identification or intent information from the server 200.

[81] In other words, the electronic device 100 can identify the subject area associated with the user's voice or text corresponding to the user's voice through a model based on the subject area classifier of the electronic device 100, and when intent information in the subject area is identified, i. information regarding a particular application function, the electronic device 100 may not separately request domain identification or intent information from the server 200, and may transmit the received usage information to the server 200 when a predetermined condition is met or requested by the server 200.

[82] The voice conflict confirmation module is a module that confirms whether there is a conflict based on the similarity to a previously registered user voice or text corresponding to the user's voice when the user's voice corresponding to a specified operation for the application, or text corresponding to the user's voice, is re-registered. In other words, the voice conflict confirmation module is a module that can confirm whether there is a text conflict as well as a voice conflict. This is detailed below.

[83] The communication interface 120 includes a circuit and is a component capable of communicating with the server 200.

[84] The communication interface 120 may transmit the usage information to the server 200 via a wired or wireless connection and receive a natural language understanding model corresponding to the identified application based on the usage information from the server 200. In this case, the communication interface 120 may transmit the identification information of the electronic device 100 as well as usage information under the control of the processor 130. Here, the identification information may include at least one of an Internet Protocol (IP) address, a model name, a device name, and a serial number of the electronic device 100. The server 200 may determine the target for transmitting a natural language understanding model based on the transmitted identification information.

[85] The processor 130 is electrically coupled to the memory device 110 and fully controls the operation of the electronic device 100. The processor 130 generally controls the operations of the electronic device 100 using various instructions or programs stored in the memory device 110. Specifically, according to an embodiment, the main CPU may copy the program to random access memory (RAM) according to instructions stored in read only memory (ROM) and access the RAM to execute the program. Here, the program may include a natural language understanding model or the like.

[86] The processor 130 may download the usage information tracking module, the natural language understanding model, the voice conflict confirmation module, and the model based on the domain classifier stored in the storage device 110 to the processor 130. For example, the processor 130 can download the usage information tracking module, the natural language understanding model, the voice conflict validation module, and the model based on the domain classifier stored in the storage device 110 outside the processor 130 to the storage device in the processor 130. Meanwhile, with reference to FIG. 3B describes in detail the process in which the electronic device 100 receives a natural language understanding model from the server 200 based on usage information for the application.

[87] FIG. 3B is a diagram for describing a process in which a natural language understanding model is received from a server according to an embodiment of the invention.

[88] The application may be installed in the electronic device 100 (step S310). The application may be installed during manufacture of the electronic device 100, such as as a built-in application, or may be installed at the user's option.

[89] Thereafter, the usage information tracking module may monitor the usage information of the installed application (step S320). In addition, the usage information tracking module can track usage information for domain and intent information related to the user's voice obtained through the AI model of the virtual assistant service.

[90] The usage information tracking module can track user input information associated with the application and application usage information according to the user input. For example, the usage information tracking module may track the number of times food delivery application "B" is executed according to user input. A case in which the food delivery application B is executed according to user input may include a case in which the food delivery application B is selected and executed by touching the user or a pen on the display, and a case in which the food delivery application B is executed by the user's voice. In this case, the domain classifier-based model may identify information such that the domain included in the user's voice is application B. For example, "application B" may be directly included in the user's voice or may not be directly included in the user's voice. user, and words related to delivery may be included instead (for example, "I want to order food delivery" or "Order food delivery"). In the second case, the domain classifier-based model may identify the user's voice domain as application B based on the "delivery" included in the user's voice.

[91] In addition, the usage information tracking module can track the number of times a particular subject area is identified through the AI-based model of the virtual secretary service. For example, when the user's voice for "Application B" is input via the virtual assistant, the user's voice for "Application B" may be converted to text through the model based on the voice recognition. The domain classifier-based model of the AI-based virtual assistant service models can identify the domain associated with the user's voice as "Application B" using the converted text as input. The usage information tracking module may keep track of the number of times a subject area is identified as described above, ie. usage information regarding the subject area.

[92] Alternatively, the usage information tracking module may track the number of identifications of the intent information. For example, the user's intent information may be identified as a selection of the Chinese menu "application B", which is a food delivery application. The usage information tracking module may track the number of times the intent information is identified as described above, ie. usage information for intent information.

[93] Alternatively, the usage information tracking module may track the number of transaction information identifications. For example, when the user operation information is a photographing operation of the camera application A, the usage information tracker can identify "CapturePhoto" and "Camera application A", which is information for the application that performs the operation, as information on the operation. In addition, the usage information tracking module may track the number of times transaction information is identified as described above, ie. usage information for transaction information.

[94] The usage information tracking module may store the received usage information in the storage device 110. The usage information tracking module may store the changed (or updated) usage information in the storage device 110 every time a usage information change occurs, or store the usage information periodically in the storage device. device 110.

[95] The usage information tracking module may identify whether the monitored usage information satisfies a predetermined condition (step S330).

[96] If the usage information satisfies the predetermined condition (Y in step S330), the usage information tracking module may transmit the usage information to the server 200 via the communication interface 120 (step S340). For example, when Application B is executed three or more times, or the accumulated time when Application B is executed is 1 hour or more, the usage information tracking module may transmit usage information for Application B to the server 200. Here, the usage information may include at least one of the application information to be executed, the number of executions of application B, the execution time of application B, and so on. For example, information regarding the number of executions that the number of executions is 5 and identification information that the application being executed is B may be transmitted to the server 200 together. In addition, the usage information tracking module may also transmit the identification information of the electronic device 100 to the server 200. Here, the identification information may include at least one of an IP address, a model name, a device name, and a serial number of the electronic device 100. The server 200 may determine a target for transmitting a natural language understanding model based on the transmitted identification information.

[97] As another example, when the subject area associated with Appendix B is identified three or more times, or when intent information for selecting a specific function in Appendix B is identified three or more times, the usage information tracking module may transmit identification information on the information domain or intent to server 200.

[98] In other words, the predetermined condition may be a condition associated with usage information for an application or usage information for domain and intent information in a virtual secretary service. However, the predetermined condition is not limited to this, and may be the occurrence of the predetermined event or the occurrence of the predetermined time. For example, the usage information tracking module may send the received usage information to the server 200 every 50 hours. In addition, the server 200 accumulates and stores the received usage information, and when the number of application executions, the number of domain identifications, or the number of intent information identifications is greater than or equal to a threshold number based on the accumulated usage information, the server 200 can also transmit a natural language understanding model corresponding to it. , electronic device 100.

[99] As another example, when it is identified that an operation of application C is performed three or more times, the usage information tracking module may transmit information on the operation and information regarding the application that performs the operation to the server 200.

[100] Meanwhile, if the usage information does not satisfy the predetermined condition (N in step S330), the usage information tracking module may continue to monitor usage information for the application or usage information for domain and intent information in the virtual secretary.

[101] Meanwhile, the usage information tracking module periodically confirms whether the usage information satisfies the predetermined condition, or the server 200 may also transmit a query signal to confirm whether the usage information satisfies the predetermined condition to the electronic device 100.

[102] The server 200 can identify whether an application running on the electronic device 100 is included among one or more applications managed by the server 200 based on the usage information. In particular, the server 200 identifies whether the application identified by the model based on the domain classifier of the electronic device 100 is an application managed by the server 200. If it is determined that the application running on the electronic device 100 is managed by the server 200, the server 200 may transmit the natural language comprehension model corresponding to the appropriate application from the plurality of natural language comprehension models to the electronic device 100. For example, if the natural language comprehension model corresponding to application B is identified from the plurality of natural language comprehension models, the server 200 may transmit the identified natural language comprehension model into an electronic device 100.

[103] As an example, the server 200 may transmit a natural language understanding model corresponding to a particular functional operation (or function) to the electronic device 100 based on usage information transmitted from the electronic device 100 from the natural language understanding models corresponding to the corresponding application. For example, when the camera application A is executed according to user input, and a self-portrait is often captured through the camera application A, the server 200 may transmit a natural language understanding model for the self-portrait capture operation of the camera application A to the electronic device 100 based on usage information including such usage history.

[104] The processor 130 may receive the natural language understanding model transmitted from the server 200 via the communication interface 120 (step S350). At this time, the adopted natural language understanding model may be a public model. The received public understanding model based on natural language may include information regarding a given voice utterance and information on the operation corresponding to the voice utterance. For example, the public comprehension model received based on natural language may include information regarding voice utterances, for example, "Include application A", "Appendix A", etc., and information on operation, with the execution of application A in response to This. At this time, processor 130 may receive the natural language understanding model itself, but may also receive resource information for training or generating a natural language understanding model.

[105] The voice conflict confirmation module may confirm whether there is a conflict (or conflict) between the voice utterance information included in the natural language understanding model and the previously registered user voice information (step S360). Here, each of the voice utterance information and the previously registered user voice information includes a voice or text corresponding to a voice.

[106] As an example, when the text corresponding to the voice utterance and the text corresponding to the previously registered voice of the user are obtained through the model based on voice recognition, the voice conflict confirmation module may confirm whether there is a conflict between the text corresponding to the voice utterance and text corresponding to the user's previously registered voice.

[107] For example, in the case where the user's previously registered voice is a pronunciation similar to "Application A" or the same name, but there is another application A on the electronic device 100, the voice "Application A" or text corresponding to the voice may correspond to different operations. In this regard, if the voice conflict is confirmed (Y in step S360), the voice conflict confirmation module may request another voice statement corresponding to the "application A execution operation" (step S370). Accordingly, when another voice utterance is newly entered to execute Application A, the voice conflict confirmation module may confirm the newly entered voice or text corresponding to the voice to confirm whether the entered voice utterance conflicts with the user's previously registered voice. If it is determined that there is no voice conflict, the voice conflict confirmation module may register information regarding the newly entered user voice or text corresponding to the user's voice as the voice or text associated with the "application A execution operation". At this time, the processor 130 may display a user interface (UI) for registering another voice statement, and may register information regarding another voice statement input via the UI.

[108] According to another example, if the number of application executions is greater than or equal to the threshold number, or the application execution time is greater than or equal to the threshold time, i. e. if the predetermined condition is met, the electronic device 100 may request the user's voice corresponding to the specific operation of the application. The voice conflict confirmation module can identify whether the entered user voice conflicts with the previously registered user voice stored in the storage device 110 when the user's voice is input according to the request, request a voice different from the input user voice when the input user voice conflicts with the previously registered user voice. , and associate the user's input voice with the application operation and store it in the storage device 110 when a previously registered voice that conflicts with the user's input voice is not identified.

[109] Meanwhile, if it is determined that the information regarding the voice utterance included in the natural language understanding model does not conflict with the previously registered user voice information (N in step S360), the voice conflict confirmation module may register information regarding the voice utterance (step S380 ).

[110] However, it should be understood that one or more other embodiments are not limited to this. For example, according to another embodiment in which the natural language comprehension model is transmitted from the server 200, whether there is a voice conflict is confirmed on the server 200 in advance, and the operation of separately confirming the voice conflict on the electronic device 100 can also be omitted.

[111] The processor 130 may store the natural language understanding model in the storage device 110 (step S390). At this time, the stored natural language understanding model may be a user-specified personal model. Thereafter, if the user's voice is input, the processor 130 may load the natural language understanding model stored in the memory 110 to the processor 130 to ascertain the user's intention included in the user's voice based on the natural language understanding model.

[112] Meanwhile, according to another embodiment, if the application is installed on the electronic device 100, the electronic device 100 may request a natural language understanding model corresponding to the application from an external server. In other words, if an application is installed on the electronic device 100, regardless of the usage information for the application or information regarding the number of identifications for the specific domain information or specific intents, the electronic device 100 can request a natural language understanding model corresponding to the installed application from an external server. Additionally and by way of example, if the usage information for an application does not satisfy a predetermined condition (e.g., the number of uses or the number of inputs of the user's voice within a predetermined period of time after installation), the natural language understanding model may be removed from the electronic device 100.

[113] FIG. 4A is a flowchart of a case in which a natural language understanding model is generated according to another embodiment.

[114] If the application is installed on the electronic device 100, the application may be stored in the storage device 110. The processor 130 may generate a natural language understanding model based on the resource information included in the application. As illustrated in FIG. 4A, the resource information may include at least one of voice utterance information, operation information, predetermined condition information, and the like. Here, the voice utterance information may be a characteristic utterance specified by the designer, and the voice utterance information may include slot information. The timeslice information can be used as a function parameter for execution. The transaction information may indicate an event to be tracked. The information regarding a given condition may mean a natural language understanding module corresponding to an application or condition for registering a given utterance. In addition, the resource information may further include function information to be executed, and the function information to be executed may store information regarding an application function to be connected with the corresponding utterance.

[115] The resource information may be consistent with information on a given voice utterance, operation information corresponding to a given voice utterance, condition information for generating a natural language understanding model, and application function information corresponding to a given voice utterance, respectively, and stored in a template form. For example, as illustrated in Table 1 below, resource information may be stored in memory 110.

[116] Tab. 1

Characteristic saying Action Description trigger condition Function to execute Take a photo ACTION_IMAGE_C APTURE >Used 3 times after installed startActivityForResult(new Intent(MediaStore.ACTIO N_IMAGE_CAPTURE) Save to Family Album [Entity_AlbumName] ACTION_IMAGE_S AVE_
[Entity_Album_Name] >Used 5 times after installation savePhto(Entity_Album_Name)

[117] The resource information may be included in the application data when the application is installed. In addition, the storage device 110 may store a list of operations and a list of timeslots based on actions and timeslots (or objects) defined in the public model to expand the scope of the public model. The list of operations may be stored by matching at least one characteristic statement corresponding to the operation, as illustrated in Table 2 below.

[118] Tab. 2

ShowContents Show photo, Show photos [Entity_Building], Show photo?... SavePhoto Save photo, Save photo, Save photo in [Entity_folder]...

[119] In addition, the slot list may store at least one common noun that may be included in one slot, as illustrated in Table 3 below.

[120] Tab. 3

Building Hotel A, Hotel B, Building C, High-rise Building D... Automobile XXX, YYY, ZZZ...

[121] Meanwhile, the above-described resource information, the operation list and the slot list may be stored in the server 200, and the electronic device 100 may receive and store the resource information, the operation list and the slot list from the server 200. In this case, when the application is installed , the electronic device 100 may receive and store the resource information, the operation list and the slot list from the server 200, and may receive and store the resource information, the operation list and the slot list from the server 200 when a predetermined condition is satisfied. In addition, the electronic device 100 can automatically generate resource information by analyzing application data. Specifically, when an application without template form resource information is installed, the electronic device 100 can generate the value of the template field by parsing the application data. More specifically, the electronic device 100 can obtain information on the operation through descriptive information of the extraction operation from the application data. In addition, the electronic device 100 may obtain a characteristic utterance corresponding to the transaction information based on the list of transactions received from the server 200. Alternatively, the electronic device 100 may obtain a characteristic utterance corresponding to the transaction information based on a natural language text generation the language stored in the electronic device 100. In addition, the electronic device 100 can obtain information regarding the predetermined condition based on the resource information of another pattern form. As described above, the electronic device 100 can obtain the template form resource information based on the received characteristic utterance, the operation information, and the information regarding the predetermined condition.

[122] In particular, the processor 130 may download the resource information, the usage information tracking module, the natural language understanding model, the voice conflict confirmation module, and the model based on the domain classifier stored in the storage device 110 to the processor 130, and generate a natural language understanding model. . Below with reference to FIG. 4B describes in detail the process of forming a natural language understanding model.

[123] FIG. 4B is a diagram for describing a process in which a natural language understanding model is generated by the electronic device 100 according to another embodiment.

[124] The electronic device 100 may install the application on the electronic device 100 (step S410). The application may be installed during manufacture of the electronic device 100, such as as a built-in application, or may be installed at the user's option. Installing an application in the electronic device 100 may mean that information regarding the application is stored in the storage device 110, and when an application execution command is input, the information regarding the application is loaded into the processor 130 such that various functions included in the application can be executed. Here, the application-related information may include resource information. As described above, the resource information may include at least one of information regarding a given utterance, information on an operation corresponding to a given utterance, information of a given condition for generating a natural language understanding model, application function information corresponding to a given utterance, and etc.

[125] Meanwhile, the usage information tracking module may track the usage information of the installed application (step S420).

[126] In addition, the usage information tracking module may identify whether the received usage information satisfies a predetermined condition included in the resource information (step S430). For example, if the predetermined condition included in the resource information is a condition for executing an application three or more times, and the usage information satisfies such a condition (Y in step S430), the processor 130 may generate a natural language understanding model corresponding to the appropriate application (step S440 ). At this time, the natural language understanding model may be a public model generated based on the application resource information. However, the predetermined condition included in the resource information is not limited to this, and may be that the specific domain identification number, the specific intent information, or the specific operation through the virtual secretary service AI model may be a predetermined number or more.

[127] Specifically, processor 130 may generate (or train) a natural language understanding model based on associating voice utterance information included in resource information with corresponding operation information. For example, the processor 130 may generate or train a natural language understanding model based on the "Turn on camera A" voice utterance information included in the resource information and the execution operation of the camera application A. Accordingly, when the voice utterance "Turn on camera A" is recognized, it is identified by the model based on the domain classifier that the domain included in the user's voice is "camera application A", and a natural language understanding model corresponding to "camera application A" to identify that the intent included in the user's voice is an operation of executing the camera application A.

[128] As another example, the processor 130 may generate the operation plan information by parsing the application operation information and obtain a characteristic statement based on the generated operation plan information. In addition, processor 130 may generate a natural language understanding model based on the transaction information and the characteristic utterance.

[129] In particular, the processor 130 may generate a list for the operation plan by analyzing the operation of the application in advance. In other words, the processor 130 may generate a list for the operation plan by recording the operation of the application when the user executes the application. In addition, the processor 130 may send a list for the plan of operations to the server 200 to obtain a characteristic utterance of the natural language understanding model corresponding to the corresponding application. Alternatively, processor 130 may parse the meaning of the list for the plan of operations via a translation module and translate the parsed meaning into a characteristic utterance. In particular, since the operation plan is a plurality of instructions, the processor 130 can receive a plurality of words through the operation plan. For example, if (1) [Activity: Execute][Property: Baemin] (2) [Activity: Search][Property: Chinese food] and (3) [Activity: Display][Property: Search Result] as the action plan is , the processor 130 may receive (1) Baemin (2) search for Chinese food, and (3) display the search result as a plurality of words. In addition, the processor 130 may generate multiple words as a single natural language sentence via the natural language generation module. At this time, processor 130 may obtain a natural language sentence by assigning a weight value to each operation. In particular, processor 130 may assign a high weight value to an operation that includes the named object and assign a low weight value to a subsequent operation. For example, the processor 130 may receive a natural language sentence such as "Find and display Chinese food in Baemin" or "Find Chinese food in Baemin" by inputting the plurality of words described above into the natural language generation module. At this time, the named object included in the application name or operation may be obtained based on the slot list. In addition, processor 130 may generate or train a natural language understanding model based on the received action plan and natural language sentence.

[130] If the usage information does not satisfy a predetermined condition for generating a natural language understanding model (N in S430), the usage information tracking module may continuously monitor usage information for an application or usage information for specific domain or specific intent information through the AI-based model. virtual secretary services.

[131] Meanwhile, the voice conflict confirmation unit may confirm whether there is a conflict between the voice statement information included in the resource information and the previously registered user voice information (step S450). Alternatively, the voice conflict confirmation module may confirm whether there is a conflict between the text corresponding to the voice utterance and the text corresponding to the user's previously registered voice.

[132] In particular, the voice conflict confirmation module can confirm whether there is a conflict based on the similarity between the voice utterance information (or text information corresponding to it) included in the resource information and the user's previously registered voice (or text corresponding to it). ). For example, it is assumed that there is a voice input "Turn on camera A" in a previously registered user's voice, and that the operation corresponding to it includes the operation of executing camera application A and shooting through the rear camera. In this case, since different operations correspond to the same voice input "Turn on camera A", the voice conflict confirmation module can identify that there is a voice conflict.

[133] In this regard, if the voice conflict is confirmed (Y in step S450), the voice conflict confirmation unit may request a voice statement different from the voice statement information included in the resource information (step S460).

[134] For example, the processor 130 may control the speaker to output the command “Perform voice input other than “Turn on camera A””, or may control the display to display a UI window that includes “Perform voice input other than “Turn on camera A”. camera A".

[135] Accordingly, when another voice utterance is newly input to execute camera application A, the voice conflict confirmation module may confirm whether the entered voice utterance conflicts with the user's previously registered voice. If it is determined that there is no voice conflict, the voice conflict confirmation module may generate a natural language understanding module corresponding to the application A for operating the camera in a state in which information regarding the newly entered user voice is registered as the voice associated with the "execution operation of the application A for camera work. As an example, a user's voice input such as "Activate camera A" other than "Enable camera A" may be input or registered as a voice (or voice input) corresponding to an execution operation of the camera application A. Therefore, based on the generated natural language comprehension model, when "Activate camera A" is input, an operation of executing the camera application A is performed, and when "Enable camera A" is input, a shooting operation is performed through the rear camera by executing the camera application A camera, which may differ from each other.

[136] As an example, a case is described in which the voice utterance information included in the resource information is the same as the user's previously registered voice, but even if the voice utterance information included in the resource information is the same as the previously registered user's voice, and the natural language understanding model may likely misinterpret the user's intent, the voice conflict confirmation module may request a voice utterance different from the voice utterance information included in the resource information.

[137] Meanwhile, if it is determined that the voice utterance information included in the natural language understanding model does not conflict with the previously registered user voice information (N in step S450), the voice conflict confirmation module may register the voice utterance information as a voice. , corresponding to the execution operation of application A for working with the camera.

[138] The processor 130 may store a natural language understanding model in which the user's voice is registered in the memory 110 (step S470). At this time, the stored natural language understanding model may be a personal model.

[139] Thereafter, if the user's voice is input, the processor 130 may load the natural language understanding model stored in the memory 110 to the processor 130 to ascertain (or determine) the user's intent included in the user's voice based on the natural language understanding model. .

[140] Meanwhile, according to the above-described embodiments, the natural language understanding model can be obtained by a method received from the server 200 or generated on the electronic device 100 based on the usage information. In addition, the electronic device 100 may delete (or erase) the received, generated, or stored natural language comprehension model from the electronic device 100 when a predetermined condition is met (for example, if a user input corresponding to the received natural language comprehension model is not entered a threshold number of times, or over a given period of time). If the user input is not entered by a threshold number of times or more, indicating that the frequency of use of the application corresponding to the natural language understanding model is low, the electronic device 100 may remove the natural language understanding model having a low frequency of use from the electronic device 100.

[141] FIG. 5 is a block diagram illustrating the electronic device 100 and the server 200 according to the embodiment.

[142] Referring to FIG. 5, the electronic device 100 includes a storage device 110 (eg, at least one storage device), a communication interface 120 (eg, at least one communication interface), and a processor 130 (eg, at least one processor), and the server 200 includes a storage device 210 (eg, at least one storage device), a communication interface 220 (eg, at least one communication interface), and a processor 230 (eg, at least one processor).

[143] Memory 110 may be electrically coupled to processor 130 and store data used in various embodiments.

[144] The storage device 110 may be implemented as a storage device built into the electronic device 100, or may also be implemented as a storage device that is connectable and removable with respect to the electronic device 100, depending on the purpose of storing the data. For example, data for operating the electronic device 100 may be stored in a storage device built into the electronic device 100, and data for advanced functions of the electronic device 100 may be stored in a storage device attached and removable with respect to the electronic device 100. Meanwhile, the storage device built into electronic device 100 may be implemented as at least one of volatile memory (e.g., dynamic random access memory (DRAM), static RAM (SRAM), synchronous dynamic RAM (SDRAM), etc.) or non-volatile memory (e.g., one-time programmable read-only memory (OTPROM), programmable ROM (PROM), erasable and programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), masked ROM, flash ROM, flash memory (e.g. NAND- flash memory, NOR flash memory, etc.), hard disk or solid-state drive (SSD)). The storage device, which is connectable and removable with respect to the electronic device 100, may be implemented in such a form as a memory card (for example, a Compact Flash (CF) memory card, a Secure Digital (SD) memory card, a Micro Secure Digital (microSD), Mini Secure Digital (MiniSD) memory card, Extreme Digital (xD) memory card, multimedia card (MMC), etc.), external storage device (such as USB memory ) connected to the USB port, etc.

[145] According to an embodiment, the storage device 110 may store a usage information tracking module, a voice conflict confirmation module, a natural language understanding model, and a model based on a domain classifier.

[146] The communication interface 120 includes a circuit and is a component capable of communicating with the server 200.

[147] The communication interface 120 may include a Wi-Fi module, a Bluetooth module, an infrared (IR) module, a local area network (LAN) module, an Ethernet module, and the like. Here, each communication module can be implemented as at least one hardware chip. The wireless communication module may include at least one communication chip that communicates according to various wireless communication protocols such as ZigBee, Universal Serial Bus (USB), Camera Serial Interface for Mobile Industrial Processor Interface (MIPI CSI), third generation standard (3G), 3rd Generation Partnership Project (3GPP) standard, Long Term Evolution (LTE), LTE-Advanced (LTE-A), fourth generation (4G), fifth generation (5G), etc., in addition to the above methods of communication. However, this is only an example, and communication interface 120 may use at least one communication module from various communication modules. In addition, the communication interface 120 can communicate with the server via a wired connection.

[148] In the configuration illustrated in FIG. 5, the storage device 210 and the communication interface 220 included in the server 200 perform similar functions with respect to the storage device 110 and the communication interface 120 included in the electronic device 100, and their description will not be repeated below.

[149] Meanwhile, the processor 130 may transmit usage information for the application to the server 200 via the communication interface 120 . The processor 230 of the server 200 may receive the usage information via the communication interface 220 and identify whether an application running on the electronic device 100 is an application stored in the storage device 210 based on the received usage information. If it is determined that the application running on the electronic device 100 is an application stored on the storage device 210, the processor 230 may identify a natural language understanding model corresponding to the appropriate application from the plurality of natural language understanding models on the storage device 210. Thereafter, the processor 230 may control communication interface 220 to transmit the identified natural language comprehension model to electronic device 100. Processor 130 may store the natural language comprehension model transmitted from server 200 in storage device 110.

[150] FIG. 6 is a view for describing the sequence in which the natural language understanding model operates according to the embodiment.

[151] Referring to FIG. 6, the electronic device 100 may receive the user's voice (eg, user voice input) (step S610). If the user's voice is input to the electronic device 100, the electronic device 100 can obtain text corresponding to the user's voice through the model based on the voice recognition. However, it should be understood that one or more other embodiments are not limited to this, and voice recognition for the user's voice may be performed by an external device, an external server, or the like, and the electronic device 100 may also receive text data corresponding to the voice. user, outside. In addition, the domain classifier-based model included in the electronic device 100 can identify a domain corresponding to the received text. The electronic device 100 can identify a natural language understanding model associated with the identified domain. In this case, the electronic device 100 can identify whether natural language comprehension can be performed on the user's voice through a personal model from the natural language comprehension models associated with the identified domain (step S620).

[152] Here, the personal model is a model generated or trained based on the voice utterance of the user of the electronic device 100, and may be a model personalized for the user of the electronic device 100. For example, the personal model may be a personalized model based on a natural understanding model. language received from the server 200 or generated in the electronic device 100 based on the usage information obtained through the usage information tracking module. In other words, the personal model may be a natural language understanding model associated with a particular application that is frequently used by the user, compared to the public model described below.

[153] If it is identified in the personal model that it is possible to perform natural language understanding for the user's voice (Y in step S620), the electronic device 100 can perform natural language understanding through the corresponding personal model (step S630) and can perform an application function based on the understanding result. natural language (step S680).

[154] Meanwhile, if it is impossible to perform natural language comprehension for the user's voice in the personal model (N in step S620), the electronic device 100 may identify whether it is possible to perform natural language comprehension for the user's voice in the public model. Here, the public model is a model that is generated or trained based on the voice utterance of a public user (for example, another user or a user who is a member of the public), and may be a model used to understand natural language included in the public public user voice. For example, the public model may be a natural language understanding model associated with an embedded application that is substantially installed in the electronic device 100 when the electronic device 100 is manufactured, or a natural language understanding model generated based on application resource information.

[155] If it is determined in the public model that it is possible to perform natural language understanding for the user's voice (Y in step S640), the electronic device 100 may perform natural language understanding through the corresponding public model (step S650) and may perform an application function based on the understanding result natural language (step S680).

[156] Meanwhile, if it is not possible to perform natural language understanding for the user's voice in the public model (N in step S640), the electronic device 100 may transmit the user's voice or text corresponding to the user's voice to the server 200 (step S660). Since a natural language understanding model capable of performing natural language understanding for the user's voice is not provided in the electronic device 100, the electronic device 100 can transmit the user's voice or text corresponding to the user's voice to the server 200 including a plurality of natural language understanding models to perform natural language understanding for the user's voice.

[157] Thereafter, the model based on the domain classifier included in the server 200 can identify the domain corresponding to the user's received voice or text corresponding to the user's voice. The server 200 may identify a natural language understanding model associated with the identified domain and perform natural language understanding for the user's voice (or text corresponding to the user's voice) via the identified natural language understanding model.

[158] The electronic device 100 may receive information regarding a natural language understanding result from the server 200 (step S670). Thereafter, the electronic device 100 may perform an application function based on the natural language understanding result (step S680).

[159] FIG. 7 is a flowchart for describing an example in which the electronic device 100 and the server 200 work together to perform the application function according to the user's voice, according to the embodiment.

[160] Referring to FIG. 7, the electronic device 100 may receive the user's voice (eg, user voice input) (step S710). If the user's voice is input to the electronic device 100, the electronic device 100 can obtain text corresponding to the user's voice through the model based on the voice recognition. However, it should be understood that one or more other embodiments are not limited to this, and voice recognition for the user's voice may be performed by an external device, an external server, or the like, and the electronic device 100 may also receive text data corresponding to the voice. user, outside.

[161] The electronic device 100 may classify the subject area corresponding to the user's voice (step S720). In particular, the electronic device 100 can identify a subject area corresponding to the received text using a model based on the subject area classifier included in the electronic device 100.

[162] The electronic device 100 may determine whether there is an application corresponding to the classified subject area (step S730). In other words, the electronic device 100 can determine whether there is a natural language understanding model corresponding to the application corresponding to the classified domain in the electronic device 100. If there is no application corresponding to the classified domain (N in step S730), the electronic device 100 can transmit information with respect to the user's voice to the server 200 (step S735).

[163] If it is determined that there is an application corresponding to the classified subject area (Y in step S730), the electronic device 100 may determine whether the operation information corresponding to the text is associated (step S740). If the operation information corresponding to the text is not linked in the electronic device 100 (N in step S745), the electronic device 100 may transmit information regarding the user's voice and information regarding the application to the server 200 (step S745).

[164] If the transaction information corresponding to the text is linked in the electronic device 100 (Y in step S745), the electronic device 100 can obtain the transaction information based on the natural language understanding model (step S750). In addition, the electronic device 100 can perform an application function according to the received operation information (step S770).

[165] Alternatively, the server 200 may obtain transaction information using the natural language understanding model stored in the server 200 based on the user's voice information (eg, text information) or application information received from the electronic device 100 (step S760). In addition, the server 200 may transmit the received transaction information to the electronic device 100 (step S765), and the electronic device 100 can perform an application function according to the received transaction information (step S770).

[166] It should be understood that the methods according to one or more embodiments of the invention described above may be implemented in the form of an application installed in an electronic device.

[167] In addition, the methods according to one or more embodiments of the invention described above can only be implemented by updating the software or upgrading the hardware of the prior art electronic device.

[168] In addition, one or more of the embodiments described above can also be performed via an embedded server included in the electronic device, or at least one external server relative to the electronic device.

[169] Meanwhile, one or more of the embodiments described above may be implemented by software including instructions that are stored on computer-readable (eg, computer-readable) storage media. A machine is a device that activates stored instructions from storage media and is configured to operate according to the activated instructions, and may include an electronic device according to the disclosed embodiments. When instructions are executed by the processor, the processor may perform the functions corresponding to the instructions directly or through other components under the control of the processor. The instructions may include codes generated or executed by a compiler or interpreter. The computer-readable storage medium may be provided as a permanent storage medium. Here, the term "permanent" means that the storage medium does not include a signal and is physical, but does not distinguish between whether the data is stored semi-permanently or temporarily on the storage medium. For example, a "persistent storage medium" may include a buffer in which data is stored temporarily.

[170] In addition, the method according to one or more of the embodiments described above may be included in and provided in a computer program product. A computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of computer-readable storage media (eg, compact disc read only memory (CD-ROM)) or online through an application store (eg, PLAYSTORE™). In the case of Internet distribution, at least a portion of the computer program product may be at least temporarily stored on a storage medium such as a storage device of a manufacturer's server, an application store server, or a relay server, or may be temporarily generated.

[171] In addition, one or more of the embodiments described above may be implemented on a computer-readable or similarly device-readable recording medium using software, hardware, or a combination thereof. In some cases, embodiments may be implemented by the processor itself. According to software implementation, the procedures and functions described above may be implemented as separate software modules. Each of the software modules may perform one or more of the functions and operations described above.

[172] Meanwhile, computer instructions for performing device processing operations according to one or more of the embodiments described above may be stored in a non-volatile computer-readable medium. Computer instructions stored on a non-volatile computer-readable medium allow a particular device to perform the device processing operations of one or more of the embodiments described above when executed by the processor of the particular device.

[173] A persistent computer-readable medium is not a medium that stores data for a short time, such as a register, a cache, a storage device, and the like, but means a computer-readable medium that stores data in a semi-permanent manner. A specific example of a permanent computer readable medium may include a compact disc (CD), a digital versatile disc (DVD), a hard disk drive, a Blu-ray disc, a universal serial bus (USB), a memory card, a read only memory (ROM), etc. P.

[174] In addition, each of the components (e.g., modules or programs) according to the embodiments described above may include one object or multiple objects, and some subcomponents of the subcomponents described above may be omitted, or other subcomponents may be further included in various embodiments. Alternatively or in addition, some components (eg, modules or programs) may be integrated into one entity to perform identical or similar functions performed by the respective components prior to integration. Operations performed by a module, program, or other component, in accordance with various embodiments, may be performed in a sequential, parallel, iterative, or heuristic manner, or at least some of the operations may be performed in a different order, or may be omitted or others may be added. operations.

[175] Although embodiments of the invention have been illustrated and described above, the invention is not limited to the above specific embodiments, but may be modified in various ways by those skilled in the art to which the invention pertains without departing from the spirit of the invention as disclosed in at least the appended claims. It should also be understood that these modifications fall within the scope and spirit of the invention.

[176] Meanwhile, the general terms that are currently widely used may be the terms used in the above description of the embodiments in view of the functions in the invention, but may change depending on the intent of those skilled in the art or judicial precedent, occurrence new technology, etc. In addition, in a particular case, terms arbitrarily chosen by the applicant may exist. In this case, the meaning of such terms may be mentioned in detail in the relevant part of the specification. Therefore, the terms used in this description should be defined based on the meaning of the terms and their content throughout the description, and not on the basis of only the names of these terms.

[177] Singular expressions include plural expressions unless the context clearly indicates otherwise. It should be further understood that the terms "comprise" or "comprise" as used herein indicate the presence of features, numbers, steps, operations, components, parts, or combinations thereof mentioned in this detailed description, but do not exclude the presence or the addition of one or more other features, numbers, steps, operations, components, parts, or combinations of the foregoing.

[178] It should be understood that expressions such as "at least one of A and/or B" represent either "A" or "B", or "A" and "B".

[179] The expressions "first", "second", etc. used in the invention may indicate different components regardless of the sequence and/or importance of the components and are used only to distinguish one component from other components, and do not limit otherwise relevant components.

[180] If it is indicated that any component (for example, the first component) is (functionally or with the possibility of communication) associated with or connected to another component (for example, the second component), it should be understood that any component is directly connected to another component or can be connected to said other component via another component (for example, a third component).

[181] As used herein, a "module" or "unit" may perform at least one function or operation and may be implemented by hardware or software, or may be implemented by a combination of hardware and software. In addition, a plurality of "modules" or a plurality of "blocks" may be combined in at least one module and implemented by at least one processor, with the exception of a "module" or "block", which needs to be implemented by specific hardware. In this detailed description, the term "user" can mean a person using a terminal device (or terminal device) or a device (eg, an artificial intelligence electronic device) using a terminal device.

[182] Although embodiments of the invention have been illustrated and described above, the invention is not limited to the above specific embodiments, but may be varied in various ways by those skilled in the art to which the invention pertains without departing from the spirit of the invention as disclosed in at least the appended claims. . It should also be understood that these changes are within the scope and spirit of the invention.

Claims (46)

1. An electronic device for performing an application operation using a natural language understanding model, comprising: - a storage device configured to store at least one instruction; and a processor configured to execute at least one instruction for: - obtaining usage information for the application installed in the electronic device, - obtaining a natural language understanding model from a plurality of natural language understanding models corresponding to the application, based on the usage information, - performing natural language comprehension for user voice input associated with the application, based on the natural language understanding model corresponding to the application, and - performing an application operation based on a pre-formed natural language understanding, wherein the natural language understanding model corresponding to the application is received from the external server based on that the number of application executions included in the usage information is equal to or greater than the threshold number, or on the basis that the application execution time is equal to or greater than threshold time. 2. An electronic device according to claim 1, further comprising: - communication interface, wherein the processor is further configured to execute at least one instruction for: - obtaining usage information including at least one of the number of executions of the application and the execution time of the application, controlling a communication interface to transmit the received usage information to an external server, - receiving a natural language understanding model corresponding to an application from a plurality of natural language understanding models corresponding to a plurality of applications from an external server via a communication interface, and storing the natural language understanding model in a storage device. 3. An electronic device according to claim 1, in which: - the application includes resource information that matches information about the given voice statement and operation information corresponding to the given voice statement; And the processor is further configured to execute at least one instruction based on that the usage information for the application meets the predetermined condition to generate a natural language understanding model corresponding to the application according to the information regarding the predetermined voice utterance and the operation information corresponding to the predetermined voice utterance. 4. The electronic device of claim 1, wherein the processor is further configured to execute at least one instruction to request a first user voice input corresponding to an application operation determined based on the usage information to have been executed a number of times greater than or equal to a threshold number of times, or for a time greater than or equal to the threshold time. 5. The electronic device of claim 4, wherein the processor is further configured to execute at least one instruction for: - determining whether the first user voice input conflicts with the user voice input stored in the storage device, based on the first user voice input entered according to the request; - requesting a second user voice input different from the first user voice input based on the determination that the first user voice input conflicts with the stored user voice input; And - storing in the storage device the first user voice input in accordance with the operation of the application, based on the determination that the first user voice input does not conflict with the stored user voice input. 6. The electronic device of claim. 1, wherein the processor is further configured to execute at least one instruction to request from an external server a natural language understanding model corresponding to an application based on an application installed in the electronic device. 7. An electronic device according to claim 1, in which: the natural language understanding model includes at least one of a personal model generated based on the user's voice utterance using an electronic device and a public model generated based on the public user's voice utterance; and the processor is further configured to execute at least one instruction for: - performing natural language understanding for user voice input in the personal model, - performing natural language comprehension for the user's voice input in the public model based on the fact that natural language comprehension for the user's voice input in the personal model was not successful, and - transmitting the user's voice input to an external server to perform natural language comprehension for the user's voice input based on the fact that natural language comprehension for the user's voice input in the public model was not successful. 8. The electronic device of claim. 1, wherein the processor is further configured to execute at least one instruction to remove the received natural language understanding model based on the fact that the user's voice input is not entered a number of times greater than or equal to a threshold number of times, or for a time greater than or equal to the threshold time. 9. A method for controlling an electronic device, wherein the control method comprises the steps of: - receive usage information for the application installed in the electronic device; obtaining, based on the usage information, a natural language understanding model from a plurality of natural language understanding models corresponding to the application; - performing natural language understanding for the user voice input associated with the application, based on the natural language understanding model corresponding to the application; And - performing an application operation based on the performed natural language understanding, wherein the natural language understanding model corresponding to the application is received from the external server on the basis that the number of application executions included in the usage information is equal to or greater than the threshold number, or on the basis that the application execution time is equal to or greater than the threshold time. 10. The control method according to claim 9, in which: - the usage information contains at least one of the number of executions of the application and the execution time of the application, and wherein obtaining a natural language understanding model comprises the steps of: - transmitting the received usage information to an external server, - receiving a natural language understanding model corresponding to an application from a plurality of natural language understanding models corresponding to a plurality of applications from an external server, and storing a natural language understanding model in the electronic device. 11. The control method according to claim 9, in which: - the application includes resource information that matches information about the given voice statement and operation information corresponding to the given voice statement; and deriving the natural language comprehension model comprises, based on the usage information for the application meeting the predetermined condition, generating a natural language understanding model corresponding to the application according to the information regarding the predetermined voice utterance and the operation information corresponding to the predetermined voice utterance. 12. The control method according to claim 9, further comprising requesting a first user voice input corresponding to an operation of the application, which is determined based on the usage information as being executed a number of times greater than or equal to a threshold number of times, or for a time greater than or equal to the threshold time. 13. The control method according to claim 9, further comprising the steps of: determining whether the first user voice input conflicts with the input user voice stored in the electronic device based on the first user voice input input according to the request; requesting a second user voice input different from the first user voice input based on the determination that the first user voice input conflicts with the stored user voice input; And - storing in the electronic device the first user voice input in accordance with the operation of the application based on the determination that the first user voice input does not conflict with the stored user voice input.

Images