KR20240054645A - Server, method and computer program for managing ensemble models about multiple single models - Google Patents

Abstract

The server that manages the ensemble model for multiple single models is a receiver that receives query data, external feature values for the query data, classification results for query data output from multiple single classification models, and reliability for the classification results. , It may include a data generation unit that generates a learning data set based on classification results, reliability, and external feature values, and a derivation unit that inputs the generated training data set into a pre-trained ensemble model to derive an output value for the query data. there is.

Description

Server, method, and computer program for managing ensemble models for a plurality of single models {SERVER, METHOD AND COMPUTER PROGRAM FOR MANAGING ENSEMBLE MODELS ABOUT MULTIPLE SINGLE MODELS}

The present invention relates to a server, method, and computer program for managing an ensemble model for a plurality of single models.

Machine learning is a field of artificial intelligence and refers to a technology that allows computers to learn the relationship between problems and answers through learning data so that they can derive the correct answer to a specific problem.

In order to create a model with excellent performance, high-quality learning data and a learning algorithm for the generalization process are required.

There is an ensemble technique as a technique to improve model performance. This ensemble technique is a technique that combines multiple weak learners to create a strong learner.

In existing ensemble models, the learning process is automated, so it is difficult to include the manager's intuition or business logic in the model learning process, and it is difficult for the manager to intuitively understand the model learning process. Additionally, it is currently difficult to track issues through an ensemble model, and it is difficult for managers to control the results of the ensemble model.

Japanese Patent Publication No. 2015-026372 (published on February 5, 2015)

The present invention is intended to solve the problems of the prior art described above, and learns based on external feature values for query data, classification results for the query data output from a plurality of single classification models, and reliability of the classification results. We want to create a data set and input the generated learning data set into a pre-trained ensemble model to derive the output value for the query data.

However, the technical challenges that this embodiment aims to achieve are not limited to the technical challenges described above, and other technical challenges may exist.

As a technical means for achieving the above-described technical problem, a server that manages an ensemble model for a plurality of single models according to the first aspect of the present invention includes query data, external feature values for the query data, and a plurality of single classification models. a receiving unit that receives a classification result value for the query data output from and a reliability level for the classification result value; a data generator that generates a learning data set based on the classification results, the reliability, and the external feature values; It may include a derivation unit that inputs the generated training data set into a pre-trained ensemble model to derive an output value for the query data.

A method of managing an ensemble model for a plurality of single models by an ensemble model management server according to a second aspect of the present invention includes query data, external feature values for the query data, and the query data output from a plurality of single classification models. Receiving a classification result value and a reliability level for the classification result value; generating a learning data set based on the classification result, the reliability, and the external feature value; and inputting the generated training data set into a pre-trained ensemble model to derive an output value for the query data.

A computer program stored in a computer-readable recording medium including a sequence of instructions for managing an ensemble model for a plurality of single models according to the third aspect of the present invention, when executed by a computing device, provides query data, information on the query data, Receive external feature values, classification results for the query data output from a plurality of single classification models, and reliability for the classification results, and create a learning data set based on the classification results, the reliability, and the external feature values. It may include a sequence of commands for generating an output value for the query data by inputting the generated training data set into a pre-trained ensemble model.

The above-described means for solving the problem are merely illustrative and should not be construed as limiting the present invention. In addition to the exemplary embodiments described above, there may be additional embodiments described in the drawings and detailed description of the invention.

According to one of the means for solving the problems of the present invention described above, the present invention is based on external feature values for query data, classification results for the query data output from a plurality of single classification models, and reliability of the classification results. You can create a learning data set and input the generated learning data set into a pre-trained ensemble model to derive the output value for the query data.

Through this, the present invention allows managers to control the results of the ensemble model while intuitively understanding the simulation process of the ensemble model for each single model. Additionally, issue tracking using the ensemble model of the present invention can be facilitated.

1 is a configuration diagram of an ensemble model management system according to an embodiment of the present invention.
Figure 2 is a block diagram of the ensemble model management server shown in Figure 1, according to an embodiment of the present invention.
Figure 3 is a diagram for explaining a method of learning an ensemble model according to an embodiment of the present invention.
Figure 4 is a diagram for explaining a method of selecting a final ensemble model according to an embodiment of the present invention.
Figure 5 is a flowchart showing a method of managing an ensemble model for a plurality of single models according to an embodiment of the present invention.

Below, with reference to the attached drawings, embodiments of the present invention will be described in detail so that those skilled in the art can easily implement the present invention. However, the present invention may be implemented in many different forms and is not limited to the embodiments described herein. In order to clearly explain the present invention in the drawings, parts unrelated to the description are omitted, and similar parts are given similar reference numerals throughout the specification.

Throughout the specification, when a part is said to be "connected" to another part, this includes not only the case where it is "directly connected," but also the case where it is "electrically connected" with another element in between. . Additionally, when a part "includes" a certain component, this means that it may further include other components rather than excluding other components, unless specifically stated to the contrary.

In this specification, 'part' includes a unit realized by hardware, a unit realized by software, and a unit realized using both. Additionally, one unit may be realized using two or more pieces of hardware, and two or more units may be realized using one piece of hardware.

In this specification, some of the operations or functions described as being performed by a terminal or device may instead be performed on a server connected to the terminal or device. Likewise, some of the operations or functions described as being performed by the server may also be performed on a terminal or device connected to the server.

Hereinafter, specific details for implementing the present invention will be described with reference to the attached configuration diagram or processing flow diagram.

1 is a configuration diagram of an ensemble model management system according to an embodiment of the present invention.

Referring to FIG. 1, the ensemble model management system may include a plurality of single model management servers 100 and ensemble model management servers 110. However, since the ensemble model management system of FIG. 1 is only one embodiment of the present invention, the present invention is not limited to FIG. 1, and may be configured differently from FIG. 1 according to various embodiments of the present invention. .

The plurality of single model management servers 100 may generate data sets for learning a plurality of single classification models constituting an ensemble model. For example, the plurality of single model management servers 100 determine the size of the data set in consideration of the update cycle required in commercial services, and according to the size of the determined data set, a preset number of data sets are collected as the entire collected data. It can be created from

The plurality of single model management server 100 may train a plurality of single classification models constituting an ensemble model based on the generated data set. For example, the first single model management server 100-1 may train a first single classification model, and the second single model management server 100-2 may train a second single classification model. Here, the plurality of single classification models may include, for example, a neural network-based classification model.

The plurality of single model management servers 100 input input data to each learned single classification model, and the ensemble model management server determines the classification results for the input data output from each single classification model and the reliability of the classification results. It can be provided to (110).

For example, when each single classification model receives input data converted into numeric information through a tokenizer, it can output a classification result for the input data and a reliability level for the classification result. At this time, the reliability of the classification result may correspond to the output value of the softmax function. For example, if the input data is ‘What are the cell phone charges?’ converted into numeric information through a tokenizer, the classification result for the input data may be ‘Wireless mobile phone charges’, and the classification of the data The reliability of the result can be 0.98.

The ensemble model management server 110 may generate a learning data set based on classification results for the input data, reliability for the classification results, and external feature values for the input data. Here, the external feature value may include information about business logic.

The ensemble model management server 110 can generate a learning data set similar to an actual service environment. For example, the ensemble model management server 110 may generate learning data by sampling the collected data for each class collected in relation to the actual service environment according to a distribution set for each class. For example, the ensemble model management server 110 samples data according to the first distribution (e.g., 10%) set for class A among a plurality of data belonging to class A, and class B among a plurality of classes belonging to class B. By sampling data according to the second distribution (e.g., 20%) set in , learning data that can be expected to provide optimized performance suitable for actual services can be generated.

The ensemble model management server 110 may learn and simulate ensemble models for a plurality of single models based on the training data set. When training an ensemble model, the ensemble model management server 100 may communicate with each single model management server 100 using an API.

The ensemble model management server 100 stores the learning results and simulation results of the ensemble model in internal storage, and when the learning/simulation of the ensemble model ends, metadata related to the learning/simulation, including the reliability of the classification result, etc. can be sent to the administrator server.

The manager server can set manual rules used to train the ensemble model. Here, manual rules can be delivered to the ensemble model in API form.

When learning/simulation of the ensemble model is completed, the manager server may store the metadata received from the ensemble model management server 100 in an internal database.

The manager server may determine a commercial model using metadata received from the ensemble model management server 100 and transmit information about the determined commercial model to the ensemble model management server 100 through an API.

When query data (eg, user speech data) is received from the API server, the query data may be distributed to a plurality of single model management servers 100 and ensemble model management servers 100.

The plurality of single model management servers 100 derive classification results for query data and reliability of the corresponding classification results through each single classification model, and aggregate the reliability of the derived classification results and the corresponding classification results. It can be delivered to the model management server 100.

The ensemble model management server 100 provides classification results for query data received from a plurality of single model management servers 100 and reliability of the corresponding classification results, as well as additional metadata (query data and external characteristics of the query data). You can create a learning data set based on (value), input the generated learning data set into a pre-trained ensemble model to derive an output value for the query data, and deliver the derived output value to the API server.

Below, the operation of each component of the ensemble model management system of FIG. 1 will be described in more detail.

Figure 2 is a block diagram of the ensemble model management server 110 shown in Figure 1, according to an embodiment of the present invention.

Referring to FIG. 2, the ensemble model management server 110 may include a receiving unit 200, a data generating unit 210, a deriving unit 220, and a learning unit 230. However, the ensemble model management server 110 shown in FIG. 2 is only one implementation example of the present invention, and various modifications are possible based on the components shown in FIG. 2.

The data generator 210 may generate a learning data set based on classification results for input data output from a plurality of single classification models, reliability for the classification results, and external feature values for the input data. Here, the external feature value may include information about business logic.

External feature values may consist of categorical data or continuous data. Here, categorical data may include information about whether a specific pattern is included. For example, if 'mobile phone fee' is included in the input data, the categorical data becomes 'True' data, and if 'mobile phone fee' is not included in the input data, the categorical data becomes 'True' data. It can be ‘false’ data.

Continuous data may include percentile information. For example, if a customer's percentile of monthly cell phone usage time is 56%, the continuous data could be 56.

In this way, business logic can be applied to the ensemble model by including external feature values in the ensemble model's training data set.

The learning unit 230 may train an ensemble model based on a decision tree (Decision Tred) including a plurality of nodes based on the generated training data set. Through this, the ensemble model can be any decision tree based on information gain.

Here, a decision tree is a classification and prediction algorithm that determines the splitting criteria from the attributes of the data and models it in a tree form according to the splitting criteria. A decision tree determines the split location and the attribute to be selected as a decision node at every step of forming a tree structure from the parent node to the child node. To determine this, the decision node calculates information gain (information gain, a standard indicator when learning a decision tree) using [Equation 1].

[Equation 1]

here, is the information gain that can be obtained by branching feature A, is the entropy value at the node before branching, is the entropy value calculated after branching feature A, is a function that imposes a penalty according to depth.

[Equation 1] where X is the depth of the decision tree, and when X is a natural number, it can be any function that has a monotonically increasing characteristic. for example, , is set by a person and can always be a positive number greater than 1.

Entropy, mentioned in the process of calculating information gain, is an indicator that measures how much deviation the data has.

Decision trees allow people to easily understand the internal logic, but as the depth of the decision tree increases, it becomes more difficult for people to intuitively understand the decision tree. Additionally, as the depth of the decision tree increases, the possibility of the ensemble model overfitting the learning data increases. To solve this problem, the present invention can limit the decision tree from becoming too deep by adding a penalty function proportional to the depth of the decision tree.

The decision tree can calculate information gain for external feature values included in multiple learning data sets.

Decision trees depend on the depth of the node to which they branch. can be given. At this time, the deeper the depth of the node, the larger the value of P(x) becomes, and the more difficult it is to branch the decision tree.

A decision tree can branch nodes based on the external feature value with the greatest information gain.

The learning unit 230 may terminate training of the ensemble model when there are no more target nodes to branch from in the decision tree.

For example, if there is no node with an information gain greater than a preset threshold, the learning unit 230 may terminate training of the ensemble model.

For example, the learning unit 230 may terminate training of the ensemble model when all learning data sets are classified into the same type of class or when there is no learning data set to be classified.

The learning unit 230 may add a new node corresponding to a manual rule to the decision tree and train an ensemble model based on the decision tree to which the new node was added.

For example, referring to Figure 3, the learning unit 230 adds a manual rule 32 that divides 'A' into a case where 'A' is in the input sentence and a case where it does not, to a specific node of the decision tree 30. An ensemble model can be trained based on the decision tree 30 to which the first new node and the second new node 34 corresponding to the rule 32 are added.

At this time, the manager evaluates the ensemble model based on the decision tree (30) before the manual rule (32) is added (e.g., entropy value: 0.2345, classification accuracy value: 88.75%) and the manual rule (32). The evaluation results (e.g., entropy value: 0.1254, classification accuracy value: 90.12%) for the ensemble model based on the decision tree (30) after this addition can be compared.

For example, the learning unit 230 may train the ensemble model after adding a new node corresponding to a manual rule to the decision tree before, after, or in the middle of training the ensemble model.

For example, if new nodes corresponding to manual rules are added to the decision tree before training the ensemble model, the administrator can add new nodes corresponding to manual rules to the decision tree in the form of a tree. At this time, the learning unit 230 may learn an ensemble model using the decision tree to which new nodes corresponding to manual rules are added as a starting point.

For example, if you want to add a new node corresponding to a manual rule to the decision tree in the middle of training an ensemble model, a new node corresponding to a manual rule can be added to the learned decision tree. At this time, the learning unit 230 may learn an ensemble model using the decision tree to which new nodes corresponding to manual rules are added as a starting point.

For example, if a new node corresponding to a manual rule is added to the decision tree after training the ensemble model, a new node corresponding to the manual rule may be added to the learned decision tree.

The learning unit 230 may calculate the entropy value and classification accuracy value before and after a new node corresponding to a manual rule is added to the decision tree. This allows managers to compare classification accuracy values and entropy values of ensemble models based on decision trees that change by adding manual rules. Through this, you can understand how manual rules added by administrators affect the overall data.

The learning unit 230 may select the final ensemble model based on information about the ensemble model learned after adding the new node and information about the ensemble model before the new node was added.

For example, referring to FIG. 4, the learning unit 230 evaluates the ensemble model based on the first decision tree before manual rules are added (e.g., entropy value: 0.2345, classification accuracy value: 88.75%) ) and the second decision tree (40) based on the evaluation results for the ensemble model based on the second decision tree (40) with added manual rules (e.g., entropy value: 0.1254, classification accuracy value: 90.12%). It can be selected as the final ensemble model.

For example, the learning unit 230 displays visualization results (or classification results for a specific sample) for an ensemble model based on a decision tree before manual rules are added and an ensemble model based on a decision tree with manual rules added. The final ensemble model can be selected based on the visualization results for the model (or classification results for specific samples).

The receiving unit 200 may receive query data, external feature values for the query data, classification results for the query data output from a plurality of single classification models, and reliability for the classification results. Here, the query data may include the content of speech uttered by the user. External feature values for query data may include information about business logic corresponding to the business included in the query data.

The data generator 210 may generate a learning data set based on classification results for the query data, reliability for the classification results, and external feature values for the query data.

The derivation unit 220 may input the generated learning data set into a pre-trained ensemble model and derive an output value for the query data through the pre-trained ensemble model. Here, the output value for the query data may be a classification result derived through an ensemble model.

Meanwhile, those skilled in the art will fully understand that the receiving unit 200, the data generating unit 210, the deriving unit 220, and the learning unit 230 may be implemented separately, or one or more of them may be integrated and implemented. .

Figure 5 is a flowchart showing a method of managing an ensemble model for a plurality of single models according to an embodiment of the present invention.

Referring to FIG. 5, in step S501, the ensemble model management server 110 determines query data, external feature values for the query data, classification results for query data output from a plurality of single classification models, and reliability for the classification results. can receive.

In step S503, the ensemble model management server 110 may generate a learning data set based on the classification result for the query data, the reliability of the classification result, and the external feature values for the query data.

In step S505, the ensemble model management server 110 may input the generated training data set into a pre-trained ensemble model to derive an output value for the query data.

In the above description, steps S501 to S505 may be further divided into additional steps or combined into fewer steps, depending on the implementation of the present invention. Additionally, some steps may be omitted or the order between steps may be changed as needed.

One embodiment of the present invention may also be implemented in the form of a recording medium containing instructions executable by a computer, such as program modules executed by a computer. Computer-readable media can be any available media that can be accessed by a computer and includes both volatile and non-volatile media, removable and non-removable media. Additionally, computer-readable media may include all computer storage media. Computer storage media includes both volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data.

The description of the present invention described above is for illustrative purposes, and those skilled in the art will understand that the present invention can be easily modified into other specific forms without changing the technical idea or essential features of the present invention. will be. Therefore, the embodiments described above should be understood in all respects as illustrative and not restrictive. For example, each component described as unitary may be implemented in a distributed manner, and similarly, components described as distributed may also be implemented in a combined form.

The scope of the present invention is indicated by the claims described later rather than the detailed description, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be construed as being included in the scope of the present invention. .

100: Multiple single model management servers
110: Ensemble model management server
200: Receiving unit
210: data generation unit
220: Derived part
230: Learning Department

Claims (15)

In the server that manages ensemble models for multiple single models,
a receiving unit that receives query data, external feature values for the query data, classification results for the query data output from a plurality of single classification models, and reliability for the classification results;
a data generator that generates a learning data set based on the classification results, the reliability, and the external feature values; and
A derivation unit that inputs the generated training data set into a pre-trained ensemble model and derives an output value for the query data.
An ensemble model management server comprising:
According to claim 1,
The data generator generates a learning data set based on classification results for input data output from the plurality of single classification models, reliability for the classification results, and external feature values for the input data. Model management server.
According to claim 2,
The external feature value includes information about business logic, and the ensemble model management server
According to claim 2,
A learning unit that trains the ensemble model based on a decision tree including a plurality of nodes based on the generated training data set.
An ensemble model management server further comprising:
According to claim 4,
The ensemble model management server wherein the learning unit reflects a penalty imposing function based on the depth value of a branch target node among the plurality of nodes in the information gain calculation formula of the decision tree.
According to claim 4,
The learning unit adds a new node corresponding to a manual rule to the decision tree, and trains the ensemble model based on the decision tree to which the new node is added.
According to claim 6,
The learning unit selects a final ensemble model based on information about the ensemble model learned by adding the new node and information about the ensemble model before the new node is added.
In a method of managing an ensemble model for a plurality of single models by an ensemble model management server,
Receiving query data, external feature values for the query data, classification results for the query data output from a plurality of single classification models, and reliability for the classification results;
generating a learning data set based on the classification result, the reliability, and the external feature value; and
Inputting the generated training data set into a pre-trained ensemble model to derive an output value for the query data
An ensemble model management method comprising:
According to claim 8,
The step of generating the learning data set is
An ensemble comprising the step of generating a learning data set based on classification results for input data output from the plurality of single classification models, reliability for the classification results, and external feature values for the input data. How to manage models.
According to clause 9,
An ensemble model management method, wherein the external feature value includes information about business logic.
According to clause 9,
An ensemble model management method further comprising training the ensemble model based on a decision tree including a plurality of nodes based on the generated training data set.
According to claim 11,
The step of learning the ensemble model is
An ensemble model management method comprising reflecting a penalty imposition function based on the depth value of a branch target node among the plurality of nodes in the information gain calculation formula of the decision tree.
According to claim 11,
The step of learning the ensemble model is
An ensemble model management method comprising adding a new node corresponding to a manual rule to the decision tree and learning the ensemble model based on the decision tree to which the new node was added.
According to claim 13,
The step of learning the ensemble model is
An ensemble model management method comprising selecting a final ensemble model based on information about the ensemble model learned after the new node is added and information about the ensemble model before the new node is added.
A computer program stored on a computer-readable recording medium comprising a sequence of instructions for managing an ensemble model for a plurality of single models, comprising:
When the computer program is executed by a computing device,
Receiving query data, external feature values for the query data, classification results for the query data output from a plurality of single classification models, and reliability for the classification results,
Generate a learning data set based on the classification result, the reliability, and the external feature value,
A computer program stored in a computer-readable recording medium, comprising a sequence of instructions for inputting the generated training data set to a pre-trained ensemble model to derive an output value for the query data.

Images