KR20230054167A - Method for generating machine learning model and device therefor - Google Patents

Abstract

Disclosed are a generating method of a machine learning model and a device thereof. A model generating device clusters a plurality of data samples into a plurality of clusters, searches for representative data sample of each cluster, and trains a machine learning model with semi-supervised learning using labeled or scored data sample after input-receiving the label or score for the representative data sample. Therefore, the present invention is capable of providing the machine learning model that can perform regression analysis or classification.

Description

Method for generating machine learning model and device therefor}

An embodiment of the present invention relates to a method and apparatus for generating a machine learning model, and more particularly, to a method and apparatus for generating a machine learning model using unlabeled data. will be.

Most machine learning is being done in the field of supervised learning. For supervised learning training, a target variable (ie, a target variable) is required in the training data. However, target variables do not exist in most existing data. That is, since most of the data is not labeled, a complex data processing process for labeling is required for supervised learning.

Unsupervised learning can be used for visualization and understanding of data structures. Unsupervised learning has the advantage of not requiring a target variable. However, since there is no target variable, very different clustering results may be obtained for the same dataset according to data scientists or domain experts. For example, users can influence key parameters such as the number of clusters, a distance scale for comparing data samples, criteria for evaluating clustering results, and interpretation of clustering results.

There are several approaches that use unsupervised learning for feature engineering of supervised learning. This approach assumes that data samples with the same cluster label have a similar relationship with the target value of the supervised learning model, so it is difficult to identify the number of clusters and other parameters as in unsupervised learning. many.

Semi-supervised learning is used to create regression or classification models by training them on datasets with partially labeled samples. Semi-supervised learning has a much smaller number of labeled samples than supervised learning. Therefore, when traditional supervised learning approaches are applied to these datasets, it is very difficult to find reliable relationships between input variables and target variables (regression problems) or good decision boundaries between clusters (or classes) (classification problems).

A technical problem to be achieved by an embodiment of the present invention is to generate a machine learning model capable of performing regression analysis or classification targeted by the user based on unlabeled data and limited information obtained from the user. It is to provide a method and an apparatus therefor.

An example of a method for generating a machine learning model according to an embodiment of the present invention for achieving the above technical problem includes clustering a plurality of data samples into a plurality of clusters; Searching for representative data samples of each cluster; Receiving a label or score for a representative data sample; and training the machine learning model by semi-supervised learning using data samples to which labels or scores have been assigned.

An example of a model generating device according to an embodiment of the present invention for achieving the above technical problem is a clustering unit for classifying a plurality of data samples into a plurality of clusters; a sample search unit that searches representative data samples of each cluster; A labeling unit that receives labels or scores for representative data samples; and a learning unit that trains the machine learning model through semi-supervised learning using data samples to which labels or scores have been assigned.

According to an embodiment of the present invention, a machine learning model capable of performing a regression analysis or classification targeted by a user with a high level of accuracy can be generated based on unlabeled data and limited information obtained from the user.

1 is a schematic diagram showing the configuration of an example of a model generating device according to an embodiment of the present invention;
2 is a flowchart showing an example of a method for generating a machine learning model according to an embodiment of the present invention;
3 is a diagram showing an example of clustering according to an embodiment of the present invention;
4 and 5 are diagrams showing an example of a method for searching for a representative data sample according to an embodiment of the present invention;
6 is a diagram showing an example of a semi-supervised learning method according to an embodiment of the present invention;
7 is a diagram showing another example of a semi-supervised learning method according to an embodiment of the present invention;
8 is a diagram showing an example of a method for evaluating a supervised learning model of semi-supervised learning according to an embodiment of the present invention, and
9 is a diagram showing the configuration of an example of a model generating device according to an embodiment of the present invention.

Hereinafter, a machine learning model generation method and device according to an embodiment of the present invention will be described in detail with reference to the accompanying drawings.

1 is a diagram briefly showing the configuration of an example of a model generating device according to an embodiment of the present invention.

Referring to FIG. 1 , the model generating device 100 trains and generates a machine learning model 120 using an unlabeled dataset 110 . The machine learning model 120 may be a regression analysis model or a classification model desired by a user. In general, unsupervised learning is applied to the dataset 110 in which there is no value for the target variable (ie, unlabeled dataset). However, since there is a problem of low accuracy of the model trained by unsupervised learning, the present embodiment uses the unlabeled dataset 110 and the machine learning model ( 120) is presented by training. This will be examined in detail below in FIG. 2 .

2 is a flowchart illustrating an example of a method for generating a machine learning model according to an embodiment of the present invention.

Referring to FIGS. 1 and 2 together, the model generator 100 clusters a plurality of data samples constituting the data set 110 into a plurality of clusters (S200). Here, the data sample may be data composed of variable values for a plurality of variables. The model generator 100 can perform clustering on various unsupervised learning models (eg, k-means, etc.), and an example thereof is shown in FIG. 3 .

The model generator 100 searches for representative data samples for a plurality of clusters (S210). The representative data sample means a data sample that can best reflect the characteristics of a plurality of clusters. For example, the model generator 100 may search for a representative data sample for each cluster based on data density. Examples of specific methods of searching for representative data samples are shown in FIGS. 4 and 5 .

The model generator 100 receives a label or score for the representative data sample from the user (S220). For example, in the case of generating the machine learning model 120 that classifies data into N clusters, the user selects a representative data sample among the N clusters and assigns a label identifying the cluster to which representative data sample corresponds. can be given to Or, if you want to create a machine learning model for regression analysis that predicts the relationship between input and output values, the user can give a score (for example, a value between 0 and 1) to which output value the representative data sample corresponds to. can In addition to this, there are various conventional labeling methods, and a method of assigning a label or a score according to the labeling method may be applied to this embodiment.

The model generator 100 may visualize and display the representative data sample using a table, various charts, or plots so that the user can help when assigning a label or score to the representative data sample. For example, when displaying a plurality of data samples belonging to a plurality of clusters using a data table, a parallel coordinate diagram, or a projection plot, the model generator 100 classifies representative data samples and displays them together, so that the user can view the representative data. It makes it easy to determine whether a sample has been classified into a corresponding cluster and assign a label or score.

The model generating device 100 trains the machine learning model 120 using a semi-supervised learning method using data samples to which labels or scores have been assigned (S230). That is, the model generator 100 performs semi-supervised learning using a dataset to which labels or scores are assigned only to representative data samples. The model generator 100 may perform semi-supervised learning using multi-view learning in order to increase the accuracy of semi-supervised learning, and examples thereof are shown in FIGS. 6 to 8 .

3 is a diagram illustrating an example of clustering according to an embodiment of the present invention.

Referring to FIG. 3 , the model generator 100 may classify a dataset 110 composed of a plurality of data samples into a plurality of clusters 310 using an unsupervised learning model 300 . The number of clusters 310 may be preset or automatically set by a user or the like. As another example, the model generating apparatus 100 may create a plurality of clusters desired by the user by reflecting the user's feedback on clustering. For example, a plurality of clusters may be generated using the method disclosed in Patent Application No. 10-2020-0163344 "Method for Reflecting User Intention in Unsupervised Learning and Apparatus Therefor".

4 and 5 are diagrams illustrating an example of a method of searching for a representative data sample according to an embodiment of the present invention.

Referring to FIGS. 4 and 5 together, the model generator 100 may search for N vectors by performing vector quantization on a plurality of data samples (S400). For example, when a data sample is composed of variable values for m variables, each data sample may be represented as an m-dimensional vector (ie, a feature vector). The range of each variable value of the data sample may be normalized and displayed as a vector. In addition, various conventional methods of displaying the data sample as a vector may be applied to this embodiment. Vector quantization is a process of mapping K feature vectors into N (<k) vectors, and the number of N vectors can be set by a user or automatically. For example, K data samples existing in the dataset are represented by k feature vectors, and N vectors can be searched from the K feature vectors through vector quantization. Since the vector quantization method itself is already widely known in the field of data mining, an additional description thereof will be omitted.

When N vectors are searched for, the model generator 100 selects a representative data sample based on the vectors (S410). Assuming the hypothesis that the distance between clusters should be greater than the distance between data samples within a cluster, it is desirable to find a representative data sample based on a point with a high local density of data points. For example, the model generating apparatus 100 identifies feature vectors located closest to the N vectors, and selects a data sample corresponding to the feature vector as a representative data sample. That is, N representative data samples may be extracted from N vectors.

The model generator 100 receives a label or score from the user for the representative data sample (S420). The model generator 100 may provide a user interface through which a label or score may be input from a user. For example, the model generator 100 classifies and displays data samples of each cluster of the data set through a projection plot, etc., and also classifies and displays representative data samples, so that the user can determine the appropriateness for the representative data samples. It can help assign labels or scores.

As another embodiment, the model generator 100 may also receive a reliability level when receiving a label or a score from a user. For example, the model generating device 100 has a numerical variable form between 0.1 and 1 (0.1: low reliability, 1: high reliability), a nominal variable form having an eigenvalue (eg, low, medium, high, etc.) ) and fuzzy sets (for example, low, medium, high, etc.).

The model generator 100 terminates the process of searching for representative data samples if the predefined stop condition is met, and otherwise repeats the process of searching for representative data samples (S430).

When the model generator 100 repeats the search process of the representative data sample, the data set is not used as it is, but the representative data sample and the data samples around it are removed from the data set and used (S440). For example, the model generating apparatus 100 may remove at least one or more neighboring data samples in order of proximity to the representative data sample. In this case, whether or not the distance is close can be determined using a distance between feature vectors of data samples (eg, Euclidean distance). By removing the representative data sample and its neighboring data samples, it is possible to prevent the same representative data sample from being searched for again when the representative data sample is repeatedly performed.

The model generator 100 removes the representative data sample and its neighboring data samples, and repeats the process of searching for N vectors again for the remaining data samples (S400 to S420) until the stop condition is satisfied.

A stop condition for determining whether to terminate the process of searching for a representative data sample may be set in various ways according to embodiments. For example, the stopping condition may be a case in which at least one representative data sample is selected for each cluster. When there are N clusters as shown in FIG. 3 , the model generator 100 performs a first search process for representative data samples and determines whether representative data samples for all N clusters are all searched. For example, if representative data samples are not searched for in clusters 4, 6, and 9, the model generator 100 performs a second search process for representative data samples, and as a result, clusters 4, 6, and Determine if all representative data samples for 9 have been searched. If representative data samples for clusters 4, 6, and 9 are all searched, the model generator 100 ends the search process for representative data samples, and otherwise repeats the search process for representative data samples.

Another example of a stopping condition is when the number of representative data samples is greater than or equal to a pre-defined number, and when at least one label of a pre-defined label set (that is, each label of a cluster that the user wants to classify) is given to at least one data sample. , when the minimum and maximum values of predefined scores (ie, the range of predicted values in the regression analysis model) are respectively assigned to at least one data sample, or when a request to stop is received from the user. In addition to this, various stop conditions may be set according to embodiments.

6 is a diagram showing an example of a semi-supervised learning method according to an embodiment of the present invention.

Referring to FIG. 6 , the model generator 100 trains the machine learning model 120 by using semi-supervised learning in a state where labels or scores are assigned to only some data samples in the dataset. In order to increase the accuracy of semi-supervised learning, this embodiment uses a plurality of supervised learning models 600 (ie, machine learning algorithms).

First, the model generator 100 trains a plurality of supervised learning models 600 using data samples to which labels or scores are assigned. In addition, the model generator 100 predicts labels or scores by inputting the unlabeled data samples 610 into the trained plurality of supervised learning models 600 .

The model generator 100 assigns a label or score determined through an agreement between the labels or scores predicted by the plurality of supervised learning models 600 to the data sample (620). For example, suppose there are five supervised learning models. In this case, if the 1st, 2nd, and 5th supervised learning models output label A and the 3rd and 4th supervised learning models output label B for the first data sample, the model generating device outputs the first data sample according to a majority vote. Label A is assigned to In addition to this, various methods of determining a label or score to be assigned to a data sample, such as reflecting the prediction reliability of each supervised learning model, may be applied to this embodiment. For example, applying various methods, such as comparing the average of the prediction reliability of the 1st, 2nd, and 5th supervised learning models with the average of the prediction reliability of the 3rd and 4th supervised learning models, and assigning a higher label to the data sample. this is possible

The model generating device 100 retrains a plurality of supervised learning models using the newly labeled data samples, and then the second data samples through an agreement between the labels predicted by the plurality of supervised learning models for the second data samples. Repeat the process of assigning labels to . When labeling is completed for all data samples in this way, the model generating device 100 can train and generate a machine learning model using the supervised learning method using the labeled dataset.

The process of the model generating device 100 evaluating each supervised learning model at each iterative learning process and adjusting the hyperparameters of the supervised learning model having a low evaluation score so that the plurality of supervised learning models 600 can more accurately predict labels or scores. can be performed. This will be reviewed again in FIG. 7 .

7 is a diagram showing another example of a semi-supervised learning method according to an embodiment of the present invention.

Referring to FIGS. 6 and 7 together, the model generator 100 trains a plurality of supervised learning models 600 using data samples to which labels or scores have been assigned (S700). The model generating apparatus 100 determines the label or score to be assigned to the data sample based on the prediction result of the plurality of supervised learning models for the data sample 610 to which no label or score has been assigned (S710).

The model generator 100 evaluates the supervised learning model based on the reliability value when each supervised learning model 600 predicts a label or a score (S720). To this end, the supervised learning model may be a model that outputs the reliability of the predicted value. For example, when a supervised learning model predicts a label or score, it can also output a prediction probability (ie, prediction reliability) of the corresponding label or score, and the prediction probability can be used as a reliability value in this embodiment.

The model generating device 100 may perform a process of assigning an evaluation score to each supervised learning model and updating the supervised learning model in order to evaluate the supervised learning model. For example, '0' is given as an initial evaluation score for the plurality of supervised learning models 600 . The initial evaluation score may be set to various values according to embodiments. The model generating device 100 increases the evaluation score of the supervised learning model with the highest reliability value when assigning a label or a score to the data sample 610 through the agreement of a plurality of supervised learning models 600 and trustworthy. Evaluation scores of supervised learning models whose values deviate from predefined standards may be reduced. For example, for the data sample 610, the first, second, and fifth supervised learning models output label A, and the third and fourth supervised learning models output label B, so that the data sample 610 has label A. In this case, the model generator 100 may increase the evaluation score of the supervised learning model having the highest reliability value among the first, second, and fifth supervised learning models that predicted the label A by '1'. As another example, an evaluation score of '-1' (ie, a decrease of '1') may be assigned to a supervised learning model whose reliability value deviate from a predefined standard. In this way, the model generator may update evaluation scores of a plurality of supervised learning models whenever a label or score is given to the data sample 610 .

The model generator 100 adjusts hyperparameters of the supervised learning model based on the evaluation scores of each supervised learning model (S730). For example, the model generator 100 adjusts hyperparameters of supervised learning models whose evaluation scores are equal to or less than a predefined reference value. The type of hyperparameter to be adjusted and the range of the adjustment value can be set in advance. As another example, the model generator 100 may adjust the hyperparameters by performing a process of optimizing the hyperparameters of the supervised learning model by applying various conventional hyperparameter optimization methods.

The model generator 100 repeatedly performs the above processes (S700 to S730) until the labeling or scoring of the data samples is completed (S740).

8 is a diagram showing an example of a method for evaluating a supervised learning model of semi-supervised learning according to an embodiment of the present invention.

Referring to FIG. 8, N supervised learning models (800, 802, 804, 806) are used for semi-supervised learning. When a plurality of supervised learning models (800, 802, 804, 806) output predicted values of labels or scores for data samples, the model generator 100 updates the evaluation score of each supervised learning model using the predicted reliability of each supervised learning model. .

For example, the model generator 100 may increase the evaluation score of the second supervised learning model 802 having the highest prediction reliability and decrease the evaluation score of the third supervised learning model 804 having the lowest prediction reliability. In addition to this, various methods of increasing or decreasing the evaluation score based on prediction reliability may be applied to this embodiment.

As another embodiment, the model generating device 100 may evaluate a supervised learning model using a data sample to which a special label or special score is assigned. To this end, when receiving a label or score for a representative data sample from the user (see S420 of FIG. 4 ), the model generator 100 may use a special label or special score indicating that the label or score cannot be assigned. there is. That is, the user can assign a general label or general score to the representative data sample, or assign a special label or special score indicating 'unknown' to the representative data sample if it is difficult to distinguish the label or score of the representative data sample. can

The model generator 100 trains a plurality of supervised learning models using data samples to which special labels/scores are assigned together with general labels/scores. As a result, the values predicted by the plurality of supervised learning models may be general labels/scores or special labels/scores.

The model generator 100 may decrease the evaluation score of the supervised learning model that predicted the special label/score for the data sample with high reliability (ie, -1).

9 is a diagram showing the configuration of an example of a model generating device according to an embodiment of the present invention.

Referring to FIG. 9 , the model generator 100 includes a clustering unit 900, a sample search unit 910, a labeling unit 920, and a learning unit 9930. The model generator 100 may be implemented as a computing device including a memory, a processor, and an input/output device. In this case, each configuration may be implemented as software, loaded into a memory, and driven by a processor.

The clustering unit 900 classifies a plurality of data samples into a plurality of clusters. An example of clustering by the clustering unit 900 is shown in FIG. 3 .

The sample search unit 910 searches for representative data samples of each cluster. The sample search unit 910 may search for a plurality of representative data samples based on data density. Specific examples of the sample search unit 910 are shown in FIGS. 4 and 5 .

The labeling unit 920 receives labels or scores for representative data samples. Specific examples of the labeling unit 920 are shown in FIGS. 4 and 5 . As another embodiment, the labeling unit 920 may receive a special label or special score indicating that a label or score cannot be assigned. An example of a method for evaluating a supervised learning model of semi-supervised learning based on special labels or special scores is shown in FIG. 8 .

The learning unit 930 trains the machine learning model through semi-supervised learning using data samples to which labels or scores have been assigned. The learning unit 930 may perform a semi-supervised learning process using a plurality of supervised learning models, examples of which are shown in FIGS. 6 and 7 . As another embodiment, the learning unit 930 may increase the accuracy of learning through a process of adjusting hyperparameters by evaluating a plurality of supervised learning models used in semi-supervised learning. An example of a method for evaluating a plurality of supervised learning models is shown in FIG. 8 .

Each embodiment of the present invention can also be implemented as computer readable codes on a computer readable recording medium. A computer-readable recording medium includes all types of recording devices in which data that can be read by a computer system is stored. Examples of computer-readable recording media include ROM, RAM, CD-ROM, SSD, and optical data storage devices. In addition, the computer-readable recording medium may be distributed to computer systems connected through a network to store and execute computer-readable codes in a distributed manner.

So far, the present invention has been looked at mainly with its preferred embodiments. Those skilled in the art to which the present invention pertains will be able to understand that the present invention can be implemented in a modified form without departing from the essential characteristics of the present invention. Therefore, the disclosed embodiments should be considered from a descriptive point of view rather than a limiting point of view. The scope of the present invention is shown in the claims rather than the foregoing description, and all differences within the equivalent scope will be construed as being included in the present invention.

Claims (12)

clustering a plurality of data samples into a plurality of clusters;
Searching for representative data samples of each cluster;
Receiving a label or score for a representative data sample; and
Training a machine learning model by semi-supervised learning using labeled or scored data samples. The method of claim 1, wherein the clustering step,
Classifying a plurality of data samples into a plurality of clusters using an unsupervised learning model. The method of claim 1, wherein the searching step,
Identifying a plurality of representative data samples based on the data density; a model generation method comprising the. The method of claim 1, wherein the searching step,
Repeating the process of removing the representative data sample and neighboring data samples located within a certain distance from the representative data sample and searching for a representative data sample with the remaining data samples until a predefined stop condition is satisfied; A method for generating a model comprising: The method of claim 4, wherein the stopping condition is,
When at least one data sample to which the label or score has been assigned exists in each of the plurality of clusters, when the number of representative data samples is greater than or equal to a predefined number, the labels of the predefined label set are at least one data sample, respectively. A method for generating a model, comprising a case in which a sample is assigned, a case in which a minimum value and a maximum value of predefined scores are respectively assigned to at least one data sample, or a case in which a stop request is received from a user. The method of claim 1, wherein the step of receiving the input,
A method for generating a model comprising the steps of visualizing and displaying the representative data sample to help input a label or score. According to claim 1,
The method of generating a model, characterized in that the label or score includes a special label or special score indicating that labeling cannot be performed. The method of claim 1, wherein the step of training the machine learning model,
training a plurality of supervised learning models using labeled or scored data samples;
assigning labels or scores to data samples through an agreement between labels or scores predicted by the plurality of supervised learning models for data samples to which labels or scores are not assigned;
adjusting evaluation scores for the plurality of supervised learning models based on reliability values of predictions of the plurality of supervised learning models;
adjusting hyperparameters of supervised learning models whose evaluation scores are less than a predefined standard; and
and repeating the training step to the adjusting step until labels or scores are assigned to all data samples. The method of claim 8, wherein the step of adjusting the evaluation score,
When a label or score is assigned to a data sample through the agreement of the plurality of supervised learning models, the evaluation score of the supervised learning model with the highest reliability value is increased and the supervised learning model whose confidence value exceeds the predefined standard Reducing the evaluation score; Model generation method characterized in that it comprises a. According to claim 8,
The label or score includes a special label or special score indicating that labeling cannot be performed,
The step of adjusting the evaluation score,
and reducing an evaluation score of a supervised learning model that predicts the special label or special score with a reliability value equal to or higher than a predetermined standard. a clustering unit that classifies a plurality of data samples into a plurality of clusters;
a sample search unit that searches representative data samples of each cluster;
A labeling unit that receives labels or scores for representative data samples; and
A learning unit for training a machine learning model by semi-supervised learning using labeled or scored data samples. A computer-readable recording medium recording a computer program for performing the method according to any one of claims 1 to 10 by a computer.

Images