JP2020038514A - Learning data generating device, learning data generating method, and program - Google Patents

Abstract

To solve the problem in which: to generate a highly accurate sorter, learning data suitable for machine learning algorithm must be collected and to generate good learning data from input data, data cleansing processing must be performed, and in particular, appropriate data cleansing is difficult because each machine learning algorithm has different characteristics in a method in which learning data is sorted on the basis of sorting results of the sorter generated by a plurality of machine learning algorithms.SOLUTION: A learning data generating device performs appropriate cleansing processing according to characteristics of each machine learning algorithm when generating a learning model using a plurality of different machine learning algorithms. This configuration makes it possible to generate learning data necessary for machine learning according to each machine learning algorithm.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a learning data generation device, a learning data generation method, and a program for generating learning data suitable for a machine learning algorithm from input data.

  In a classifier that classifies input data based on a learning model using a machine learning algorithm, a classifier having different advantages can be generated by using different types of machine learning algorithms. Therefore, by using a plurality of different classifiers, a plurality of classification results can be obtained, and the classification results can be used as a clue to improve the classification accuracy. As a method of further improving the classification accuracy by statistically processing the classification results by a plurality of classifiers, a method called ensemble learning or stacking has been developed.

By the way, in order to create a highly accurate classifier, learning data suitable for a machine learning algorithm constituting the classifier is required.
Therefore, it is necessary to preprocess input data so as to be suitable for learning by a machine learning algorithm. The pre-processing includes, for example, processing such as data normalization / standardization and generation of new data, and processing such as data filtering (refer to, for example, Patent Document 1) for removing inappropriate data (hereinafter, collectively referred to as “pre-processing”). Called “cleansing process”).

JP 2005-181928 A

  As described above, in order to generate a high-precision classifier, it is necessary to collect training data suitable for a machine learning algorithm. Have to do it.

  In particular, in a method of performing classification based on a classification result of a classifier generated by a plurality of machine learning algorithms, it is difficult to perform appropriate cleansing processing of data because the characteristics of each machine learning algorithm are different.

  The present invention includes an acquisition unit that acquires input data, a processing unit that performs a cleansing process on the input data to generate learning data, and a machine learning algorithm. Generating means for generating a learning model based on the learning data, wherein the generating means includes, as the machine learning algorithm, a plurality of different types of machine learning algorithms, and the processing means A first cleansing process is performed on data corresponding to each of the plurality of machine learning algorithms.

  According to the present invention, learning data necessary for generating a learning model using a plurality of different machine learning algorithms can be generated according to each machine learning algorithm. This can improve the efficiency of machine learning and the accuracy of classification estimation. In addition, common input data can be used regardless of the machine learning algorithm, and the effect of reducing the load on the user can be obtained.

FIG. 4 is a diagram illustrating a functional configuration of a learning data generation program. It is the flowchart which shows the cleansing process performed in a cleansing processing part, and an example of an algorithm database. It is an example of a personal computer operation log and a task list. It is a flowchart which shows the process which produces | generates a learning model. It is a flowchart which shows the classification estimation process with respect to input data. It is an example of an appearance phrase for each task and data after cleansing.

Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings. However, the components described in this embodiment are merely examples, and are not intended to limit the scope of the present invention to each embodiment.
<First embodiment>

<Functional configuration of learning data generation program>
FIG. 1 is a diagram showing a functional configuration for generating learning data in the classifier 100.
FIG. 1A shows the overall function of the classifier 100.
The common input data 110 is data used when generating a learning model for performing classification processing and performing classification estimation processing. The common input data 110 is collected as common data for generating learning data to be input to a plurality of classification estimating units 140 described later, and stored as a database.
The input unit 120 is a processing unit that acquires the common input data 110 as input data.

FIG. 1B shows the function of the cleansing processing unit 130.
The cleansing processing unit 130 is a processing unit for generating learning data (post-cleansing data 170) that is optimal for learning and classification estimation in the classification estimating unit 140 described below. As shown in FIG. 1B, the cleansing processing execution unit 132 of the cleansing processing unit 130 receives an input corresponding to the feature of the machine learning algorithm recorded in the algorithm database 131 (described later in FIG. 2C). Perform data cleansing.

The classification estimating unit 140 is a processing unit that classifies and determines learning data using a machine learning algorithm. A plurality of classification estimating units 140 are provided corresponding to the number of types of machine learning algorithms.
This is because, as described above, a classifier having different advantages can be generated by using a plurality of types of machine learning algorithms. Further, using a plurality of classification estimation results as clues leads to improvement in classification accuracy.

FIG. 1C shows a functional configuration of each classification estimating unit 140.
As shown in FIG. 1C, each classification estimating unit 140 includes a learning model generating unit 141 and a learning model executing unit 143.
The learning model generation unit 141 performs learning based on a machine learning algorithm using the learning data (the post-cleansing data 170) generated by the cleansing processing unit 130, and generates a learning model 142.
The learning model execution unit 143 performs a classification process based on the learning data and the learning model 142, and calculates an evaluation value of the learning data for each label to be classified. Then, the label having the largest evaluation value is output as the classification estimation result 144 of the learning data.

The output determination unit 150 integrally determines the classification estimation results 144 output from the plurality of classification estimation units 140, and determines a final output value.
For example, the classification estimation results 144 estimated by the classification estimation units 140 may be totaled, and the classification estimation result estimated most frequently may be output. Alternatively, the probabilities of the classification estimation results 144 may be compared, and the classification estimation result having the highest probability may be output. The determination method is not limited to these.
The output unit 160 presents the output value determined by the output determination unit 150 to the user by a predetermined method.

  The post-cleansing data 170 is data that becomes learning data after the cleansing processing unit 130 performs cleansing processing of input data corresponding to each of the classification estimating units 140, and is stored in the database. Details of the post-cleansing data 170 will be described later with reference to FIG.

<About the cleansing processing unit 130>
FIG. 2 is a flowchart illustrating an example of the input data cleansing process performed by the cleansing processing unit 130.

In S210 of the flowchart in FIG. 2A, the cleansing processing unit 130 acquires input data necessary for generating learning data from the common input data 110 via the input unit 120.
In S220, the cleansing processing unit 130 sequentially selects the classification estimation units 140 to be processed from the plurality of classification estimation units 140.

In S230, the cleansing processing unit 130 performs cleansing processing of the input data according to the machine learning algorithm, and generates post-cleansing data 170 as learning data. The cleansing process is, for example, a process of deleting data inappropriate for learning or processing (normalization / standardization, etc.) of data to increase learning efficiency. The details of the data cleansing process will be described later with reference to the flowchart of FIG.
In S240, cleansing processing section 130 checks whether or not all classification estimating sections 140 have been selected. Then, the processing of S220 to S240 is repeated until there is no unprocessed classification estimation unit 140.
When the learning data for all the classifying and estimating units 140 has been generated, the cleansing process in FIG. 2A ends.

<Example of use case for performing cleansing processing of input data>
Next, in order to show the details of the input data cleansing process performed in S230 of the flowchart of FIG. 2A described above, an example of a use case in which the input data cleansing process is required will be described.
In the following, a description will be given of an example of a use case in which the content of operation of a personal computer performed by a worker is used as input data, and each input data is classified into the task content (hereinafter, referred to as “task”) of the worker. Here, it is assumed that the tasks of the worker are classified into projects such as a project A (regular meeting), a project B (creation of materials), and a project C (morning meeting), which are performed in a company. Needless to say, the assumed use cases and data to be used are not limited to these.

FIG. 3A shows an example of a personal computer operation log as input data.
As shown in FIG. 3A, the operation log 310 of the personal computer is data in which the details of the operation of the personal computer performed by the worker are recorded in time series. The operation log 310 of the personal computer includes field information such as ID 311, time 312, application name 313, operation target information 314, operation content 315, input key information 316, cursor position 317, file property 318, and the like.
For example, in the case of a document document, information such as a path name where a file is stored and a file name are stored in the operation target information 314. In the case of Web browsing, the URL and Web page title are stored, and in the case of mail software, information such as the transmission destination and subject are stored, respectively.

  The field information collected as the operation log is not limited to these. For example, it is possible to collect the entire text of a document document or Web page, the text of an e-mail, the name of an attached file, and GUI part information (menu item names, button names, etc.) with mouse operation, and add it as new field information. It is.

FIG. 3B is an example of a list of tasks in which the personal computer operation log 310 shown in FIG. 3A is classified.
The task list 320 is data in which a history of tasks performed by the worker is recorded in chronological order. The task list 320 is used when generating the learning model 142.
The task list 320 includes field information such as an ID 321, a start time 322, an end time 323, and a task name 324. Then, by referring to the personal computer operation log 310 of FIG. 3A and the task list 320 of FIG. 3B, the personal computer operation log within the execution time of each task (from the start time 322 to the end time 323) is obtained. The record 310 is associated with the task that is the correct answer label.

  Based on the personal computer operation log 310 corresponding to the correct label obtained in this way, the classification estimating unit 140 performs the task classification of the input data. Then, when estimating the task classification, when the personal computer operation log to be estimated is input, the task with the highest reliability can be estimated by the learned classifier. In addition, since various classification methods used as the machine learning when learning the classifier can be considered, it is necessary to generate appropriate learning data according to the machine learning algorithm used for each classification method.

<Example of machine learning algorithm used for task classification>
Various well-known methods that focus on the state of elements included in the input data can be applied to the machine learning algorithm for classifying the input data composed of the personal computer operation log 310 as described above.
Here, the elements included in the input data are, specifically, words included in the personal computer operation log 310 (for example, “spreadsheet software”, “YYY” described later with reference to FIG. 6). The state of an element is, for example, “word appearance frequency”, “word appearance pattern”, “word appearance order (continuity)”, and the like. Each machine learning algorithm has a feature of what element state is focused on.

An algorithm called BoW (Bags of Words) or SVM (Support Vector Machine) can be used as a machine learning algorithm that focuses on the frequency of appearance of words.
BoW is a method of counting the number of appearances of words and phrases appearing in data associated with each correct answer label, and learning a characteristic word for each label. In this case, it is desirable to perform a cleansing process on the input data so that words that appear at a high frequency in common are excluded and words that appear at a high frequency only at each correct label are collected.
The SVM is a technique of calculating a feature vector of a word appearing in an operation log and generating a classifier that finds a boundary for classifying two labels based on a relationship between a correct label and a feature vector. Then, classifiers for all correct answer labels are generated, and those having the highest evaluation values output by the respective classifiers are used as the classification results. Also in this case, it is desirable that the input data be subjected to cleansing processing to learning data in which words having feature vectors that appear at a high frequency only in each correct label are collected.

As a machine learning algorithm that focuses on the feature pattern of a word, an algorithm called CNN (Convolution Neural Network) often used in image classification can be used.
The CNN is a method of treating a predetermined time or a predetermined number of records as one image data and performing learning so as to extract a feature associated with a correct label. In this case, it is desirable to perform a cleansing process on the input data into learning data in which features appear concentrated in a series of record data.

As a machine learning algorithm that pays attention to the continuity of input data, an algorithm that handles time-series data, such as is often used in automatic translation and interactive sentence generation, can be used. For example, a machine learning algorithm such as RNN (Recurrent Neural Network) or LSTM (Long Short Term Memory) corresponds to this.
Both RNN and LSTM are methods for learning the order of the records that have been worked on and learning the characteristics of the work procedure performed for each task. In this case, it is desirable to perform a cleansing process on the input data into learning data that makes it easy to grasp a rough work procedure.

<Details of input data cleansing process>
The details of the input data cleansing process performed by the cleansing processing unit 130 when performing the task classification using the above-described machine learning algorithm will be described with reference to FIG.

  In step S231 of the flowchart of FIG. 2B, the cleansing processing unit 130 shows the characteristics of the machine learning algorithm used in the classification estimating unit 140 selected in step S220 of FIG. From the algorithm database 131 to be executed.

In S232, the cleansing processing unit 130 determines whether or not the machine learning algorithm is a method of performing classification based on the appearance frequency of words appearing in the personal computer operation log 310.
If the machine learning algorithm corresponds to this (for example, if it is Bow or SVM), the process proceeds to S233. If not, the process proceeds to S234.

In S233, the cleansing processing unit 130 executes a cleansing process for characterizing the frequency of occurrence of the word.
In the case of Bow or SVM described above, data processing such as deleting data common to a plurality of correct labels is performed as cleansing processing in order to give characteristics to the frequency of appearance in correct labels.

For example, using the input data shown in FIGS. 3A and 3B, the number of appearances of a phrase that appears for each task is counted, and a list of phrases that appear for each task is generated.
FIG. 6A is an example of an appearance phrase list generated for each task. (A1) is a list of appearing phrases for project A (regular meeting), and (a2) is a list of appearing phrases for project B (material creation). In the appearing phrase list, the appearing phrase 332 and the number of appearances 333 are linked and recorded.
From this information, it can be seen that the terms “spreadsheet software” and “data” appear in common for the tasks of project A and project B. Therefore, from the task of the project A, the appearance words whose IDs 331 are ta01 (spreadsheet software) and ta04 (data) are deleted. Also, the terms appearing in tb03 (spreadsheet software) and tb04 (data) are deleted from the task of project B.

  As a result, as shown in FIG. 6B, only words highly relevant to each task can be extracted from the appearing phrase list as shown in FIG. Here, (b1) is a list of appearing phrases after the cleansing process is performed on the project A (regular meeting), and (ba2) is a list of appearing phrases after the cleansing process is performed on the project B (material creation).

In S234, the cleansing processing unit 130 determines whether or not the machine learning algorithm is a method of performing classification based on the characteristic pattern of the operation log content within a certain period of time.
If the machine learning algorithm corresponds to this (for example, if it is CNN), the process proceeds to S235. If not, the process proceeds to S236.

In S235, the cleansing processing unit 130 executes a cleansing process that makes it easier to detect the characteristic pattern.
In the case of the CCN described above, in order to make it easier to detect a characteristic pattern, data processing such as how to arrange field information and selection of an appropriate record is performed as cleansing processing.

For example, in the personal computer operation log 310 shown in FIG. 3A, data analysis software is used as an application for records with IDs 311 of a007 and a009 to a011. However, in the middle of the operation, an operation (record with ID a008) for the file viewer is performed in a single short time. By deleting such a single short record, features can be concentrated.
Alternatively, data normalization processing such as increase / decrease in the number of data according to work time can be performed so that the weight of a record that is processed in a short time and the weight of a record that is processed for a long time are considered. . For example, in the personal computer operation log 310 of FIG. 3A, the record of a007, which is a work time of 5 minutes, is weighted according to the work time as compared with the record of a006, which is work time of 5 seconds. You can also. As a result, it is possible to assign the feature of the weight of the work time, instead of treating records having different work times equally as one record.
As a result, data as shown in FIG. 6C is generated as the data subjected to the cleansing process.

In S236, the cleansing processing unit 130 determines whether or not the machine learning algorithm is a method of performing classification based on the continuity of operation logs (record order).
If the machine learning algorithm corresponds to this (for example, if it is RNN or LSTM), the process proceeds to S237. If not, the process proceeds to S238.

In S237, the cleansing processing unit 130 executes a cleansing process that makes it easier to see the tendency of continuity.
In the case of RNN or LSTM described above, in order to be able to roughly extract the characteristics of the work procedure, data such as performing thinning-out processing to limit the number of consecutive appearances of records to the same operation target as cleansing processing Perform processing.

For example, in the personal computer operation log 310 shown in FIG. 3A, in the records having IDs a009 to a011 of ID311, the operation to the data analysis software is continuously performed for about 90 seconds. On the other hand, if the thinning is performed so that the operation in a predetermined time (for example, one minute) is performed once, the record of a010 is deleted, so that the number of continuous appearances can be adjusted. Further, a record such as a006 in which the processing time is shorter than the predetermined time may be deleted, and then the thinning processing may be performed.
As a result, data as shown in FIG. 6D is generated as the data subjected to the cleansing process.

In S <b> 238, the cleansing processing unit 130 associates the data cleansed by the processing up to S <b> 237 with the machine learning algorithm of the classification estimating unit 140 and stores the data as the post-cleansing data 170.
When the above cleansing process is completed, the process returns to S240 of FIG. 2A and the process is restarted.

The task classification using the personal computer operation log 310 as input data has been described above as an example. However, as input data, in addition to this, information on worker's actions such as worker's conversation, call history, action history and action schedule is recorded in various formats including PC operation log, recording and recording. Data may be used.
Also, algorithms for classifying input data, viewpoints for learning with each algorithm, data cleansing methods, and the like are not limited to those described above, and can correspond to various situations.

<About the learning model generation unit 141>
FIG. 4 is a flowchart illustrating a process of generating the learning model 142 in the learning model generation unit 141 of the classification estimation unit 140.

  In S410, the learning model generation unit 141 of the classification estimation unit 140 acquires the post-cleansing data 170 generated by the cleansing processing unit 130 as learning data.

In S420, the learning model generation unit 141 generates a learning model using a machine learning algorithm based on the learning data and a task assigned to each learning data.
The learning model is a model for estimating the evaluation values of all the tasks to be classified with respect to one learning data or a plurality of learning data within a predetermined time or a predetermined amount. By associating the input data with the task and learning, it is possible to estimate the evaluation value of each task even for unknown input data.

In S430, the learning model generation unit 141 determines whether or not the accuracy of the classification estimation using the learning model generated in S420 is sufficient.
If it is determined that the classification estimation accuracy is sufficient, the process proceeds to S440. If it is determined that the estimation accuracy is insufficient, the process ends.
In order to determine the classification estimation accuracy, for example, a cross-validation technique can be used. In the cross-validation method, the learning data is divided into learning data and verification data, and a learning model is generated based on the learning data assigned as the learning data. Then, the accuracy of the learning model is calculated by determining the rate at which the learning data assigned as the verification data is output as the correct task.

In S440, the learning model generation unit 141 updates the learning model generated in S430 as the learning model 142 for task classification.
Thereby, the flowchart of FIG. 4 ends.

<About the learning model execution unit 143>
FIG. 5 is a flowchart showing a classification estimation process for input data acquired from the input unit 120 in the learning model execution unit 143 of the classification estimation unit 140.

  In S510, the learning model execution unit 143 acquires the learning data to be analyzed generated by the cleansing processing unit 130 from the post-cleansing data 170.

In S520, the learning model execution unit 143 estimates a task to be classified using the learning model 142 generated in advance for the acquired learning data to be analyzed.
When estimating the tasks to be classified, the learning data to be analyzed and the evaluation value indicating the degree of association with each task to be classified are output together. Note that one task may be estimated for one input data, or one task may be estimated for a plurality of pieces of input data within a predetermined time or a predetermined amount.

  In S530, based on the evaluation value estimated in S520, the learning model execution unit 143 determines the task having the largest evaluation value as the task estimation result for the learning data to be analyzed, and as the classification estimation result 144 Output.

As described above, in the present embodiment, the means for performing classification determination using a plurality of machine learning algorithms executes cleansing processing of input data according to each machine learning algorithm.
Thereby, learning data suitable for the machine learning algorithm can be generated. In addition, since it is possible to reduce the trouble of collecting input data in accordance with a plurality of machine learning algorithms, it is possible to reduce the time required for generating a learning model and improve the accuracy.

<Second embodiment>
In the first embodiment, a data cleansing process is provided in which input data is appropriately cleaned for each machine learning algorithm to generate learning data.
On the other hand, the second embodiment further provides a common data cleansing process for cleansing inappropriate input data common to all machine learning algorithms.

Also in the second embodiment, the task classification using the personal computer operation log 310 shown in FIG. 3A and used in the first embodiment will be described as an example.
In the personal computer operation log 310 shown in FIG. 3A, there are records that do not include valid field information (missing records) and records that appear in common to all tasks (duplicate records). . Therefore, common data cleansing processing for deleting such a missing record or a duplicate record is performed.

For example, in the record in which the ID 311 is a005 in FIG. 3A, it is known that the mail software is newly created, but the transmission destination and the subject are unknown. Therefore, this record is determined to be invalid and is deleted.
Also, a cleansing process such as deleting a record (designated record) containing a predetermined word (keyword) designated in advance, or assigning the designated record to a specific task may be performed.

  When a task which is a correct answer label is assigned to the personal computer operation log 310, the task is assigned to records within a certain time, for example, "9:00 to 10:00: Project A work". Sometimes. However, actually, there are many cases in which the operator performs another operation due to interruption during this time. Common cleansing processing such as detecting and deleting in advance records relating to tasks that are not clearly related to each task may be performed.

Note that the common data cleansing process is not limited to these, and various methods such as a process of deleting a record having an execution time shorter than a predetermined time can be used. Further, as in the first embodiment, the assumed use cases and data to be used are not limited to these.
As described above, in the second embodiment, by performing common cleansing processing on input data, it is possible to improve the accuracy of generating a learning model for all machine learning algorithms.

(Other embodiments)
The present invention supplies a program for realizing one or more functions of the above-described embodiments to a system or an apparatus via a network or a storage medium, and one or more processors in a computer of the system or the apparatus read and execute the program. This processing can be realized. Further, it can also be realized by a circuit (for example, an ASIC) that realizes one or more functions.
Further, the present invention may be applied to a system including a plurality of devices or to an apparatus including a single device.
The present invention is not limited to the above embodiments, and various modifications (including organic combinations of the embodiments) are possible based on the spirit of the present invention, and they are excluded from the scope of the present invention. Not something. That is, the present invention also includes all configurations obtained by combining the above-described embodiments and their modifications.

110 common input data 120 input unit 130 cleansing processing unit 131 algorithm database 132 cleansing processing execution unit 140 classification estimation unit 141 learning model generation unit 142 learning model 143 learning model execution unit 144 classification estimation result 150 output determination unit 160 output unit 170 after cleansing data

Claims (17)

Acquisition means for acquiring input data;
Processing means for performing cleansing processing on the input data to generate learning data;
Generating means for generating a learning model based on learning data, comprising: a machine learning algorithm;
A learning data generation device having
The generating means includes a plurality of different types of machine learning algorithms as the machine learning algorithm,
The learning data generation device, wherein the processing unit performs a first cleansing process on the input data in correspondence with each of the plurality of machine learning algorithms. The learning data generation device according to claim 1, wherein the processing unit performs a different first cleansing process in accordance with each feature of the plurality of machine learning algorithms. At least one of the plurality of machine learning algorithms includes a feature in which the feature focuses on one of an appearance frequency, an appearance pattern, and an appearance order of elements included in input data. Item 3. The learning data generation device according to item 2. The learning data generation device according to claim 3, wherein at least one of the plurality of machine learning algorithms includes one in which the feature focuses on an appearance frequency of an element included in input data. The learning data generation device according to claim 3, wherein at least one of the plurality of machine learning algorithms includes one in which the feature focuses on an appearance pattern of an element included in input data. . 6. The method according to claim 3, wherein at least one of the plurality of machine learning algorithms includes one in which the feature focuses on an appearance order of elements included in input data. 7. Learning data generation device. The learning data generation device according to claim 3, wherein the element is a word. The said processing means performs the 2nd cleansing process common to all of the said some machine learning algorithm in addition to the said 1st cleansing process. The Claims 3 thru | or 7 characterized by the above-mentioned. Learning data generation device. The learning data generation device according to claim 8, wherein the second cleansing process deletes input data that does not include valid field information. The learning data generation device according to claim 8, wherein the second cleansing process deletes input data including an element specified in advance. The input data is classified into one of a plurality of labels based on the learning model, and input data that appears in common with all of the plurality of labels is deleted. The learning data generation device according to any one of claims 8 to 10. The learning data generation device according to any one of claims 1 to 11, wherein the input data is information on a behavior of a worker. The learning data generation device according to claim 12, wherein the action is a personal computer operation. The learning data generation device according to any one of claims 1 to 13, wherein each input data is classified into one of a plurality of labels based on the learning model. Based on an evaluation value indicating the degree of association between each input data and label,
The learning data generation device according to claim 14, wherein the input data is classified into one of a plurality of the labels. An acquisition step for acquiring input data;
A processing step of performing a cleansing process on the input data to generate learning data;
Comprising a machine learning algorithm, using the machine learning algorithm, generating a learning model based on learning data,
A learning data generation method having
In the generating step, as the machine learning algorithm, different types, a plurality of machine learning algorithms are provided,
In the processing step, a first cleansing process is performed on input data corresponding to each of the plurality of machine learning algorithms.   A program for causing a computer to execute the learning data generation method according to claim 16.