JP6506360B2 - Method of generating teacher data, method of generating learned model, learned model, computer and program - Google Patents

Description

  The present invention relates to machine learning, and more particularly to a method of generating teacher data, a method of generating a learned model, a learned model, a computer and a program.

  Techniques for labeling data using machine learning are known. For example, labeling for image data is applicable to image recognition and the like. In one method of machine learning, parameters that define a labeling model are mechanically learned using data (teacher data) to which a correct label is attached in advance, and unknown data is labeled using the learned labeling model. I do. Examples of such techniques are disclosed in Patent Document 1 and Non-Patent Document 1. In Patent Document 1 and Non-Patent Document 1, image data is treated as data.

Unexamined-Japanese-Patent No. 10-21393

Takaya Okaya, "Deep Learning and Image Recognition-Basics and Recent Trends-", [online], [Search on April 22, 2016], Internet <URL: http://www.orsj.or.jp/archive2 /or60-4/or60_4_198.pdf>

  However, in the conventional configuration, there is a problem that it is difficult to generate a large amount of teacher data having correct answer labels.

  For example, a large amount of data may be prepared, and teacher data may be generated by giving correct labels one by one visually, but it is difficult to execute because the amount of work is enormous.

  The present invention has been made to solve such problems, and a method of generating teacher data for machine learning, capable of generating a large amount of teacher data having correct labels, and a learned model are generated. The purpose is to provide a method, a learned model, a computer and a program.

In order to solve the above-mentioned problems, the method according to the present invention is
A computer is a method of generating teacher data for machine learning, wherein
By performing machine learning using a predetermined number A of classified first teacher data to which a first label has been given and classified second teacher data to which the first label has not been given, teacher data and A first learning step of generating a first learned model representing a relationship with the first label;
Determining whether or not the first label should be given to a predetermined number B of unclassified teacher data based on the first learned model ;
An additional determination step of determining whether or not the determination step should be performed on X unclassified teacher data;
Equipped with
X = B. (C-A-D) / D,
C is the required number of classified teacher data given the first label,
D is Ru number der teacher data is determined to be given a first label in the determination step.
According to a particular aspect, depending on the result of the determining step, based on the unclassified teacher data, generating classified third teacher data to which the first label is given, or the first label The method includes a teacher data generation step of generating classified fourth teacher data which is not given.
According to a particular aspect,
After the teacher data generation step,
The method may further comprise a correction step of deleting the first label given to the classified third teacher data or giving the first label to the classified fourth teacher data.
Further, a method according to the present invention is a method in which a computer generates a learned model by machine learning, and
Generating teacher data using the method described above;
Wherein the classified first teacher data, wherein the classified second training data, by performing machine learning using at least one of the classified third teacher data and the classified fourth teacher data, teacher data And a second learning step of generating a second learned model representing the relationship between and and the first label.
According to a particular aspect, the first learning step and the second learning step are performed based on the same pre-learning model.
Further, a learned model according to the present invention is a second learned model generated using the method described above, and it is determined whether to give the first label to test data based on teacher data. Make the computer function as you
Also, a computer according to the present invention performs the method described above.
Further, a program according to the present invention causes a computer to execute the method described above.

  According to the technique of the present invention, since the remaining unclassified teacher data is classified based on a part of the teacher data, it is possible to easily generate a large amount of teacher data.

It is a figure which shows the example of a structure of the machine learning system which concerns on Embodiment 1 of this invention. It is a flowchart explaining the flow of the processing which the machine learning system of FIG. 1 performs. It is a figure explaining the input-output of the process of step S1-S3 of FIG. It is a figure explaining the input-output of the process of step S5 of FIG. 2, and S6.

Hereinafter, an embodiment of the present invention will be described based on the attached drawings.
Embodiment 1
FIG. 1 shows an example of the configuration of a machine learning system 10 according to the first embodiment of the present invention. The machine learning system 10 is a system that executes a machine learning method, and generates a labeling model for labeling data by machine learning. Machine learning system 10 is also a computer that performs the methods described herein.

  "Labeling" means performing some judgment on each data to obtain a result that may differ depending on the data. The data is, for example, image data. The representation format of the image data may be any format, but may be, for example, a bitmap format, or may be a format using a known feature quantity.

  Labeling of image data is, for example, what each image data represents (a white cane appears, a stroller appears, a person appears, an object appears, an animal appears, etc. , Etc.) can be realized. The determination is made based on the labeling model. The labeling model is a trained model generated by machine learning, and can be expressed using, for example, one or more parameters.

  In the present embodiment, learning is realized using a neural network. For example, the labeling model in the present embodiment is defined using a weight connecting neurons of different layers in a neural network including a plurality of neuron layers. Specific examples of such neural networks and labeling models are disclosed, for example, in Non-Patent Document 1. Non-Patent Document 1 discloses a labeling model according to deep learning.

  As shown in FIG. 1, the machine learning system 10 has a configuration as a known computer, and includes an operation unit 11 that performs an operation and a storage unit 12 that stores information. Arithmetic unit 11 includes, for example, a CPU (central processing unit), and storage unit 12 includes, for example, a semiconductor memory and an HDD (hard disk drive). The storage unit 12 stores various image data as shown in FIG. The machine learning system 10 executes the method according to the present embodiment by using these image data.

  The storage means 12 also stores a program (not shown), and the machine learning system 10 implements the functions described herein by the computing means 11 executing this program. That is, this program causes the computer to function as the machine learning system 10 by causing the computer to execute the method described herein.

  The machine learning system 10 may also include other components that a known computer usually comprises. For example, a display and a printer which are output devices, a keyboard and a mouse which are input devices, a network interface which combines an input device and an output device for a communication network, and the like may be provided.

  As shown in FIG. 1, the image data includes a teacher image T representing teacher data and a test image E representing test data. The teacher image T is used to learn a labeling model. The test image E is used to evaluate the accuracy of the labeling model.

  The teacher image T can be generated, for example, by extracting it from images constituting a moving image. This extraction may be performed automatically by the machine learning system 10 or another computer, or may be performed visually by the user of the machine learning system 10 or the like.

  The teacher image T includes a classified teacher image TC and an unclassified teacher image TU. The classified teacher image TC is an image given a specific label as the correct answer label. For example, if a white cane appears in the image, the image is given the label "white cane". In the present embodiment, at least one of the classified teacher images TC is a white cane image T1 (first teacher data) to which a label (first label) of "white cane" is given.

  Further, at least one of the classified teacher images TC is a non-white cane image T2 (second teacher data) which is not given the label "white cane". In the present embodiment, as an example of the non-white cane image T2, a label (second label) given as another label "stroller" is used. For the classified teacher image TC, labels such as "white cane" and "stroller" are given in advance. The labeling of the classified teacher image TC can be performed, for example, visually.

  Note that the labeling does not have to be exclusive or selective. For example, the classified teacher image TC may include an image without any label, and may include an image with both the label “white stick” and the label “stroller”, and another label (eg, “person ') May be included.

  The unclassified teacher image TU is an image to which no label is given. Or, the image is not processed to give a label. Or, the image is not processed to determine whether to give a label. In particular, with regard to the unclassified teacher image TU, it is not necessary to perform labeling processing by visual inspection or the like in advance.

  The types of labels related to machine learning are common to the teacher image T and the test image E. For example, in the present embodiment, both the teacher image T and the test image E include those representing white canes and those representing strollers.

The operation of the machine learning system 10 having the configuration as described above will be described below with reference to FIGS.
FIG. 2 is a flow chart for explaining the flow of processing executed by the machine learning system 10. Steps S1 to S3 in FIG. 2 particularly relate to a method of generating teacher data for machine learning.
FIG. 3 is a diagram for explaining the input and output of the process of steps S1 to S3 of FIG.

  In the process of FIG. 2, first, the machine learning system 10 generates a first labeling model (first learned model) using the classified teacher image TC (step S1, first learning step). The first labeling model is a learned model generated by performing machine learning using the white cane image T1 and the non-white cane image T2, and the first labeling model is a teacher image T (strictly, it is a part of it. It is a learned model that represents the relationship between the completed teacher image TC) and the label “white stick”.

  Step S1 can be performed using a deep learning technique using, for example, a neural network. In this case, the first labeling model can be defined using weights connecting neurons of different layers in a neural network comprising a plurality of neuron layers. For example, the caffe framework may be used.

  Next, the machine learning system 10 classifies the unclassified teacher image TU based on the first labeling model (Step S2, determination step). In this step S2, it is determined whether the unclassified teacher image TU should be labeled as "white stick". Step S2 is executed, for example, by inputting the unclassified teacher image TU into the first labeling model and acquiring the output label.

  Next, the machine learning system 10 generates new classified teacher data based on the unclassified teacher image TU according to the result of step S2 (step S3, teacher data generation step). For example, if it is determined that an unclassified teacher image TU should be labeled "white cane", a white labeled "white cane" is given based on the unclassified teacher image TU. A cane image T3 (third teacher data) is generated. On the other hand, when it is determined that the label "white cane" should not be given to an unclassified teacher image TU, an image (fourth teacher data) not having the label "white cane" is generated.

  The non-white cane image T4 (fourth teacher data) that is not given the label "white cane" is, for example, an image that is given the label "stroller". Here, depending on the content of the first labeling model, the determination in step S2 is not necessarily accurate, and it may be determined that the label "white cane" should be given although it is not so, or the reverse may occur. There is sex. Therefore, for example, the “non-white cane image T4” generated in step S3 may actually be an image in which “white cane” appears, but labeling may be performed with at least a certain degree of accuracy. It is possible.

  A specific data format representing a given label can be arbitrarily designed, but may be the same format as, for example, the white cane image T1 and the non-white cane image T2. In that case, the white cane image T1 and the white cane image T3 can not be distinguished in data format, and the non-white cane image T2 and the non-white cane image T4 can not be distinguished in data format.

  As described above, the machine learning system 10 according to the first embodiment of the present invention does not have any remaining based on the classified teacher image TC (white cane image T1 and non-white cane image T2) forming a part of the teacher image T. Since the classified teacher image TU is classified, it is possible to easily generate a large number of classified teacher images T (white cane image T3 and non-white cane image T4).

  For example, when 10000 teacher images T exist, only 500 of them are visually labeled and classified as classified teacher images TC, and the remaining 9500 are unclassified teacher images TU and the present invention is applied. Labels of 10000 teacher images T can be determined efficiently.

  The generated white cane image T3 and non-white cane image T4 may not necessarily all have the correct label, but can be considered to at least generally have the correct label. For this reason, according to the machine learning system 10, it is possible to easily and massly generate teacher images T having roughly correct labels.

  Although part of the effect of the present invention can be obtained in the processing up to step S3, further subsequent processing may be performed.

  For example, after step S3, the machine learning system 10 deletes the label “white cane” given to the white cane image T3 or gives the label “white cane” to the non-white cane image T4 (see FIG. Step S4, correction step).

  This correction step may be performed, for example, in response to the operation of the user of the machine learning system 10. As a specific example, the machine learning system 10 includes a white stick image T3, information indicating that the white stick image T3 is given the label "white stick", a non-white stick image T4, and a non-white stick image T4. An operation to display information indicating that the label "White Cane" was not given, to select either the white cane image T3 or the non-white cane image T4, and to change the label of the selected image It may have a function of accepting In addition, the machine learning system 10 may correct the labels of the white cane image T3 and the non-white cane image T4 according to the input operation.

  For example, if the user of the machine learning system 10 browses the white cane image T3 and the non-white cane image T4 and finds out that the wrong label is given, the teacher image T for which the label should be changed is used. Information to be identified (such as a file name) and information representing a label change (deleting a particular label, giving a particular label, etc.) may be input to the machine learning system 10. As a result of step S4, the white cane image T3 generated in step S3 is changed to a non-white cane image T4, or conversely the non-white cane image T4 generated in step S3 is changed to a white cane image T3 there is a possibility.

  Although such a correction requires some visual work, the work efficiency is improved as compared to the case where a large number (for example, 9500 pieces) of unclassified teacher images TU are all visually classified.

  Also, step S4 may be performed automatically. For example, the machine learning system 10 deletes the label “white cane” given to the white cane image T3 based on predetermined criteria determined in advance, or “white cane” on the non-white cane image T4. "" May be given.

  The teacher image T generated in steps S1 to S3 (or steps S1 to S4) can be used in various ways. For example, it can be used for the process of labeling the test image E. Steps S5 and S6 show an example of such a method of use.

  FIG. 4 is a diagram for explaining the input and output of the processes of steps S5 and S6 of FIG. After step S3 (or after step S4), the machine learning system 10 generates a second labeling model (second learned model) using the teacher image T (step S5, second learning step). The second labeling model is a learned model generated by performing machine learning using the white cane image T1, the non-white cane image T2, the white cane image T3, and the non-white cane image T4. , And a learned model representing the relationship between the teacher image T and the label "white stick". Although all teacher images T may be used, not all teacher images T need to be used. For example, it is not necessary to use both the white cane image T3 and the non-white cane image T4, and the process of step S5 can be performed using only one of them. Thus, the machine learning system 10 generates a learned model by machine learning.

  Step S5 can be performed using a deep learning technique using, for example, a neural network. In this case, a second labeling model can be defined using a weight connecting neurons of different layers in a neural network comprising a plurality of neuron layers. For example, the caffe framework may be used.

  Here, steps S1 and S5 may be performed based on the same pre-learning model. The definition of the “pre-learning model” can be determined as appropriate by those skilled in the art. In the case of deep learning using a neural network, for example, it refers to a specific structure of the neural network before the start of correction of weighting by error back propagation. More specifically, it refers to a function for determining the output of each neuron, the number of layers, the number of neurons in each layer, the presence or absence of a connection relation between the neurons, and the like. When part of the pre-learning model includes random numbers (initial values of weights, etc.), differences in models due to random numbers are not taken into account (for example, pre-learning models having different initial values of weights are identical to the pre-learning models). Is).

  Next, the machine learning system 10 classifies the test image E based on the second labeling model (step S6). In step S6, it is determined whether the label “white cane” should be given to the test image E using the white cane image T1 or T3 and the non-white cane image T2 or T4. Step S6 is executed, for example, by inputting the test image E into the second labeling model and acquiring the output label. Here, the second labeling model is a learned model for causing the computer to function to determine whether to label the test image E with the label “white stick” based on the teacher image T. Can.

  Classification of the test image E is performed in this manner. After step S6, the test image E may be labeled according to the determination result. This process may be performed similarly to step S3.

The following modifications can be made in the first embodiment described above.
The number and combination of label types to be determined can be arbitrarily designed. It may be determined whether to give a single label or not, it may be selected whether to give (or not give) any of a plurality of labels, or for each of a plurality of labels Whether to give a label may be determined individually.

  In the first embodiment, all data are image data, but any other type of data (document data, numerical data, etc.) may be used as long as the data can be labeled by machine learning. .

The operation of the machine learning system 10 described with reference to FIG. 2 proceeds to step S2 when 10000 teacher images T are present and 500 are classified teacher images TC and the remaining 9500 are unclassified teacher images TU. Is configured to classify all unclassified teacher images TU in the processing of.
As a modification, it is constituted so that processing of Step S2-Step S3 may be repeated. The unclassified teacher image TU is divided into, for example, 500 pieces to generate a teacher image newly classified about 500 pieces in step S2, and a newly classified teacher data is generated in step S3. After step S3, the cumulative number of teacher data is calculated, and it is determined whether the cumulative number has reached a predetermined number. If it is determined that the cumulative number has not reached a predetermined number, the next 500 are determined. The processing of steps S2 to S3 is repeated. If it is determined that the cumulative number of teacher data reaches the predetermined number after step S3, the process ends.
Although it may take processing time to generate a teacher image TC by one process for an unclassified teacher image TU, it is configured to sequentially process a constant of the unclassified teacher image TU in this way This increases the possibility that the predetermined number of teacher images TC can be generated efficiently.

In step S2, first, a teacher image is generated for a fixed number (for example, 500), and after step S3, the number of unclassified teacher images TU processed in step S2 is the teacher image TC accumulated in step S3. When it is determined that the number of unsorted teacher images TU increases in proportion to the number of unsorted teacher images TU, the remaining unclassified teacher images TU are summarized in the process from the next step S2. It may be configured to generate a teacher image. In the case of such a teacher image, the teacher image TC is expected to increase efficiently in proportion to the number of unclassified teacher images TU, but it is confirmed in advance whether or not it has the characteristics of such a teacher image. The possibility of shortening the processing time is increased.
Note that the predetermined number is a number that requires a preset teacher image TC.

  Although how to use the proportional principle can be designed as appropriate, one specific example will be described below. First, target numbers are set for teacher images having specific labels. For example, if 1000 teacher images with the label "white cane" are needed, the target number is 1000. Next, in steps S2 and S3, the machine learning system 10 generates teacher images for a predetermined number (for example, 500) of the unclassified teacher images TU, and among them, the label "white cane" is given. Calculate the ratio of things. If 500 classified teacher images TC are generated in step S3 and 100 of them correspond to "white cane", the ratio is 100/500 = 0.2. Then, the machine learning system 10 calculates the number of classified teacher images TC to be additionally processed based on the ratio and the target number. For example, if 200 out of the 500 classified teacher images TC that were originally present (that is, determined visually) were to correspond to "white cane", then they were combined with 100 generated in step S3. There will be a total of 300 "white cane" teacher images, and the rest will be 1000-300 = 700. Therefore, processing is not necessary to obtain these 700 "white cane" teacher images. The number of classified teacher images TU is 700 / 0.2 = 3500. Therefore, the machine learning system 10 executes steps S2 and S3 for the newly 3500 unclassified teacher images TU to obtain the required number of "white cane" teacher images. If the total number of "white cane" teacher images including those obtained here does not reach the target number as a result, etc., similar proportional calculations are performed and steps S2 and S3 are repeatedly executed. Good.

  The concrete implementation method of machine learning can be designed arbitrarily. Although deep learning using a neural network is used in the first embodiment, another machine learning method may be used. Further, even in the case of using deep learning, a specific learning method can be designed arbitrarily. For example, the number of times of learning, the accuracy evaluation method and evaluation criteria during learning, the directory structure and file structure of input / output data, and the like may be arbitrarily designed. Also, for example, the known caffe framework may be used.

  10 machine learning system (computer), S1 first learning step, S2 determination step, S3 teacher data generation step, S4 correction step, S5 second learning step, E test image (test data), T teacher image (teacher data), T1 white cane image (first teacher data), T2 non-white cane image (second teacher data), T3 white cane image (third teacher data), T4 non white cane image (fourth teacher data), TU unclassified teacher Image (Unclassified teacher data).

Claims (7)

A computer is a method of generating teacher data for machine learning, wherein
By performing machine learning using a predetermined number A of classified first teacher data to which a first label has been given and classified second teacher data to which the first label has not been given, teacher data and A first learning step of generating a first learned model representing a relationship with the first label;
A determining step of determining whether the first label should be given to a predetermined number B of unclassified teacher data based on the first learned model ;
An additional determination step of determining whether or not the determination step should be performed on X unclassified teacher data;
Equipped with
X = B. (C-A-D) / D,
C is the required number of classified teacher data given the first label,
D is Ru number der teacher data is determined to be given a first label in the determination step, the method. According to the result of the determination step, based on the unclassified teacher data, the classified third teacher data to which the first label is given is generated, or the classified not to which the first label is given The method of claim 1, comprising generating teacher data, generating fourth teacher data. After the teacher data generation step,
The method according to claim 2, further comprising: a correction step of deleting the first label given to the classified third teacher data or giving the first label to the classified fourth teacher data. . A computer is a method of generating a learned model by machine learning,
Generating teacher data using the method according to claim 2;
Wherein the classified first teacher data, wherein the classified second training data, by performing machine learning using at least one of the classified third teacher data and the classified fourth teacher data, teacher data And a second learning step, generating a second learned model that represents the relationship between and the first label.   The method according to claim 4, wherein the first learning step and the second learning step are performed based on the same pre-learning model.   A computer implementing the method according to any one of the preceding claims.   The program which makes a computer perform the method as described in any one of Claims 1-5.

Images