WO2021255778A1 - Learning data selection method, learning data selection device, and learning data selection program - Google Patents

Learning data selection method, learning data selection device, and learning data selection program Download PDF

Info

Publication number
WO2021255778A1
WO2021255778A1 PCT/JP2020/023371 JP2020023371W WO2021255778A1 WO 2021255778 A1 WO2021255778 A1 WO 2021255778A1 JP 2020023371 W JP2020023371 W JP 2020023371W WO 2021255778 A1 WO2021255778 A1 WO 2021255778A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
candidate data
label assignment
assignment candidate
class
Prior art date
Application number
PCT/JP2020/023371
Other languages
French (fr)
Japanese (ja)
Inventor
俊介 塚谷
和彦 村崎
慎吾 安藤
潤 島村
Original Assignee
日本電信電話株式会社
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 日本電信電話株式会社 filed Critical 日本電信電話株式会社
Priority to PCT/JP2020/023371 priority Critical patent/WO2021255778A1/en
Publication of WO2021255778A1 publication Critical patent/WO2021255778A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00—Machine learning

Definitions

  • the present invention relates to a learning data selection method, a learning data selection device, and a learning data selection program.
  • labels are given to the data with the highest learning effect from among multiple data that are not used for the trained data. , It is preferable to use additional learning data.
  • Non-Patent Document 1 discloses a method of selecting data having as high a learning effect as possible in the index of AUC (Area Under the Curve) score represented by the area under the ROC (Receiving Operating Characteristics). There is.
  • Non-Patent Document 1 Culver, Matt, Deng Kun, and Stephen Scott. "Active learning to maximize area under the ROC curve.” Sixth International Conference on Data Mining (ICDM'06). IEEE, 2006.
  • Non-Patent Document 1 is a method of eliminating the class imbalance of learning data, but the improvement range of the AUC score is not explicitly optimized. Therefore, if the selected labeling candidate data is similar to the data contained in the trained data, additional training of the training model is performed using the additional training data generated by adding a label to the selected labeling candidate data. However, the AUC score may not improve.
  • the label assignment candidate data prepared for the additional learning of the learning model that improves the learning accuracy of the learning model more efficiently than before the additional learning is performed by causing the learning model to perform additional learning.
  • the first aspect of the present disclosure is a training data selection method, which is prepared for each feature amount of the trained data included in the trained data set used for training the training model and for additional training of the training model.
  • a parameter representing the probability distribution of the feature quantity of the trained data in the positive class and the negative class is estimated.
  • the probability that the label assignment candidate data belongs to the positive class and the probability that the label assignment candidate data belongs to the negative class are determined for each of the label assignment candidate data by using the step to be performed, the feature amount of the label assignment candidate data, and the parameter.
  • the step of calculating the score which is the output of the training model for each of the label assignment candidate data and the score for each of the trained data and the label assignment candidate data are used.
  • the width the probability that the label assignment candidate data belongs to the positive class, and the probability that the label assignment candidate data belongs to the negative class
  • the expected value of the improvement width of the AUC score is calculated for each of the label assignment candidate data.
  • the second aspect of the present disclosure is a training data selection device, which is prepared for each feature amount of the trained data included in the trained data set used for training the training model and for additional training of the training model.
  • the feature amount extraction unit that extracts the feature amount of each of the label assignment candidate data included in the labeled candidate data set, and the feature amount extraction unit that belongs to the positive class indicating the event to be estimated by the learning model.
  • the feature amount is used.
  • the distribution estimation unit that estimates the parameters representing the probability distribution of the feature amount of the learned data in the positive class and the negative class, the feature amount of the label assignment candidate data extracted by the feature amount extraction unit, and the distribution.
  • the probability estimation unit that estimates the probability that the label assignment candidate data belongs to the positive class and the probability that the label assignment candidate data belongs to the negative class for each of the label assignment candidate data, and the estimation.
  • the trained model parameters of the function representing the input / output relationship of the training model for estimating the likelihood of the target event, the trained data, and the label assignment candidate data the trained data and the label assignment candidate data can be obtained.
  • the label assignment candidate is used by using the score calculation unit that calculates the score that is the output of the learning model for each, and the scores for each of the trained data and the label assignment candidate data calculated by the score calculation unit.
  • the AUC improvement width calculation unit that calculates the improvement width of the AUC score in each case assuming that the label assignment candidate data belongs to either the positive class or the negative class, and the AUC improvement The improvement width of the AUC score for each of the label assignment candidate data calculated by the width calculation unit, the probability that the label assignment candidate data estimated by the probability estimation unit belongs to the positive class, and the estimation by the probability estimation unit.
  • the expected value of the improvement range of the AUC score is calculated for each of the label assignment candidate data, and the improvement range of the AUC score in the label assignment candidate data set is calculated. It includes a selection unit that selects the label assignment candidate data having the highest expected value as the label assignment target data used for generating the additional training data of the learning model.
  • the third aspect of the present disclosure is a learning data selection program, in which a computer functions as each part of a learning data selection device.
  • the learning model is additionally trained from the labeling candidate data prepared for the additional learning of the learning model. It has the effect of being able to preferentially select labeling candidate data that efficiently improves the learning accuracy of the learning model compared to before learning.
  • FIG. 1 is a diagram showing a functional configuration example of the learning data selection device 100. As shown in FIG. 1, the learning data selection device 100 includes an input unit 10 and a calculation unit 20.
  • the input unit 10 includes a trained data set A, which is a set of trained data a used for learning a learning model generated by supervised learning, and labeling candidate data prepared for additional learning of the learning model. Accepts the label assignment candidate data set B, which is a set of b. Further, the input unit 10 receives the trained model parameter ⁇ of the function f representing the input / output relationship of the learning model generated by the learning of the trained data a, that is, the function f for estimating the event-likeness to be estimated.
  • the trained model parameter ⁇ is a parameter that defines the input / output relationship of the generated training model.
  • an event that the estimated target of the learning model is referred to as a "positive class C +", an event other than the estimation target of the learning model "negative class C -" that.
  • a positive class C + an event that the estimated target of the learning model
  • an event other than the estimation target of the learning model negative class C -
  • the animal represented by the image is estimated by the learning model whether the "cat”
  • event of a "cat” is a positive class C +
  • the an event that is other than cat negative Class C - a is other than cat negative Class C - a.
  • the calculation unit 20 performs an operation for selecting label assignment candidate data b to be used for generating additional learning data of the learning model from the label assignment candidate data set B using various data received by the input unit 10. ..
  • the calculation unit 20 includes a feature amount extraction unit 21, a distribution estimation unit 22, a probability estimation unit 23, a score calculation unit 24, an AUC improvement width calculation unit 25, and a selection unit 26.
  • the feature amount extraction unit 21 inputs the trained data set A and the label assignment candidate data set B received by the input unit 10, and inputs each trained data a and the label assignment candidate data set B included in the trained data set A.
  • the feature amount of each label addition candidate data b included is extracted.
  • a feature extractor using a convolutional neural network (CNN) learned from ImageNet data is used. Instead of extracting the features using CNN, even if the features of the trained data a and the labeling candidate data b are extracted using the heuristically designed features such as the local image features. good.
  • the features of the trained data a and the label assignment candidate data b are extracted by using the reconstruction error of the Variational AutoEncoder obtained from the trained data set A. You may.
  • the feature amount of each trained data a included in the trained data set A is represented by g (a)
  • the feature amount of each label assignment candidate data b included in the label assignment candidate data set B is g ( It is expressed as b).
  • the feature amount extraction unit 21 may be separated into an extraction unit for extracting the feature amount g (a) of the learned data a and an extraction unit for extracting the feature amount g (b) of the label assignment candidate data b. ..
  • Distribution estimating unit 22 each of the feature quantity g of the learned data a extracted by the feature amount extracting section 21 (a) as input for the learned data a, the positive class C + and the negative class C - each in parameters of the probability distribution of ⁇ +, ⁇ - to estimate.
  • the distribution estimation unit 22 uses each feature amount g (a) as a feature amount of the trained data a + (a + ⁇ A + ) included in the trained data set A + belonging to the positive class C +. It is classified into g (a + ) and the feature quantity g (a ⁇ ) of the trained data a ⁇ (a ⁇ ⁇ A ⁇ ) contained in the trained data set A ⁇ belonging to the negative class C ⁇ .
  • the distribution estimation unit 22 has a positive class C + probability distribution h + (g (a + ); ⁇ + ) and a negative class C indicated by the classified feature amount g (a + ) and feature amount g (a ⁇ ), respectively.
  • the distribution estimation unit 22 has an estimation unit that estimates the parameter ⁇ + of the probability distribution h + (g (a + ); ⁇ + ) using the feature amount g (a + ), and the feature amount g (a ⁇ ). parameters of omega - may be separated in the estimation unit for estimating a; (- - ⁇ g (a -)) probability distribution h using.
  • the probability estimation unit 23 extracts each feature amount g (b) of the label assignment candidate data b extracted by the feature amount extraction unit 21, and the parameters ⁇ + and ⁇ ⁇ estimated by the distribution estimation unit 22, respectively, in the feature amount extraction unit 21. And received from the distribution estimation unit 22. Then, the probability estimation unit 23 inputs the feature amount g (b) and the parameters ⁇ + and ⁇ ⁇ , and the probability p that the label assignment candidate data b belongs to the positive class C + for each label assignment candidate data b, And the probability p belonging to the negative class C -is estimated respectively.
  • labeling candidate data b which may in particular labeling candidate data b representing the "labeling candidate data b i".
  • I is an index for uniquely indicating the label assignment candidate data b.
  • Class contained in the probability distribution that is modeled by the trained data set A - labeling candidate data b i is a positive class C + or negative class C - when estimating the probability belonging to p, each class C +, C And, it is considered that it is composed of two kinds of sets of classes not included in the probability distribution.
  • class C + I included in the probability distribution in the positive class C + the class as a probability distribution outside the C + O, minus Class C - consisting of classes in the probability distribution in C -I, and the probability distribution outside
  • the class be CO.
  • the probability p (c generated from class C + I contained in the probability distribution i C + I
  • is the sum of
  • (g (b i c i C + O)) and g (b i)), the probability generated from class C + O outside the probability distribution p.
  • the probability p (c i C +
  • g (b i)) is represented by equation (1) is expanded as (2).
  • probability p (c i C -
  • g (b i)) is generated from the class C -I contained in the probability distribution probability p is the sum of
  • the probability p (c i C -
  • g (b i)) is represented by equation (3).
  • a negative class C - probability distributions h - (g (a -) ; ⁇ -) as a parameter omega - using a labeling candidate data b i probability distributions h - (g (a -) ; ⁇ -) probabilities generated from the distribution h - (g (b i) ; ⁇ -) may be used a value obtained by calculating the.
  • the learned data a + number of n (A +), the learned data a - the number of n (A -), among the labeling candidate data set B, the labeling candidate data b i is a probability distribution h + (g (b i); ⁇ +) and h - (g (b i) ; ⁇ -) , respectively outside a proportion of the t.
  • p (c i C + I)
  • p (c i C -I)
  • p (c i C + O)
  • the ratio t is a value determined by an experiment.
  • the score calculation unit 24 includes the trained model parameter ⁇ of the function f represented by the learning model for estimating the degree to which the event belongs to the positive class C + , that is, the positive class C + of the event, the trained data set A, and the trained data set A.
  • the label assignment candidate data set B is received from the input unit 10.
  • the score calculation unit 24 has, in the function f represented by the trained model parameter ⁇ , each of the trained data a included in the trained data set A and each of the label assignment candidate data b included in the label assignment candidate data set B. Enter. As a result, the score calculation unit 24 calculates the score f ⁇ (a) and the score f ⁇ (b), which are the outputs of the training model for each of the trained data a and the label assignment candidate data b.
  • the score calculation unit 24 is separated into a calculation unit that calculates the score f ⁇ (a) using the learned data a and a calculation unit that calculates the score f ⁇ (b) using the label assignment candidate data b. You may.
  • the AUC improvement width calculation unit 25 receives the calculated score f ⁇ (a) and score f ⁇ (b) from the score calculation unit 24.
  • the AUC improvement width calculation unit 25 uses the score f ⁇ (a) and the score f ⁇ (b), and the label assignment candidate data b is either positive class C + or negative class C ⁇ for each label assignment candidate data b.
  • the improvement width I of the AUC score in each case assuming that it belongs to one is calculated. Improvements width I of the AUC score, and AUC score calculated by the current learning model for current learning model, the additional learning model obtained by additional learning using label assignment candidate data b i imparted with some label It is represented by the difference between the calculated AUC scores.
  • the AUC score AUC in the current learning model is expressed by Eq. (8).
  • labels labeling candidate data b i represents belong to positive class C +, i.e. when applying the C + labels, is calculated by the current learning model AUC score AUC + (b i) it is represented by equation (9).
  • AUC score is calculated by adding the learning model in addition to the additional learning data added learning the current status of the learning models AUC + target (b i). Assuming that there is no variation element in AUC score than C + Labels imparted with grant candidate data b i, all learned data a - and until - against f (b i)> f ( a) The learning model will be updated. Is calculated by such additional learning model AUC score AUC + target (b i) is represented by equation (10).
  • AUC improvements width calculating unit 25 for each of the labeling candidate data b i, improved width I + (b i) of the AUC scores and I - to calculate the (b i).
  • Selecting unit 26 the probability from a probability estimation section 23 labeling candidate data b i belonging to the positive class C + p
  • a (c i C + g ( b i)), the labeling candidate data b i is negative class C - probability belong to p -
  • accepts (c i C g (b i)).
  • with (c i C g (b i)), calculates the expected value of improving the width I of the AUC score E (b i) for each label applying candidate data b i.
  • the selection unit 26 of the labeling candidate data set B, and expectation E (b i) is the highest labeling candidate data b i improvements width I of AUC scores, the learning model of additional training data Select as the data to be labeled as used for generation.
  • the expected value E of the improved width I of AUC score in the label applying candidate data b i (b i) is represented by (13).
  • the selection unit 26 the labeling candidate data b i is a positive class C probability p that belongs to +
  • when the C + label given to ( c i C + g ( b i)) and labeling candidate data b i calculating the product of the improved width I + (b i) of the AUC score.
  • (13) is an example of a calculation formula for calculating the expected value E (b i) improvements width I of the AUC score.
  • selecting section 26 may calculate the (13) by integrating the coefficients representing the weighting sections the right side of the equation, the expected value of the improvements width I of AUC score E (b i).
  • the learning data selection device 100 is configured by using a computer 30 as an example.
  • FIG. 2 is a diagram showing a configuration example of a main part of the computer 30 applied to the learning data selection device 100.
  • the computer 30 includes a CPU (Central Processing Unit) 31 that is responsible for processing in each part of the learning data selection device 100 shown in FIG. Further, the computer 30 has a ROM (Read Only Memory) 32 for storing a learning data selection program that causes the computer 30 to function as a learning data selection device 100, and a RAM (Random Access Memory) used as a temporary work area of the CPU 31. Includes 33. Further, the CPU 31 includes a non-volatile memory 34 and an input / output interface (I / O) 35. Then, the CPU 31, ROM 32, RAM 33, non-volatile memory 34, and I / O 35 are each connected by the bus 36.
  • CPU Central Processing Unit
  • ROM Read Only Memory
  • RAM Random Access Memory
  • the non-volatile memory 34 is an example of a storage device in which the stored information is maintained even if the power supplied to the non-volatile memory 34 is cut off.
  • a semiconductor memory is used, but a hard disk may be used.
  • the non-volatile memory 34 does not have to be included in the computer 30, and for example, a portable storage device that can be attached to and detached from the computer 30 may be used as the non-volatile memory 34.
  • a communication unit 37 for example, a communication unit 37, an input unit 38, and a display unit 39 are connected to the I / O 35.
  • the communication unit 37 is connected to a communication line such as the Internet and a LAN (Local Area Network), and includes a communication protocol for performing data communication with an external device connected to the communication line. Wired communication or wireless communication such as Wi-Fi (registered trademark) is used as the communication line.
  • the input unit 38 is a device that receives a user's instruction and notifies the CPU 31, for example, a button, a touch panel, a keyboard, and a mouse are used.
  • a microphone may be used as the input unit 38.
  • the display unit 39 is an example of a device that visually displays information processed by the CPU 31, and for example, a liquid crystal display, an organic EL (Electroluminescence) display, or a projector is used.
  • a liquid crystal display an organic EL (Electroluminescence) display, or a projector is used.
  • the trained data set A, the label assignment candidate data set B, and the trained model parameter ⁇ are input to the input unit 10 via, for example, a portable non-volatile memory 34 that can be attached to and detached from the communication unit 37 or the computer 30. Will be.
  • the computer 30 does not necessarily have to include the communication unit 37.
  • the learning data selection device 100 is installed in an unmanned data center and receives control from a remote location through a communication line, the computer 30 does not necessarily have to include the input unit 38 and the display unit 39.
  • various units connected to the I / O 35 are an example, and for example, an image forming unit that forms information on a recording medium as characters or images may be connected to the I / O 35.
  • the learning data selection device 100 of the present disclosure When the input unit 10 receives the trained data set A, the label assignment candidate data set B, and the trained model parameter ⁇ of the training model, the CPU 31 of the training data selection device 100 performs the training data selection process according to the flowchart shown in FIG. To execute.
  • the learning data selection program that defines the learning data selection process is stored in advance in, for example, the ROM 32 of the learning data selection device 100.
  • the CPU 31 of the learning data selection device 100 reads the learning data selection program stored in the ROM 32 and executes the learning data selection process.
  • the trained data set A, the label assignment candidate data set B, and the trained model parameter ⁇ received by the input unit 10 are stored in the RAM 33.
  • step S10 the CPU 31 features the feature amount g (a) of each trained data a included in the trained data set A and the features of each label assignment candidate data b included in the label assignment candidate data set B.
  • the amount g (b) is extracted and stored in the RAM 33.
  • step S20 the CPU 31 classifies the trained data a into the trained data a + and the trained data a ⁇ . Further, CPU 31 is learned data a + feature quantity g (a +) and the learned data a - feature quantity g (a -) with respect to the learned data a, the positive class C + and the negative The parameters ⁇ + and ⁇ ⁇ representing the probability distribution of each feature amount g (a) in class C ⁇ are estimated, and the estimation result is stored in the RAM 33.
  • step S30 the CPU 31 acquires the feature amount g (b) of the label assignment candidate data b extracted in step S10 and the parameters ⁇ + and ⁇ ⁇ estimated in step S20 from the RAM 33.
  • g (b i)), and the negative class C - probability p belonging to (c i C -
  • step S40 the CPU 31 acquires the trained data set A, the label assignment candidate data set B, and the trained model parameter ⁇ received by the input unit 10 from the RAM 33. Then, the CPU 31 has a score f ⁇ (a) and a score for each trained data a included in the trained data set A and each label assignment candidate data b included in the label assignment candidate data set B. f ⁇ (b) is calculated, and the score f ⁇ (a) and the score f ⁇ (b) are stored in the RAM 33.
  • step S50 the CPU 31 acquires the score f ⁇ (a) and the score f ⁇ (b) calculated in step S40 from the RAM 33.
  • CPU31 for each label applying candidate data b i, improved width I + (b i) of the AUC scores If granted the C + labels labeling candidate data b i, and C in labeling candidate data b i - label improvements width of AUC scores If granted the I - (b i) the calculated according respectively (11) and (12).
  • CPU31 is improved calculated AUC score width I + (b i) and I - storing (b i) into RAM 33.
  • g (b i)) and probability p (c i C -
  • the expected value of the improvements width I of AUC score E a (b i) (13) is calculated according to equation of the calculated respectively in labeling candidate data b i storing the expected value E of the improved width I of scores (b i) into RAM 33.
  • the expected value of the improvements width I of AUC score if the label for each label applying candidate data b i is assigned E (b i) It is calculated and selects the highest labeling candidate data b i of the expected value E (b i) as a labeling target data.
  • Training data selecting apparatus 100 the labeling candidate data b i is a positive class C + and the negative class C - calculating the expected value of improving the width I of the AUC score based on the likelihood belonging to E (b i). Accordingly, the learning data selecting apparatus 100, the labeling candidates from the data set B, the identification difficult labeling the current learning model candidate data b i, and the positive class C + and the negative class C - likelihood of the same degree in selecting labeling candidate data b i to be outliers preferentially.
  • the learning data selecting apparatus 100 from among the label assignment candidate data set B, and compared with the case in which the additional learning using all labeling candidate data b i, to improve the learning accuracy of the learning model efficiently It will select the labeling candidate data b i preferentially.
  • the disclosed form of the learning data selection device 100 is an example, and the form of the learning data selection device 100 is limited to the range described in the embodiment. Not done.
  • Various changes or improvements may be made to the embodiments without departing from the gist of the present disclosure, and the modified or improved forms are also included in the technical scope of the disclosure.
  • the order of the learning data selection processing shown in FIG. 3 may be changed without departing from the gist of the present disclosure.
  • the same processing as the flowchart shown in FIG. 3 can be implemented by, for example, ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Gate Array), or PLD (Programmable Logic Device). May be good. In this case, the processing speed can be increased as compared with the case where the learning data selection process is realized by software.
  • ASIC Application Specific Integrated Circuit
  • FPGA Field Programmable Gate Array
  • PLD Programmable Logic Device
  • the CPU 31 of the learning data selection device 100 may be replaced with a dedicated processor specialized for a specific process such as ASIC, FPGA, PLD, GPU (Graphics Processing Unit), and FPU (Footing Point Unit).
  • the processing of the learning data selection device 100 is executed by a combination of two or more processors of the same type or different types, such as a plurality of CPU 31, or a combination of the CPU 31 and the FPGA, in addition to the form realized by one CPU 31. May be good. Further, the processing of the learning data selection device 100 may be realized by the cooperation of a processor located outside the housing of the learning data selection device 100 and located at a physically distant place.
  • the storage destination of the learning data selection program is not limited to the ROM 32.
  • the learning data selection program of the present disclosure can also be provided in a form recorded on a storage medium readable by a computer 30.
  • the learning data selection program may be provided in the form of being recorded on an optical disk such as a CD-ROM (Compact Disk Read Only Memory) and a DVD-ROM (Digital Versaille Disk Ready Memory).
  • the learning data selection program may be provided in the form of being recorded in a portable semiconductor memory such as a USB (Universal Serial Bus) memory and a memory card.
  • the ROM 32, the non-volatile memory 34, the CD-ROM, the DVD-ROM, the USB, and the memory card are examples of non-transitory storage media.
  • the learning data selection device 100 may acquire a learning data selection program from an external device through the communication unit 37 and store the downloaded learning data selection program in, for example, the ROM 32 or the non-volatile memory 34. In this case, the learning data selection device 100 reads the learning data selection program downloaded from the external device and executes the learning data selection process.
  • Appendix 1 With memory With at least one processor connected to the memory Including The processor Each feature amount of the trained data included in the trained data set used for training the training model, and each of the labeling candidate data included in the labeling candidate data set prepared for the additional learning of the training model. Extract the feature amount of Using the feature amount of the trained data belonging to the positive class indicating the event to be estimated in the learning model and the feature amount of the trained data belonging to the negative class indicating the event other than the estimation target in the learning model. Then, a parameter representing the probability distribution of the feature amount of the learned data in the positive class and the negative class is estimated.
  • the probability that the label assignment candidate data belongs to the positive class and the probability that the label assignment candidate data belongs to the negative class are estimated for each label assignment candidate data.
  • the trained data and the label assignment candidate data are used using the trained model parameters of the function representing the input / output relationship of the learning model for estimating the event-likeness to be estimated, the trained data, and the label assignment candidate data.
  • the score which is the output of the learning model, is calculated for each of the above.
  • the improvement width of the AUC score for each label assignment candidate data the probability that the label assignment candidate data belongs to the positive class, and the probability that the label assignment candidate data belongs to the negative class are used.
  • the expected value of the improvement range of the AUC score is calculated, and the label assignment candidate data having the highest expected value of the improvement range of the AUC score among the label assignment candidate data sets is used to generate additional training data of the training model.
  • a training data selection device that is configured to be selected as the data to be labeled.
  • (Appendix 2) A non-temporary storage medium that stores a program that can be executed by a computer to perform training data selection processing.
  • the learning data selection process is Each feature amount of the trained data included in the trained data set used for training the training model, and each of the labeling candidate data included in the labeling candidate data set prepared for the additional learning of the training model. Extract the feature amount of Using the feature amount of the trained data belonging to the positive class indicating the event to be estimated in the learning model and the feature amount of the trained data belonging to the negative class indicating the event other than the estimation target in the learning model. Then, a parameter representing the probability distribution of the feature amount of the learned data in the positive class and the negative class is estimated.
  • the probability that the label assignment candidate data belongs to the positive class and the probability that the label assignment candidate data belongs to the negative class are estimated for each label assignment candidate data.
  • the trained data and the label assignment candidate data are used using the trained model parameters of the function representing the input / output relationship of the learning model for estimating the event-likeness to be estimated, the trained data, and the label assignment candidate data.
  • the score which is the output of the learning model, is calculated for each of the above.
  • the improvement width of the AUC score for each label assignment candidate data the probability that the label assignment candidate data belongs to the positive class, and the probability that the label assignment candidate data belongs to the negative class are used.
  • the expected value of the improvement range of the AUC score is calculated, and the label assignment candidate data having the highest expected value of the improvement range of the AUC score among the label assignment candidate data sets is used to generate additional training data of the training model.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Software Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Medical Informatics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Mathematical Physics (AREA)
  • Artificial Intelligence (AREA)
  • Image Analysis (AREA)

Abstract

This learning data selection device: calculates an expected range of improvement in AUC score for each candidate item of data to be labeled, using a range of improvement in AUC score for the candidate item of data to be labeled, the probability that the candidate item of data to be labeled belongs to the positive class, and the probability that the candidate item of data to be labeled belongs to the negative class; and selects, as data to be labeled that is used to generate additional learning data for a learning model, the candidate item of data to be labeled for which the expected range of improvement in AUC score is the highest among a set of candidate items of data to be labeled.

Description

学習データ選択方法、学習データ選択装置、及び学習データ選択プログラムTraining data selection method, training data selection device, and training data selection program
　本発明は、学習データ選択方法、学習データ選択装置、及び学習データ選択プログラムに関する。 The present invention relates to a learning data selection method, a learning data selection device, and a learning data selection program.
　機械学習の方法として教師あり学習を適用する場合、データに対して正解を表す情報、すなわちラベルを付与した学習データを用いる。 When supervised learning is applied as a machine learning method, information indicating the correct answer to the data, that is, learning data with a label is used.
　学習データを用いた機械学習により生成された学習モデルの精度を更に向上させるためには、学習済みデータと異なる新たなデータにラベルを付与して生成した追加学習データを用いて、再学習すればよい。 In order to further improve the accuracy of the training model generated by machine learning using the training data, it is necessary to retrain using the additional training data generated by adding a label to new data different from the trained data. good.
　しかしながら、データとラベルの対応付けを行うための人的リソース及び時間的リソースには制限がある。従って、学習済みデータと異なるすべての新たなデータ、すなわち、すべてのラベル付与候補データに対してラベルを対応付けて追加学習データを生成することは困難な場合が多い。 However, there are limits to the human and time resources for associating data with labels. Therefore, it is often difficult to generate additional training data by associating labels with all new data different from the trained data, that is, all labeling candidate data.
　上記の理由から、限られたリソースの中で学習モデルの精度を向上させるためには、学習済みデータに用いられていない複数のデータの中からできるだけ学習効果の高いデータに対してラベルを付与し、追加学習データとすることが好ましい。 For the above reasons, in order to improve the accuracy of the training model within the limited resources, labels are given to the data with the highest learning effect from among multiple data that are not used for the trained data. , It is preferable to use additional learning data.
　そうした中、ＲＯＣ（Ｒｅｃｅｉｖｅｒ　Ｏｐｅｒａｔｉｎｇ　Ｃｈａｒａｃｔｅｒｉｓｔｉｃ）の下側の面積によって表されるＡＵＣ（Ａｒｅａ　Ｕｎｄｅｒ　ｔｈｅ　Ｃｕｒｖｅ)スコアの指標において、できるだけ学習効果の高いデータを選択する手法が例えば非特許文献１に開示されている。 Under such circumstances, for example, Non-Patent Document 1 discloses a method of selecting data having as high a learning effect as possible in the index of AUC (Area Under the Curve) score represented by the area under the ROC (Receiving Operating Characteristics). There is.
＜非特許文献１＞
Culver, Matt, Deng Kun, and Stephen Scott. "Active learning to maximize area under the ROC curve." Sixth International Conference on Data Mining (ICDM'06). IEEE, 2006. <Non-Patent Document 1>
Culver, Matt, Deng Kun, and Stephen Scott. "Active learning to maximize area under the ROC curve." Sixth International Conference on Data Mining (ICDM'06). IEEE, 2006.
　非特許文献１の選択手法では学習データのクラス不均衡を解消する手法となっているが、明示的にＡＵＣスコアの改善幅を最適化していない。従って、選択したラベル付与候補データが学習済みデータに含まれるデータと類似している場合、選択したラベル付与候補データにラベルを付与して生成した追加学習データを用いて学習モデルの追加学習を行っても、ＡＵＣスコアが改善しないことがある。 The selection method of Non-Patent Document 1 is a method of eliminating the class imbalance of learning data, but the improvement range of the AUC score is not explicitly optimized. Therefore, if the selected labeling candidate data is similar to the data contained in the trained data, additional training of the training model is performed using the additional training data generated by adding a label to the selected labeling candidate data. However, the AUC score may not improve.
　従って、学習モデルの追加学習のために用意されたラベル付与候補データの中から、学習モデルに追加学習させることで追加学習を行う前よりも学習モデルの学習精度を効率よく向上させるラベル付与候補データを優先的に選択できる学習データ選択方法、学習データ選択装置、及び学習データ選択プログラムを開示する。 Therefore, from the label assignment candidate data prepared for the additional learning of the learning model, the label assignment candidate data that improves the learning accuracy of the learning model more efficiently than before the additional learning is performed by causing the learning model to perform additional learning. Disclose a learning data selection method, a learning data selection device, and a learning data selection program that can preferentially select.
　本開示の第１態様は、学習データ選択方法であって、学習モデルの学習に用いた学習済みデータ集合に含まれる学習済みデータの各々の特徴量、及び前記学習モデルの追加学習のために用意されたラベル付与候補データ集合に含まれるラベル付与候補データの各々の特徴量を抽出するステップと、前記学習モデルでの推定対象となる事象を示す正クラスに属する前記学習済みデータの特徴量、及び前記学習モデルでの推定対象以外の事象を示す負クラスに属する前記学習済みデータの特徴量を用いて、前記正クラス及び前記負クラスにおける前記学習済みデータの特徴量の確率分布を表すパラメータを推定するステップと、前記ラベル付与候補データの特徴量、及び前記パラメータを用いて、前記ラベル付与候補データ毎に、前記ラベル付与候補データが前記正クラスに属する確率、及び前記負クラスに属する確率をそれぞれ推定するステップと、推定対象である事象らしさを推定する前記学習モデルの入出力関係を表す関数の学習済みモデルパラメータ、前記学習済みデータ、及び前記ラベル付与候補データを用いて、前記学習済みデータ及び前記ラベル付与候補データの各々に対して前記学習モデルの出力であるスコアを算出するステップと、前記学習済みデータ及び前記ラベル付与候補データの各々に対するスコアを用いて、前記ラベル付与候補データ毎に、前記ラベル付与候補データが前記正クラス及び前記負クラスの何れか一方に属すると仮定した場合のそれぞれの場合におけるＡＵＣスコアの改善幅を算出するステップと、前記ラベル付与候補データ毎のＡＵＣスコアの改善幅、前記ラベル付与候補データが前記正クラスに属する確率、及び前記ラベル付与候補データが前記負クラスに属する確率を用いて、前記ラベル付与候補データ毎にＡＵＣスコアの改善幅の期待値を算出し、前記ラベル付与候補データ集合のうち、ＡＵＣスコアの改善幅の期待値が最も高い前記ラベル付与候補データを、前記学習モデルの追加学習データの生成に用いられるラベル付与対象データとして選択するステップと、を含む。 The first aspect of the present disclosure is a training data selection method, which is prepared for each feature amount of the trained data included in the trained data set used for training the training model and for additional training of the training model. The step of extracting the feature amount of each of the label assignment candidate data included in the label assignment candidate data set, the feature amount of the trained data belonging to the positive class indicating the event to be estimated by the training model, and the feature amount of the trained data. Using the feature quantity of the trained data belonging to the negative class indicating an event other than the estimation target in the training model, a parameter representing the probability distribution of the feature quantity of the trained data in the positive class and the negative class is estimated. The probability that the label assignment candidate data belongs to the positive class and the probability that the label assignment candidate data belongs to the negative class are determined for each of the label assignment candidate data by using the step to be performed, the feature amount of the label assignment candidate data, and the parameter. The trained data and the trained data and the trained data using the trained model parameters, the trained data, and the labeling candidate data of the function representing the input / output relationship of the training model for estimating the event-likeness to be estimated and the step to be estimated. For each of the label assignment candidate data, the step of calculating the score which is the output of the training model for each of the label assignment candidate data and the score for each of the trained data and the label assignment candidate data are used. The step of calculating the improvement range of the AUC score in each case assuming that the label assignment candidate data belongs to either the positive class or the negative class, and the improvement of the AUC score for each of the label assignment candidate data. Using the width, the probability that the label assignment candidate data belongs to the positive class, and the probability that the label assignment candidate data belongs to the negative class, the expected value of the improvement width of the AUC score is calculated for each of the label assignment candidate data. , The step of selecting the label assignment candidate data having the highest expected value of the improvement range of the AUC score from the label assignment candidate data set as the label assignment target data used for generating the additional training data of the training model. including.
　本開示の第２態様は、学習データ選択装置であって、学習モデルの学習に用いた学習済みデータ集合に含まれる学習済みデータの各々の特徴量、及び前記学習モデルの追加学習のために用意されたラベル付与候補データ集合に含まれるラベル付与候補データの各々の特徴量を抽出する特徴量抽出部と、前記学習モデルでの推定対象となる事象を示す正クラスに属する、前記特徴量抽出部で抽出された前記学習済みデータの特徴量、及び前記学習モデルでの推定対象以外の事象を示す負クラスに属する、前記特徴量抽出部で抽出された前記学習済みデータの特徴量を用いて、前記正クラス及び前記負クラスにおける前記学習済みデータの特徴量の確率分布を表すパラメータを推定する分布推定部と、前記特徴量抽出部で抽出された前記ラベル付与候補データの特徴量、及び前記分布推定部で推定された前記パラメータを用いて、前記ラベル付与候補データ毎に、前記ラベル付与候補データが前記正クラスに属する確率、及び前記負クラスに属する確率をそれぞれ推定する確率推定部と、推定対象である事象らしさを推定する前記学習モデルの入出力関係を表す関数の学習済みモデルパラメータ、前記学習済みデータ、及び前記ラベル付与候補データを用いて、前記学習済みデータ及び前記ラベル付与候補データの各々に対して前記学習モデルの出力であるスコアを算出するスコア算出部と、前記スコア算出部で算出された前記学習済みデータ及び前記ラベル付与候補データの各々に対するスコアを用いて、前記ラベル付与候補データ毎に、前記ラベル付与候補データが前記正クラス及び前記負クラスの何れか一方に属すると仮定した場合のそれぞれの場合におけるＡＵＣスコアの改善幅を算出するＡＵＣ改善幅算出部と、前記ＡＵＣ改善幅算出部で算出された前記ラベル付与候補データ毎のＡＵＣスコアの改善幅、前記確率推定部で推定された前記ラベル付与候補データが前記正クラスに属する確率、及び前記確率推定部で推定された前記ラベル付与候補データが前記負クラスに属する確率を用いて、前記ラベル付与候補データ毎にＡＵＣスコアの改善幅の期待値を算出し、前記ラベル付与候補データ集合のうち、ＡＵＣスコアの改善幅の期待値が最も高い前記ラベル付与候補データを、前記学習モデルの追加学習データの生成に用いられるラベル付与対象データとして選択する選択部と、を含む。 The second aspect of the present disclosure is a training data selection device, which is prepared for each feature amount of the trained data included in the trained data set used for training the training model and for additional training of the training model. The feature amount extraction unit that extracts the feature amount of each of the label assignment candidate data included in the labeled candidate data set, and the feature amount extraction unit that belongs to the positive class indicating the event to be estimated by the learning model. Using the feature amount of the trained data extracted in the above and the feature amount of the trained data extracted by the feature amount extraction unit belonging to the negative class indicating an event other than the estimation target in the learning model, the feature amount is used. The distribution estimation unit that estimates the parameters representing the probability distribution of the feature amount of the learned data in the positive class and the negative class, the feature amount of the label assignment candidate data extracted by the feature amount extraction unit, and the distribution. Using the parameters estimated by the estimation unit, the probability estimation unit that estimates the probability that the label assignment candidate data belongs to the positive class and the probability that the label assignment candidate data belongs to the negative class for each of the label assignment candidate data, and the estimation. Using the trained model parameters of the function representing the input / output relationship of the training model for estimating the likelihood of the target event, the trained data, and the label assignment candidate data, the trained data and the label assignment candidate data can be obtained. The label assignment candidate is used by using the score calculation unit that calculates the score that is the output of the learning model for each, and the scores for each of the trained data and the label assignment candidate data calculated by the score calculation unit. For each data, the AUC improvement width calculation unit that calculates the improvement width of the AUC score in each case assuming that the label assignment candidate data belongs to either the positive class or the negative class, and the AUC improvement The improvement width of the AUC score for each of the label assignment candidate data calculated by the width calculation unit, the probability that the label assignment candidate data estimated by the probability estimation unit belongs to the positive class, and the estimation by the probability estimation unit. Using the probability that the label assignment candidate data belongs to the negative class, the expected value of the improvement range of the AUC score is calculated for each of the label assignment candidate data, and the improvement range of the AUC score in the label assignment candidate data set is calculated. It includes a selection unit that selects the label assignment candidate data having the highest expected value as the label assignment target data used for generating the additional training data of the learning model.
　本開示の第３態様は、学習データ選択プログラムであって、コンピュータを、学習データ選択装置の各部として機能させる。 The third aspect of the present disclosure is a learning data selection program, in which a computer functions as each part of a learning data selection device.
　本開示の学習データ選択方法、学習データ選択装置、及び学習データ選択プログラムによれば、学習モデルの追加学習のために用意されたラベル付与候補データの中から、学習モデルに追加学習させることで追加学習を行う前よりも学習モデルの学習精度を効率よく向上させるラベル付与候補データを優先的に選択できる、という効果を有する。 According to the learning data selection method, the learning data selection device, and the learning data selection program of the present disclosure, the learning model is additionally trained from the labeling candidate data prepared for the additional learning of the learning model. It has the effect of being able to preferentially select labeling candidate data that efficiently improves the learning accuracy of the learning model compared to before learning.
学習データ選択装置の機能構成例を示す図である。It is a figure which shows the functional configuration example of the learning data selection apparatus. 学習データ選択装置に適用されるコンピュータの要部構成例を示す図である。It is a figure which shows the example of the main part structure of the computer applied to the learning data selection apparatus. 学習データ選択処理の流れの一例を示すフローチャートである。It is a flowchart which shows an example of the flow of a learning data selection process.
　以下、本開示の学習データ選択装置１００に係る実施形態例について図面を参照しながら説明する。なお、同じ構成要素及び同じ処理には全図面を通して同じ符号を付与し、重複する説明を省略する。 Hereinafter, an embodiment of the learning data selection device 100 of the present disclosure will be described with reference to the drawings. The same components and the same processing are given the same reference numerals throughout the drawings, and duplicate description will be omitted.
　図１は、学習データ選択装置１００の機能構成例を示す図である。学習データ選択装置１００は、図１に示すように入力部１０と演算部２０とを含む。 FIG. 1 is a diagram showing a functional configuration example of the learning data selection device 100. As shown in FIG. 1, the learning data selection device 100 includes an input unit 10 and a calculation unit 20.
　入力部１０は、教師あり学習によって生成される学習モデルの学習に用いられた学習済みデータａの集合である学習済みデータ集合Ａと、学習モデルの追加学習のために用意されたラベル付与候補データｂの集合であるラベル付与候補データ集合Ｂを受け付ける。また、入力部１０は、学習済みデータａの学習によって生成された学習モデルの入出力関係を表す関数ｆ、すなわち、推定対象である事象らしさを推定する関数ｆの学習済みモデルパラメータθを受け付ける。学習済みモデルパラメータθは、生成された学習モデルの入出力関係を規定するパラメータである。 The input unit 10 includes a trained data set A, which is a set of trained data a used for learning a learning model generated by supervised learning, and labeling candidate data prepared for additional learning of the learning model. Accepts the label assignment candidate data set B, which is a set of b. Further, the input unit 10 receives the trained model parameter θ of the function f representing the input / output relationship of the learning model generated by the learning of the trained data a, that is, the function f for estimating the event-likeness to be estimated. The trained model parameter θ is a parameter that defines the input / output relationship of the generated training model.
　なお、学習モデルによる推定対象に制約はなく、如何なる事象を推定対象としてもよい。以降、学習モデルでの推定対象となる事象を「正クラスＣ＋」といい、学習モデルの推定対象以外の事象を「負クラスＣ－」という。例えば画像によって表される動物が「ねこ」であるかを学習モデルで推定する場合、「ねこ」であるという事象が正クラスＣ＋となり、ねこ以外であるという事象が負クラスＣ－となる。 There are no restrictions on the estimation target by the learning model, and any event may be the estimation target. Later, an event that the estimated target of the learning model is referred to as a "positive class C +", an event other than the estimation target of the learning model "negative class C -" that. For example, when the animal represented by the image is estimated by the learning model whether the "cat", event of a "cat" is a positive class C +, and the an event that is other than cat negative Class C - a.
　演算部２０は、入力部１０で受け付けた各種データを用いて、ラベル付与候補データ集合Ｂの中から、学習モデルの追加学習データの生成に用いるラベル付与候補データｂを選択するための演算を行う。 The calculation unit 20 performs an operation for selecting label assignment candidate data b to be used for generating additional learning data of the learning model from the label assignment candidate data set B using various data received by the input unit 10. ..
　演算部２０は一例として、特徴量抽出部２１、分布推定部２２、確率推定部２３、スコア算出部２４、ＡＵＣ改善幅算出部２５、及び選択部２６を含む。 As an example, the calculation unit 20 includes a feature amount extraction unit 21, a distribution estimation unit 22, a probability estimation unit 23, a score calculation unit 24, an AUC improvement width calculation unit 25, and a selection unit 26.
　特徴量抽出部２１は、入力部１０で受け付けた学習済みデータ集合Ａとラベル付与候補データ集合Ｂを入力とし、学習済みデータ集合Ａに含まれる各学習済みデータａとラベル付与候補データ集合Ｂに含まれる各ラベル付与候補データｂの特徴量を抽出する。学習済みデータａとラベル付与候補データｂの特徴量の抽出方法に制約はない。抽出方法の一例として、ＩｍａｇｅＮｅｔデータで学習した畳み込みニューラルネットワーク（Ｃｏｎｖｏｌｕｔｉｏｎａｌ　Ｎｅｕｒａｌ　Ｎｅｔｗｏｒｋ：ＣＮＮ）を利用した特徴抽出器を用いる。ＣＮＮを利用して特徴量を抽出する代わりに、例えば局所画像特徴量のようにヒューリスティックに設計された特徴量を用いて、学習済みデータａとラベル付与候補データｂの特徴量を抽出してもよい。また、ＣＮＮを利用して特徴量を抽出する代わりに、学習済みデータ集合Ａから得られるＶａｒｉａｔｉｏｎａｌ　Ａｕｔｏ　Ｅｎｃｏｄｅｒの再構成誤差を用いて、学習済みデータａとラベル付与候補データｂの特徴量を抽出してもよい。 The feature amount extraction unit 21 inputs the trained data set A and the label assignment candidate data set B received by the input unit 10, and inputs each trained data a and the label assignment candidate data set B included in the trained data set A. The feature amount of each label addition candidate data b included is extracted. There are no restrictions on the method of extracting the features of the trained data a and the label assignment candidate data b. As an example of the extraction method, a feature extractor using a convolutional neural network (CNN) learned from ImageNet data is used. Instead of extracting the features using CNN, even if the features of the trained data a and the labeling candidate data b are extracted using the heuristically designed features such as the local image features. good. Further, instead of extracting the features using CNN, the features of the trained data a and the label assignment candidate data b are extracted by using the reconstruction error of the Variational AutoEncoder obtained from the trained data set A. You may.
　以降では、学習済みデータ集合Ａに含まれる各々の学習済みデータａの特徴量をｇ（ａ）と表し、ラベル付与候補データ集合Ｂに含まれる各々のラベル付与候補データｂの特徴量をｇ（ｂ）と表す。なお、特徴量抽出部２１を、学習済みデータａの特徴量ｇ（ａ）を抽出する抽出部と、ラベル付与候補データｂの特徴量ｇ（ｂ）を抽出する抽出部に分離してもよい。 Hereinafter, the feature amount of each trained data a included in the trained data set A is represented by g (a), and the feature amount of each label assignment candidate data b included in the label assignment candidate data set B is g ( It is expressed as b). The feature amount extraction unit 21 may be separated into an extraction unit for extracting the feature amount g (a) of the learned data a and an extraction unit for extracting the feature amount g (b) of the label assignment candidate data b. ..
　分布推定部２２は、特徴量抽出部２１で抽出した学習済みデータａの各々の特徴量ｇ（ａ）を入力とし、学習済みデータａに対して、正クラスＣ＋及び負クラスＣ－における各々の確率分布のパラメータω＋、ω－を推定する。 Distribution estimating unit 22, each of the feature quantity g of the learned data a extracted by the feature amount extracting section 21 (a) as input for the learned data a, the positive class C + and the negative class C - each in parameters of the probability distribution of ω +, ω - to estimate.
　具体的には、分布推定部２２は各々の特徴量ｇ（ａ）を、正クラスＣ＋に属する学習済みデータ集合Ａ＋に含まれる学習済みデータａ＋(ａ＋∈Ａ＋)の特徴量ｇ（ａ＋）と、負クラスＣ－に属する学習済みデータ集合Ａ－に含まれる学習済みデータａ－（ａ－∈Ａ－）の特徴量ｇ（ａ－）に分類する。分布推定部２２は、分類した特徴量ｇ（ａ＋）及び特徴量ｇ（ａ－）がそれぞれ示す正クラスＣ＋の確率分布ｈ＋（ｇ（ａ＋）；ω＋）、及び負クラスＣ－の確率分布ｈ－（ｇ（ａ－）；ω－)の各々について、例えば正規分布モデルを適用してモデル化した確率分布のパラメータω＋、ω－を推定する。なお、確率分布ｈ＋（ｇ（ａ＋）；ω＋）及び確率分布ｈ－（ｇ（ａ－）；ω－)のモデル化には、正規分布の代わりにベルヌーイ分布又はポアソン分布といった他の確率分布を適用してもよい。 Specifically, the distribution estimation unit 22 uses each feature amount g (a) as a feature amount of the trained data a + (a + ∈ A + ) included in the trained data set A + belonging to the positive class C +. It is classified into g (a + ) and the feature quantity g (a − ) of the trained data a − (a − ∈ A − ) contained in the trained data set A − belonging to the negative class C −. The distribution estimation unit 22 has a positive class C + probability distribution h + (g (a + ); ω + ) and a negative class C indicated by the classified feature amount g (a + ) and feature amount g (a −), respectively. - probability distribution h of - (g (a -); ω -) for each of, for example, the parameters of the probability distribution modeled by applying a normal distribution model omega +, omega - estimated. Incidentally, the probability distribution h + (g (a +) ; ω +) and probability distributions h - (g (a -) ; ω -) of the modeling, other such Bernoulli distribution or a Poisson distribution instead of the normal distribution A probability distribution may be applied.
　また、分布推定部２２を、特徴量ｇ（ａ＋）を用いて確率分布ｈ＋（ｇ（ａ＋）；ω＋）のパラメータω＋を推定する推定部と、特徴量ｇ（ａ－）を用いて確率分布ｈ－（ｇ（ａ－）；ω－)のパラメータω－を推定する推定部に分離してもよい。 Further, the distribution estimation unit 22 has an estimation unit that estimates the parameter ω + of the probability distribution h + (g (a + ); ω + ) using the feature amount g (a + ), and the feature amount g (a − ). parameters of omega - may be separated in the estimation unit for estimating a; (- - ω g (a -)) probability distribution h using.
　確率推定部２３は、特徴量抽出部２１で抽出したラベル付与候補データｂの各々の特徴量ｇ（ｂ）、及び分布推定部２２で推定したパラメータω＋、ω－をそれぞれ特徴量抽出部２１及び分布推定部２２から受け付ける。その上で、確率推定部２３は、特徴量ｇ（ｂ）及びパラメータω＋、ω－を入力として、ラベル付与候補データｂ毎に、ラベル付与候補データｂが正クラスＣ＋に属する確率p、及び負クラスＣ－に属する確率pをそれぞれ推定する。 The probability estimation unit 23 extracts each feature amount g (b) of the label assignment candidate data b extracted by the feature amount extraction unit 21, and the parameters ω + and ω − estimated by the distribution estimation unit 22, respectively, in the feature amount extraction unit 21. And received from the distribution estimation unit 22. Then, the probability estimation unit 23 inputs the feature amount g (b) and the parameters ω + and ω −, and the probability p that the label assignment candidate data b belongs to the positive class C + for each label assignment candidate data b, And the probability p belonging to the negative class C -is estimated respectively.
　以降では説明の便宜上、ラベル付与候補データｂの各々に注目して説明を行う場合、ラベル付与候補データｂを特に「ラベル付与候補データｂｉ」と表すことがある。“ｉ”はラベル付与候補データｂを一意に示すためのインデックスである。 For convenience of explanation below, the case of performing description by focusing on each of the labeling candidate data b, which may in particular labeling candidate data b representing the "labeling candidate data b i". “I” is an index for uniquely indicating the label assignment candidate data b.
　ラベル付与候補データｂｉが正クラスＣ＋若しくは負クラスＣ－に属する確率ｐを推定するにあたって、各クラスＣ＋、Ｃ－は、学習済みデータ集合Ａによってモデル化された確率分布に含まれるクラスと、当該確率分布に含まれないクラスの２種類の集合によって構成されていると考えられる。 Class contained in the probability distribution that is modeled by the trained data set A - labeling candidate data b i is a positive class C + or negative class C - when estimating the probability belonging to p, each class C +, C And, it is considered that it is composed of two kinds of sets of classes not included in the probability distribution.
　ここで、正クラスＣ＋において確率分布に含まれるクラスをＣ＋Ｉ、確率分布外となるクラスをＣ＋Ｏとし、負クラスＣ－において確率分布に含まれるクラスをＣ－Ｉ、確率分布外となるクラスをＣ－Ｏとする。 Here, class C + I included in the probability distribution in the positive class C +, the class as a probability distribution outside the C + O, minus Class C - consisting of classes in the probability distribution in C -I, and the probability distribution outside Let the class be CO.
　ラベル付与候補データｂｉの属するクラスＣｉが正クラスＣ＋となる確率ｐ（ｃｉ＝Ｃ＋｜ｇ（ｂｉ））は、確率分布に含まれるクラスＣ＋Ｉから発生する確率ｐ（ｃｉ＝Ｃ＋Ｉ｜ｇ（ｂｉ））と、確率分布外のクラスＣ＋Ｏから発生する確率ｐ（ｃｉ＝Ｃ＋Ｏ｜ｇ（ｂｉ））の和となる。従って、確率ｐ（ｃｉ＝Ｃ＋｜ｇ（ｂｉ））は（１）式で表され、（２）式のように展開される。 Probability p class C i which belongs labeling candidate data b i is a positive class C + (c i = C + | g (b i)) , the probability p (c generated from class C + I contained in the probability distribution i = C + I | is the sum of | (g (b i c i = C + O)) and g (b i)), the probability generated from class C + O outside the probability distribution p. Thus, the probability p (c i = C + | g (b i)) is represented by equation (1) is expanded as (2).
Figure JPOXMLDOC01-appb-M000001
Figure JPOXMLDOC01-appb-M000001
　同様に、ラベル付与候補データｂｉの属するクラスＣｉが負クラスＣ－となる確率ｐ（ｃｉ＝Ｃ－｜ｇ（ｂｉ））は、確率分布に含まれるクラスＣ－Ｉから発生する確率ｐ（ｃｉ＝Ｃ－Ｉ｜ｇ（ｂｉ））と、確率分布外のクラスＣ－Ｏから発生する確率ｐ（ｃｉ＝Ｃ－Ｏ｜ｇ（ｂｉ））の和となる。従って、確率ｐ（ｃｉ＝Ｃ－｜ｇ（ｂｉ））は（３）式によって表される。 Similarly, labeling candidate data b i class C i is negative class belongs C - become probability p (c i = C - | g (b i)) is generated from the class C -I contained in the probability distribution probability p is the sum of | | (g (b i) c i = C -O) and (c i = C -I g ( b i)), the probability generated from the probability distribution outside of class C -O p. Thus, the probability p (c i = C - | g (b i)) is represented by equation (3).
Figure JPOXMLDOC01-appb-M000002
Figure JPOXMLDOC01-appb-M000002
　ここで、ｐ（ｇ（ｂｉ）｜ｃｉ＝Ｃ＋Ｉ）は、正クラスＣ＋の確率分布ｈ＋（ｇ（ａ＋）；ω＋）とパラメータω＋を用いて、ラベル付与候補データｂｉについて確率分布ｈ＋（ｇ（ａ＋）；ω＋）から発生する確率分布ｈ＋（ｇ（ｂｉ）；ω＋）を計算することによって得られる値を用いればよい。ｐ（ｇ（ｂｉ）｜ｃｉ＝Ｃ－Ｉ）は、負クラスＣ－の確率分布ｈ－（ｇ（ａ－）；ω－)とパラメータω－を用いて、ラベル付与候補データｂｉについて確率分布ｈ－（ｇ（ａ－）；ω－)から発生する確率分布ｈ－（ｇ（ｂｉ）；ω－）を計算することによって得られる値を用いればよい。 Here, p (g (b i) | c i = C + I) is a positive class C + probability distribution h + (g (a +) ; ω +) and by using the parameter omega +, labeling candidate data b i for the probability distribution h + (g (a +) ; ω +) probabilities generated from the distribution h + (g (b i) ; ω +) may be used a value obtained by calculating the. p (g (b i) | c i = C -I) , a negative class C - probability distributions h - (g (a -) ; ω -) as a parameter omega - using a labeling candidate data b i probability distributions h - (g (a -) ; ω -) probabilities generated from the distribution h - (g (b i) ; ω -) may be used a value obtained by calculating the.
　一方、ｐ（ｇ（ｂｉ）｜ｃｉ＝Ｃ＋Ｏ）及びｐ（ｇ（ｂｉ）｜ｃｉ＝Ｃ－Ｏ）に関して、ラベル付与候補データｂｉが正クラスＣ＋及び負クラスＣ－に属する場合、ｐ（ｇ（ｂｉ）｜ｃｉ＝Ｃ＋Ｏ）及びｐ（ｇ（ｂｉ）｜ｃｉ＝Ｃ－Ｏ）の各々の確率分布は一様分布に従うとする。正クラスＣ＋の確率分布ｈ＋（ｇ（ａ＋）；ω＋）を表す正規分布モデルの平均値をμ＋、分散をσ＋ ２とすれば、３σ＋の範囲内に９９．７％の学習済みデータａが含まれることから、ｐ（ｇ（ｂｉ）｜ｃｉ＝Ｃ＋Ｏ）の確率分布はｈ＋（μ＋＋３σ＋；ω＋)で表されるが、他の定義を用いて表してもよい。同様に、負クラスＣ－の確率分布ｈ－（ｇ（ａ－）；ω－）を表す正規分布モデルの平均値をμ－、分散をσ－ ２とすれば、ｐ（ｇ（ｂｉ）｜ｃｉ＝Ｃ－Ｏ）の確率分布は、例えばｈ－(μ－+３σ－；ω－)で表される。 On the other hand, p (g (b i) | c i = C + O) and p (g (b i) | c i = C -O) respect, labeling candidate data b i is a positive class C + and the negative class C - If belonging, p (g (b i) | c i = C + O) and p (g (b i) | c i = C -O) each of the probability distribution of a uniformly distributed. If the average value of the normal distribution model representing the probability distribution h + (g (a + ); ω + ) of the positive class C + is μ + and the variance is σ + 2 , then 99.7% within the range of 3σ +. from be included learned data a, p (g (b i ) | c i = C + O) probability distribution of h +; is represented by (μ + + 3σ + ω + ), the other definitions It may be expressed using. Similarly, the negative class C - probability distributions h - (g (a -) ; ω -) the average value of the normal distribution model representing the mu -, the variance sigma - if 2, p (g (b i) The probability distribution of | c i = C − O ) is represented by, for example, h − (μ − + 3σ − ; ω − ).
　ここで、学習済みデータａ＋の個数をｎ（Ａ＋）、学習済みデータａ－の個数をｎ（Ａ－）、ラベル付与候補データ集合Ｂのうち、ラベル付与候補データｂｉが確率分布ｈ＋（ｇ（ｂｉ）；ω＋）及びｈ－（ｇ（ｂｉ）；ω－）からそれぞれ外れる割合をｔとする。この場合、ｐ（ｃｉ＝Ｃ＋Ｉ）、ｐ（ｃｉ＝Ｃ－Ｉ）、ｐ（ｃｉ＝Ｃ＋Ｏ）、及びｐ（ｃｉ＝Ｃ－Ｏ）はそれぞれ（４）式～（７）式によって表される。なお、割合ｔは、実験によって定められる値である。 Here, the learned data a + number of n (A +), the learned data a - the number of n (A -), among the labeling candidate data set B, the labeling candidate data b i is a probability distribution h + (g (b i); ω +) and h - (g (b i) ; ω -) , respectively outside a proportion of the t. In this case, p (c i = C + I), p (c i = C -I), p (c i = C + O), and p (c i = C -O), respectively (4) to (7 ) Is expressed by the formula. The ratio t is a value determined by an experiment.
Figure JPOXMLDOC01-appb-M000003
Figure JPOXMLDOC01-appb-M000003
　スコア算出部２４は、事象が正クラスＣ＋に属する度合い、すなわち、事象の正クラスＣ＋らしさを推定する学習モデルによって表される関数ｆの学習済みモデルパラメータθ、学習済みデータ集合Ａ、及びラベル付与候補データ集合Ｂを入力部１０から受け付ける。 The score calculation unit 24 includes the trained model parameter θ of the function f represented by the learning model for estimating the degree to which the event belongs to the positive class C + , that is, the positive class C + of the event, the trained data set A, and the trained data set A. The label assignment candidate data set B is received from the input unit 10.
　スコア算出部２４は、学習済みモデルパラメータθによって表される関数ｆに、学習済みデータ集合Ａに含まれる学習済みデータａの各々とラベル付与候補データ集合Ｂに含まれるラベル付与候補データｂの各々を入力する。これにより、スコア算出部２４は、学習済みデータａ及びラベル付与候補データｂの各々に対する学習モデルの出力であるスコアｆθ（ａ）及びスコアｆθ（ｂ）を算出する。 The score calculation unit 24 has, in the function f represented by the trained model parameter θ, each of the trained data a included in the trained data set A and each of the label assignment candidate data b included in the label assignment candidate data set B. Enter. As a result, the score calculation unit 24 calculates the score f θ (a) and the score f θ (b), which are the outputs of the training model for each of the trained data a and the label assignment candidate data b.
　なお、スコア算出部２４を、学習済みデータａを用いてスコアｆθ（ａ）を算出する算出部と、ラベル付与候補データｂを用いてスコアｆθ（ｂ）を算出する算出部に分離してもよい。 The score calculation unit 24 is separated into a calculation unit that calculates the score f θ (a) using the learned data a and a calculation unit that calculates the score f θ (b) using the label assignment candidate data b. You may.
　ＡＵＣ改善幅算出部２５は、算出されたスコアｆθ（ａ）及びスコアｆθ（ｂ）をスコア算出部２４から受け付ける。ＡＵＣ改善幅算出部２５はスコアｆθ（ａ）及びスコアｆθ（ｂ）を用いて、ラベル付与候補データｂ毎に、ラベル付与候補データｂが正クラスＣ＋及び負クラスＣ－の何れか一方に属すると仮定したそれぞれの場合におけるＡＵＣスコアの改善幅Ｉを算出する。ＡＵＣスコアの改善幅Ｉは、現状の学習モデルによって算出されるＡＵＣスコアと、現状の学習モデルに対して、何らかのラベルを付与したラベル付与候補データｂｉを用いて追加学習させた追加学習モデルによって算出されるＡＵＣスコアの差分で表される。 The AUC improvement width calculation unit 25 receives the calculated score f θ (a) and score f θ (b) from the score calculation unit 24. The AUC improvement width calculation unit 25 uses the score f θ (a) and the score f θ (b), and the label assignment candidate data b is either positive class C + or negative class C − for each label assignment candidate data b. The improvement width I of the AUC score in each case assuming that it belongs to one is calculated. Improvements width I of the AUC score, and AUC score calculated by the current learning model for current learning model, the additional learning model obtained by additional learning using label assignment candidate data b i imparted with some label It is represented by the difference between the calculated AUC scores.
　関数Ｈをヘヴィサイド関数とすれば、現状の学習モデルにおけるＡＵＣスコアＡＵＣは（８）式によって表される。 If the function H is a heavyside function, the AUC score AUC in the current learning model is expressed by Eq. (8).
Figure JPOXMLDOC01-appb-M000004
Figure JPOXMLDOC01-appb-M000004
　任意の１つのラベル付与候補データｂｉに対して、ラベル付与候補データｂｉが正クラスＣ＋に属することを表すラベル、すなわちＣ＋ラベルを付与した場合に、現状の学習モデルによって算出されるＡＵＣスコアＡＵＣ＋（ｂｉ）は（９）式によって表される。 For any one label assignment candidate data b i, labels labeling candidate data b i represents belong to positive class C +, i.e. when applying the C + labels, is calculated by the current learning model AUC score AUC + (b i) it is represented by equation (9).
Figure JPOXMLDOC01-appb-M000005
Figure JPOXMLDOC01-appb-M000005
　次に、Ｃ＋ラベルを付与したラベル付与候補データｂｉを追加学習データに加えて現状の学習モデルを追加学習した追加学習モデルによって算出されるＡＵＣスコアＡＵＣ＋ｔａｒｇｅｔ（ｂｉ）について検討する。Ｃ＋ラベルを付与したラベル付与候補データｂｉ以外にＡＵＣスコアの変動要素がないと仮定すれば、すべての学習済みデータａ－に対してｆ（ｂｉ）＞ｆ（ａ－）となるまで学習モデルの更新が行われることになる。こうした追加学習モデルによって算出されるＡＵＣスコアＡＵＣ＋ｔａｒｇｅｔ（ｂｉ）は（１０）式によって表される。 Next, consider the C + label imparted with labeling candidate data b i AUC score is calculated by adding the learning model in addition to the additional learning data added learning the current status of the learning models AUC + target (b i). Assuming that there is no variation element in AUC score than C + Labels imparted with grant candidate data b i, all learned data a - and until - against f (b i)> f ( a) The learning model will be updated. Is calculated by such additional learning model AUC score AUC + target (b i) is represented by equation (10).
Figure JPOXMLDOC01-appb-M000006
Figure JPOXMLDOC01-appb-M000006
　従って、ラベル付与候補データｂｉにＣ＋ラベルを付与した場合のＡＵＣスコアの改善幅Ｉ＋（ｂｉ）は、（１１）式によって表される。 Accordingly, + improving width of AUC scores If granted the C + labels labeling candidate data b i I (b i) is represented by equation (11).
Figure JPOXMLDOC01-appb-M000007
Figure JPOXMLDOC01-appb-M000007
　同様に、ラベル付与候補データｂｉにＣ－ラベルを付与した場合のＡＵＣスコアの改善幅Ｉ－（ｂｉ）は、（１２）式によって表される。 Similarly, C to labeling candidate data b i - improving width of AUC scores If granted the label I - (b i) is represented by equation (12).
Figure JPOXMLDOC01-appb-M000008
Figure JPOXMLDOC01-appb-M000008
　ＡＵＣ改善幅算出部２５は、ラベル付与候補データｂｉの各々について、ＡＵＣスコアの改善幅Ｉ＋（ｂｉ）及びＩ－（ｂｉ）を算出する。 AUC improvements width calculating unit 25, for each of the labeling candidate data b i, improved width I + (b i) of the AUC scores and I - to calculate the (b i).
　選択部２６は、確率推定部２３からラベル付与候補データｂｉが正クラスＣ＋に属する確率ｐ（ｃｉ＝Ｃ＋｜ｇ（ｂｉ））と、ラベル付与候補データｂｉが負クラスＣ－に属する確率ｐ（ｃｉ＝Ｃ－｜ｇ（ｂｉ））を受け付ける。また、選択部２６は、ＡＵＣ改善幅算出部２５からラベル付与候補データｂｉ毎のＡＵＣスコアの改善幅Ｉ＋（ｂｉ）及びＩ－（ｂｉ）を受け付ける。 Selecting unit 26, the probability from a probability estimation section 23 labeling candidate data b i belonging to the positive class C + p | a (c i = C + g ( b i)), the labeling candidate data b i is negative class C - probability belong to p - | accepts (c i = C g (b i)). The selection unit 26, improved from AUC improve width calculating section 25 of the AUC scores for each labeling candidate data b i width I + (b i) and I - accepting (b i).
　選択部２６は、ラベル付与候補データｂｉ毎のＡＵＣスコアの改善幅Ｉ＋（ｂｉ）及びＩ－（ｂｉ）、並びに、確率ｐ（ｃｉ＝Ｃ＋｜ｇ（ｂｉ））及び確率ｐ（ｃｉ＝Ｃ－｜ｇ（ｂｉ））を用いて、ラベル付与候補データｂｉ毎にＡＵＣスコアの改善幅Ｉの期待値Ｅ（ｂｉ）を算出する。その上で、選択部２６は、ラベル付与候補データ集合Ｂのうち、ＡＵＣスコアの改善幅Ｉの期待値Ｅ（ｂｉ）が最も高いラベル付与候補データｂｉを、学習モデルの追加学習データの生成に用いられるラベル付与対象データとして選択する。 Selecting unit 26, improved width I + (b i) and I of AUC scores for each labeling candidate data b i - (b i), and the probability p (c i = C + | g (b i)) and probability p - | with (c i = C g (b i)), calculates the expected value of improving the width I of the AUC score E (b i) for each label applying candidate data b i. On top of that, the selection unit 26 of the labeling candidate data set B, and expectation E (b i) is the highest labeling candidate data b i improvements width I of AUC scores, the learning model of additional training data Select as the data to be labeled as used for generation.
　具体的には、ラベル付与候補データｂｉにおけるＡＵＣスコアの改善幅Ｉの期待値Ｅ（ｂｉ）は、（１３）式によって表される。 Specifically, the expected value E of the improved width I of AUC score in the label applying candidate data b i (b i) is represented by (13).
Figure JPOXMLDOC01-appb-M000009
Figure JPOXMLDOC01-appb-M000009
　すなわち、選択部２６は、ラベル付与候補データｂｉが正クラスＣ＋に属する確率ｐ（ｃｉ＝Ｃ＋｜ｇ（ｂｉ））とラベル付与候補データｂｉにＣ＋ラベルを付与した場合のＡＵＣスコアの改善幅Ｉ＋（ｂｉ）との積を算出する。その上で、選択部２６は、当該算出した積にラベル付与候補データｂｉが負クラスＣ－に属する確率ｐ（ｃｉ＝Ｃ－｜ｇ（ｂｉ））とラベル付与候補データｂｉにＣ－ラベルを付与した場合のＡＵＣスコアの改善幅Ｉ－（ｂｉ）との積を加算して、ＡＵＣスコアの改善幅Ｉの期待値Ｅ（ｂｉ）を算出する。 That is, the selection unit 26, the labeling candidate data b i is a positive class C probability p that belongs to + | when the C + label given to (= c i C + g ( b i)) and labeling candidate data b i calculating the product of the improved width I + (b i) of the AUC score. On top of that, the selection unit 26, the calculated product to the label applying candidate data b i is negative Class C - probability p that belongs to (c i = C - | g (b i)) and the label assignment candidate data b i C - improvement width of AUC scores If granted the label I - by adding the product of (b i), and calculates the expected value of improving the width I of the AUC score E (b i).
　なお、（１３）式はＡＵＣスコアの改善幅Ｉの期待値Ｅ（ｂｉ）を算出する算出式の一例である。例えば選択部２６は、（１３）式の右辺の各項に重み付けを表す係数を積算して、ＡＵＣスコアの改善幅Ｉの期待値Ｅ（ｂｉ）を算出してもよい。 Note that (13) is an example of a calculation formula for calculating the expected value E (b i) improvements width I of the AUC score. For example selecting section 26 may calculate the (13) by integrating the coefficients representing the weighting sections the right side of the equation, the expected value of the improvements width I of AUC score E (b i).
　学習データ選択装置１００によってラベル付与候補データ集合Ｂの中から選択されたラベル付与候補データｂｉにラベル付けを行えば、学習モデルでのＡＵＣスコアの改善幅Ｉを最大にする追加学習データが得られることになる。従って、当該追加学習データを用いて学習モデルの追加学習を行えば、ラベル付与候補データ集合Ｂの中からランダムに選択したラベル付与候補データｂｉで生成した追加学習データを用いる場合と比較して、学習モデルの学習精度が効率よく向上することになる。 By performing the labeling labeling candidate data b i, which is selected from the label assignment candidate data set B by the learning data selecting apparatus 100, additional training data to maximize the improvement width I of the AUC scores in learning model is obtained Will be. Therefore, by performing the additional learning of the learning model using the additional learning data, as compared with the case of using the additional learning data generated by the label applying candidate data b i selected at random from the label assignment candidate data set B , The learning accuracy of the learning model will be improved efficiently.
　こうした学習データ選択装置１００は、一例としてコンピュータ３０を用いて構成される。 The learning data selection device 100 is configured by using a computer 30 as an example.
　図２は、学習データ選択装置１００に適用されるコンピュータ３０の要部構成例を示す図である。 FIG. 2 is a diagram showing a configuration example of a main part of the computer 30 applied to the learning data selection device 100.
　コンピュータ３０は、図１に示した学習データ選択装置１００の各部における処理を担うＣＰＵ（Ｃｅｎｔｒａｌ　Ｐｒｏｃｅｓｓｉｎｇ　Ｕｎｉｔ）３１を含む。また、コンピュータ３０は、コンピュータ３０を学習データ選択装置１００として機能させる学習データ選択プログラムを記憶するＲＯＭ（Ｒｅａｄ　Ｏｎｌｙ　Ｍｅｍｏｒｙ）３２、及びＣＰＵ３１の一時的な作業領域として使用されるＲＡＭ（Ｒａｎｄｏｍ　Ａｃｃｅｓｓ　Ｍｅｍｏｒｙ）３３を含む。更に、ＣＰＵ３１は、不揮発性メモリ３４、及び入出力インターフェース（Ｉ／Ｏ）３５を備える。そして、ＣＰＵ３１、ＲＯＭ３２、ＲＡＭ３３、不揮発性メモリ３４、及びＩ／Ｏ３５がバス３６によって各々接続される。 The computer 30 includes a CPU (Central Processing Unit) 31 that is responsible for processing in each part of the learning data selection device 100 shown in FIG. Further, the computer 30 has a ROM (Read Only Memory) 32 for storing a learning data selection program that causes the computer 30 to function as a learning data selection device 100, and a RAM (Random Access Memory) used as a temporary work area of the CPU 31. Includes 33. Further, the CPU 31 includes a non-volatile memory 34 and an input / output interface (I / O) 35. Then, the CPU 31, ROM 32, RAM 33, non-volatile memory 34, and I / O 35 are each connected by the bus 36.
　不揮発性メモリ３４は、不揮発性メモリ３４に供給される電力が遮断されても、記憶した情報が維持される記憶装置の一例であり、例えば半導体メモリが用いられるがハードディスクを用いてもよい。不揮発性メモリ３４はコンピュータ３０に含まれなくてもよく、例えばコンピュータ３０に着脱可能な可搬型の記憶装置を不揮発性メモリ３４として利用してもよい。 The non-volatile memory 34 is an example of a storage device in which the stored information is maintained even if the power supplied to the non-volatile memory 34 is cut off. For example, a semiconductor memory is used, but a hard disk may be used. The non-volatile memory 34 does not have to be included in the computer 30, and for example, a portable storage device that can be attached to and detached from the computer 30 may be used as the non-volatile memory 34.
　Ｉ／Ｏ３５には、例えば通信ユニット３７、入力ユニット３８、及び表示ユニット３９が接続される。 For example, a communication unit 37, an input unit 38, and a display unit 39 are connected to the I / O 35.
　通信ユニット３７は例えばインターネット及びＬＡＮ（Ｌｏｃａｌ　Ａｒｅａ　Ｎｅｔｗｏｒｋ）のような通信回線に接続され、通信回線に接続される外部装置との間でデータ通信を行う通信プロトコルを備える。通信回線には有線通信、又はＷｉ－Ｆｉ（登録商標）といった無線通信が用いられる。 The communication unit 37 is connected to a communication line such as the Internet and a LAN (Local Area Network), and includes a communication protocol for performing data communication with an external device connected to the communication line. Wired communication or wireless communication such as Wi-Fi (registered trademark) is used as the communication line.
　入力ユニット３８は、ユーザの指示を受け付けてＣＰＵ３１に通知する装置であり、例えばボタン、タッチパネル、キーボード、及びマウスが用いられる。音声によって指示を受け付ける場合には、入力ユニット３８としてマイクが用いられることがある。 The input unit 38 is a device that receives a user's instruction and notifies the CPU 31, for example, a button, a touch panel, a keyboard, and a mouse are used. When receiving an instruction by voice, a microphone may be used as the input unit 38.
　表示ユニット３９は、ＣＰＵ３１によって処理された情報を視覚的に表示する装置の一例であり、例えば液晶ディスプレイ、有機ＥＬ（Ｅｌｅｃｔｒｏ Ｌｕｍｉｎｅｓｃｅｎｃｅ）ディスプレイ、又はプロジェクタが用いられる。 The display unit 39 is an example of a device that visually displays information processed by the CPU 31, and for example, a liquid crystal display, an organic EL (Electroluminescence) display, or a projector is used.
　入力部１０への学習済みデータ集合Ａ、ラベル付与候補データ集合Ｂ、及び学習済みモデルパラメータθの入力は、例えば通信ユニット３７又はコンピュータ３０に着脱可能な可搬型の不揮発性メモリ３４を介して行われる。特に、コンピュータ３０に着脱可能な可搬型の不揮発性メモリ３４を介して入力部１０に各種データが入力される場合、コンピュータ３０は必ずしも通信ユニット３７を備える必要はない。また、例えば学習データ選択装置１００が無人のデータセンターに設置され、通信回線を通じて遠隔地から制御を受け付ける場合には、コンピュータ３０は必ずしも入力ユニット３８及び表示ユニット３９を備える必要はない。 The trained data set A, the label assignment candidate data set B, and the trained model parameter θ are input to the input unit 10 via, for example, a portable non-volatile memory 34 that can be attached to and detached from the communication unit 37 or the computer 30. Will be. In particular, when various data are input to the input unit 10 via the portable non-volatile memory 34 that can be attached to and detached from the computer 30, the computer 30 does not necessarily have to include the communication unit 37. Further, for example, when the learning data selection device 100 is installed in an unmanned data center and receives control from a remote location through a communication line, the computer 30 does not necessarily have to include the input unit 38 and the display unit 39.
　また、Ｉ／Ｏ３５に接続される各種ユニットは一例であり、例えば情報を文字又は画像として記録媒体に形成する画像形成ユニットをＩ／Ｏ３５に接続してもよい。 Further, various units connected to the I / O 35 are an example, and for example, an image forming unit that forms information on a recording medium as characters or images may be connected to the I / O 35.
　次に、本開示の学習データ選択装置１００の作用について説明する。入力部１０において、学習済みデータ集合Ａ、ラベル付与候補データ集合Ｂ、及び学習モデルの学習済みモデルパラメータθを受け付けると、学習データ選択装置１００のＣＰＵ３１は、図３に示すフローチャートに従って学習データ選択処理を実行する。 Next, the operation of the learning data selection device 100 of the present disclosure will be described. When the input unit 10 receives the trained data set A, the label assignment candidate data set B, and the trained model parameter θ of the training model, the CPU 31 of the training data selection device 100 performs the training data selection process according to the flowchart shown in FIG. To execute.
　学習データ選択処理を規定する学習データ選択プログラムは、例えば学習データ選択装置１００のＲＯＭ３２に予め記憶されている。学習データ選択装置１００のＣＰＵ３１は、ＲＯＭ３２に記憶される学習データ選択プログラムを読み込んで学習データ選択処理を実行する。なお、入力部１０で受け付けた学習済みデータ集合Ａ、ラベル付与候補データ集合Ｂ、及び学習済みモデルパラメータθはＲＡＭ３３に記憶される。 The learning data selection program that defines the learning data selection process is stored in advance in, for example, the ROM 32 of the learning data selection device 100. The CPU 31 of the learning data selection device 100 reads the learning data selection program stored in the ROM 32 and executes the learning data selection process. The trained data set A, the label assignment candidate data set B, and the trained model parameter θ received by the input unit 10 are stored in the RAM 33.
　まず、ステップＳ１０において、ＣＰＵ３１は、学習済みデータ集合Ａに含まれる各々の学習済みデータａの特徴量ｇ（ａ）、及びラベル付与候補データ集合Ｂに含まれる各々のラベル付与候補データｂの特徴量ｇ（ｂ）を抽出し、ＲＡＭ３３に記憶する。 First, in step S10, the CPU 31 features the feature amount g (a) of each trained data a included in the trained data set A and the features of each label assignment candidate data b included in the label assignment candidate data set B. The amount g (b) is extracted and stored in the RAM 33.
　ステップＳ２０において、ＣＰＵ３１は、学習済みデータａを学習済みデータａ＋と学習済みデータａ－に分類する。更に、ＣＰＵ３１は、学習済みデータａ＋の特徴量ｇ（ａ＋）及び学習済みデータａ－の特徴量ｇ（ａ－）を用いて、学習済みデータａに対して、正クラスＣ＋及び負クラスＣ－における各々の特徴量ｇ（ａ）の確率分布を表すパラメータω＋、ω－を推定し、推定結果をＲＡＭ３３に記憶する。 In step S20, the CPU 31 classifies the trained data a into the trained data a + and the trained data a −. Further, CPU 31 is learned data a + feature quantity g (a +) and the learned data a - feature quantity g (a -) with respect to the learned data a, the positive class C + and the negative The parameters ω + and ω − representing the probability distribution of each feature amount g (a) in class C − are estimated, and the estimation result is stored in the RAM 33.
　ステップＳ３０において、ＣＰＵ３１は、ステップＳ１０で抽出したラベル付与候補データｂの特徴量ｇ（ｂ）、及びステップＳ２０で推定したパラメータω＋、ω－をＲＡＭ３３から取得する。その上で、ＣＰＵ３１は、ラベル付与候補データｂｉ毎に、ラベル付与候補データｂｉが正クラスＣ＋に属する確率ｐ（ｃｉ＝Ｃ＋｜ｇ（ｂｉ））、及び負クラスＣ－に属する確率ｐ（ｃｉ＝Ｃ－｜ｇ（ｂｉ））を（２）式～（７）式に従ってそれぞれ推定し、推定結果をＲＡＭ３３に記憶する。 In step S30, the CPU 31 acquires the feature amount g (b) of the label assignment candidate data b extracted in step S10 and the parameters ω + and ω − estimated in step S20 from the RAM 33. On top of that, CPU 31, for each label applying candidate data b i, the probability labeling candidate data b i belong to the positive class C + p (c i = C + | g (b i)), and the negative class C - probability p belonging to (c i = C - | g (b i)) and (2) were respectively estimated according to expression (7), stores the estimation result to RAM 33.
　ステップＳ４０において、ＣＰＵ３１は、入力部１０で受け付けた学習済みデータ集合Ａ、ラベル付与候補データ集合Ｂ、及び学習済みモデルパラメータθをＲＡＭ３３から取得する。その上で、ＣＰＵ３１は、学習済みデータ集合Ａに含まれる各々の学習済みデータａとラベル付与候補データ集合Ｂに含まれる各々のラベル付与候補データｂに対して、スコアｆθ（ａ）及びスコアｆθ（ｂ）を算出し、スコアｆθ（ａ）及びスコアｆθ（ｂ）をＲＡＭ３３に記憶する。 In step S40, the CPU 31 acquires the trained data set A, the label assignment candidate data set B, and the trained model parameter θ received by the input unit 10 from the RAM 33. Then, the CPU 31 has a score f θ (a) and a score for each trained data a included in the trained data set A and each label assignment candidate data b included in the label assignment candidate data set B. f θ (b) is calculated, and the score f θ (a) and the score f θ (b) are stored in the RAM 33.
　ステップＳ５０において、ＣＰＵ３１は、ステップＳ４０で算出したスコアｆθ（ａ）及びスコアｆθ（ｂ）をＲＡＭ３３から取得する。ＣＰＵ３１は、ラベル付与候補データｂｉ毎に、ラベル付与候補データｂｉにＣ＋ラベルを付与した場合のＡＵＣスコアの改善幅Ｉ＋（ｂｉ）、及びラベル付与候補データｂｉにＣ－ラベルを付与した場合のＡＵＣスコアの改善幅Ｉ－（ｂｉ）をそれぞれ（１１）式及び（１２）式に従って算出する。ＣＰＵ３１は、算出したＡＵＣスコアの改善幅Ｉ＋（ｂｉ）及びＩ－（ｂｉ）をＲＡＭ３３に記憶する。 In step S50, the CPU 31 acquires the score f θ (a) and the score f θ (b) calculated in step S40 from the RAM 33. CPU31, for each label applying candidate data b i, improved width I + (b i) of the AUC scores If granted the C + labels labeling candidate data b i, and C in labeling candidate data b i - label improvements width of AUC scores If granted the I - (b i) the calculated according respectively (11) and (12). CPU31 is improved calculated AUC score width I + (b i) and I - storing (b i) into RAM 33.
　ステップＳ６０において、ＣＰＵ３１は、ステップＳ５０で算出したＡＵＣスコアの改善幅Ｉ＋（ｂｉ）及びＩ－（ｂｉ）、並びに、ステップＳ３０で推定した確率ｐ（ｃｉ＝Ｃ＋｜ｇ（ｂｉ））及び確率ｐ（ｃｉ＝Ｃ－｜ｇ（ｂｉ））をＲＡＭ３３から取得する。その上で、ＣＰＵ３１は、ラベル付与候補データｂｉ毎に、ＡＵＣスコアの改善幅Ｉの期待値Ｅ（ｂｉ）を（１３）式に従って算出し、算出した各々のラベル付与候補データｂｉにおけるスコアの改善幅Ｉの期待値Ｅ（ｂｉ）をＲＡＭ３３に記憶する。 In step S60, CPU 31 is improved calculated AUC score step S50 width I + (b i) and I - (b i), and the probability estimated in step S30 p (c i = C + | g (b i)) and probability p (c i = C - | acquiring g of (b i)) from the RAM 33. On top of that, CPU 31, for each label applying candidate data b i, the expected value of the improvements width I of AUC score E a (b i) (13) is calculated according to equation of the calculated respectively in labeling candidate data b i storing the expected value E of the improved width I of scores (b i) into RAM 33.
　ＣＰＵ３１は、ラベル付与候補データ集合Ｂのうち、ＡＵＣスコアの改善幅Ｉの期待値Ｅ（ｂｉ）が最も高いラベル付与候補データｂｉを、学習モデルの追加学習データの生成に用いられるラベル付与対象データとして選択し、図３に示す学習データ選択処理を終了する。 CPU31, of the labeling candidate data set B, labeling used the expectation E (b i) is the highest labeling candidate data b i improvements width I of AUC score, the generation of the additional learning data of the learning model It is selected as the target data, and the learning data selection process shown in FIG. 3 is completed.
　このように、本開示の学習データ選択装置１００の例によれば、ラベル付与候補データｂｉの各々に対してラベルが付与された場合のＡＵＣスコアの改善幅Ｉの期待値Ｅ（ｂｉ）を算出し、期待値Ｅ（ｂｉ）の最も高いラベル付与候補データｂｉをラベル付与対象データとして選択する。 Thus, according to the example of the training data selecting apparatus 100 of the present disclosure, the expected value of the improvements width I of AUC score if the label for each label applying candidate data b i is assigned E (b i) It is calculated and selects the highest labeling candidate data b i of the expected value E (b i) as a labeling target data.
　学習データ選択装置１００は、ラベル付与候補データｂｉが正クラスＣ＋及び負クラスＣ－に属する尤度に基づいてＡＵＣスコアの改善幅Ｉの期待値Ｅ（ｂｉ）を算出する。従って、学習データ選択装置１００は、ラベル付与候補データ集合Ｂの中から、現状の学習モデルでは識別困難なラベル付与候補データｂｉ、及び正クラスＣ＋及び負クラスＣ－における尤度が同程度で外れ値となるラベル付与候補データｂｉを優先的に選択する。その結果、学習データ選択装置１００は、ラベル付与候補データ集合Ｂの中から、全てのラベル付与候補データｂｉを用いて追加学習を行う場合に比べて、学習モデルの学習精度を効率よく向上させるラベル付与候補データｂｉを優先的に選択することになる。 Training data selecting apparatus 100, the labeling candidate data b i is a positive class C + and the negative class C - calculating the expected value of improving the width I of the AUC score based on the likelihood belonging to E (b i). Accordingly, the learning data selecting apparatus 100, the labeling candidates from the data set B, the identification difficult labeling the current learning model candidate data b i, and the positive class C + and the negative class C - likelihood of the same degree in selecting labeling candidate data b i to be outliers preferentially. As a result, the learning data selecting apparatus 100, from among the label assignment candidate data set B, and compared with the case in which the additional learning using all labeling candidate data b i, to improve the learning accuracy of the learning model efficiently It will select the labeling candidate data b i preferentially.
　以上、実施形態を用いて学習データ選択装置１００の一態様について説明したが、開示した学習データ選択装置１００の形態は一例であり、学習データ選択装置１００の形態は実施形態に記載の範囲に限定されない。本開示の要旨を逸脱しない範囲で実施形態に多様な変更又は改良を加えることができ、当該変更又は改良を加えた形態も開示の技術的範囲に含まれる。例えば、本開示の要旨を逸脱しない範囲で、図３に示した学習データ選択処理の順序を変更してもよい。 Although one aspect of the learning data selection device 100 has been described above using the embodiment, the disclosed form of the learning data selection device 100 is an example, and the form of the learning data selection device 100 is limited to the range described in the embodiment. Not done. Various changes or improvements may be made to the embodiments without departing from the gist of the present disclosure, and the modified or improved forms are also included in the technical scope of the disclosure. For example, the order of the learning data selection processing shown in FIG. 3 may be changed without departing from the gist of the present disclosure.
　また、本開示では、一例として学習データ選択処理をソフトウェアで実現する形態について説明した。しかしながら、図３に示したフローチャートと同等の処理を、例えばＡＳＩＣ（Ａｐｐｌｉｃａｔｉｏｎ Ｓｐｅｃｉｆｉｃ　Ｉｎｔｅｇｒａｔｅｄ　Ｃｉｒｃｕｉｔ）、ＦＰＧＡ（Ｆｉｅｌｄ　Ｐｒｏｇｒａｍｍａｂｌｅ　Ｇａｔｅ　Ａｒｒａｙ）、又はＰＬＤ（Ｐｒｏｇｒａｍｍａｂｌｅ　Ｌｏｇｉｃ　Ｄｅｖｉｃｅ）に実装し、ハードウェアで処理させるようにしてもよい。この場合、学習データ選択処理をソフトウェアで実現した場合と比較して処理の高速化が図られる。 Further, in this disclosure, as an example, a form in which learning data selection processing is realized by software has been described. However, the same processing as the flowchart shown in FIG. 3 can be implemented by, for example, ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Gate Array), or PLD (Programmable Logic Device). May be good. In this case, the processing speed can be increased as compared with the case where the learning data selection process is realized by software.
　このように、学習データ選択装置１００のＣＰＵ３１を例えばＡＳＩＣ、ＦＰＧＡ、ＰＬＤ、ＧＰＵ（Ｇｒａｐｈｉｃｓ　Ｐｒｏｃｅｓｓｉｎｇ　Ｕｎｉｔ）、及びＦＰＵ（Ｆｏａｔｉｎｇ　Ｐｏｉｎｔ　Ｕｎｉｔ）といった特定の処理に特化した専用のプロセッサに置き換えてもよい。 As described above, the CPU 31 of the learning data selection device 100 may be replaced with a dedicated processor specialized for a specific process such as ASIC, FPGA, PLD, GPU (Graphics Processing Unit), and FPU (Footing Point Unit).
　学習データ選択装置１００の処理は、１つのＣＰＵ３１によって実現される形態の他、複数のＣＰＵ３１、又はＣＰＵ３１とＦＰＧＡとの組み合わせというように、同種又は異種の２つ以上のプロセッサの組み合わせで実行してもよい。更に、学習データ選択装置１００の処理は、学習データ選択装置１００の筐体の外部に位置する、物理的に離れた場所に存在するプロセッサの協働によって実現されるものであってもよい。 The processing of the learning data selection device 100 is executed by a combination of two or more processors of the same type or different types, such as a plurality of CPU 31, or a combination of the CPU 31 and the FPGA, in addition to the form realized by one CPU 31. May be good. Further, the processing of the learning data selection device 100 may be realized by the cooperation of a processor located outside the housing of the learning data selection device 100 and located at a physically distant place.
　実施形態では、学習データ選択装置１００のＲＯＭ３２に学習データ選択プログラムが記憶されている例について説明したが、学習データ選択プログラムの記憶先はＲＯＭ３２に限定されない。本開示の学習データ選択プログラムは、コンピュータ３０で読み取り可能な記憶媒体に記録された形態で提供することも可能である。例えば学習データ選択プログラムをＣＤ－ＲＯＭ（Ｃｏｍｐａｃｔ　Ｄｉｓｋ　Ｒｅａｄ　Ｏｎｌｙ　Ｍｅｍｏｒｙ）及びＤＶＤ－ＲＯＭ（Ｄｉｇｉｔａｌ　Ｖｅｒｓａｔｉｌｅ　Ｄｉｓｋ　Ｒｅａｄ　Ｏｎｌｙ　Ｍｅｍｏｒｙ）のような光ディスクに記録した形態で提供してもよい。また、学習データ選択プログラムを、ＵＳＢ（Ｕｎｉｖｅｒｓａｌ　Ｓｅｒｉａｌ　Ｂｕｓ）メモリ及びメモリカードのような可搬型の半導体メモリに記録した形態で提供してもよい。ＲＯＭ３２、不揮発性メモリ３４、ＣＤ－ＲＯＭ、ＤＶＤ－ＲＯＭ、ＵＳＢ、及びメモリカードは非一時的（ｎｏｎ－ｔｒａｎｓｉｔｏｒｙ）記憶媒体の一例である。 In the embodiment, an example in which the learning data selection program is stored in the ROM 32 of the learning data selection device 100 has been described, but the storage destination of the learning data selection program is not limited to the ROM 32. The learning data selection program of the present disclosure can also be provided in a form recorded on a storage medium readable by a computer 30. For example, the learning data selection program may be provided in the form of being recorded on an optical disk such as a CD-ROM (Compact Disk Read Only Memory) and a DVD-ROM (Digital Versaille Disk Ready Memory). Further, the learning data selection program may be provided in the form of being recorded in a portable semiconductor memory such as a USB (Universal Serial Bus) memory and a memory card. The ROM 32, the non-volatile memory 34, the CD-ROM, the DVD-ROM, the USB, and the memory card are examples of non-transitory storage media.
　更に、学習データ選択装置１００は、通信ユニット３７を通じて外部装置から学習データ選択プログラムを取得し、ダウンロードした学習データ選択プログラムを、例えばＲＯＭ３２又は不揮発性メモリ３４に記憶してもよい。この場合、学習データ選択装置１００は、外部装置からダウンロードした学習データ選択プログラムを読み込んで学習データ選択処理を実行する。 Further, the learning data selection device 100 may acquire a learning data selection program from an external device through the communication unit 37 and store the downloaded learning data selection program in, for example, the ROM 32 or the non-volatile memory 34. In this case, the learning data selection device 100 reads the learning data selection program downloaded from the external device and executes the learning data selection process.
　本明細書に記載された全ての文献、特許出願、及び技術規格は、個々の文献、特許出願、及び技術規格が参照により取り込まれることが具体的かつ個々に記された場合と同程度に、本明細書中に参照により取り込まれる。 All documents, patent applications, and technical standards described herein are to the same extent as if the individual documents, patent applications, and technical standards were specifically and individually stated to be incorporated by reference. Incorporated by reference herein.
　以上の実施形態に関し、更に以下の付記を開示する。 Regarding the above embodiments, the following additional notes will be further disclosed.
（付記項１）
　メモリと、
　前記メモリに接続された少なくとも１つのプロセッサと、
　を含み、
　前記プロセッサは、
　学習モデルの学習に用いた学習済みデータ集合に含まれる学習済みデータの各々の特徴量、及び前記学習モデルの追加学習のために用意されたラベル付与候補データ集合に含まれるラベル付与候補データの各々の特徴量を抽出し、
　前記学習モデルでの推定対象となる事象を示す正クラスに属する前記学習済みデータの特徴量、及び前記学習モデルでの推定対象以外の事象を示す負クラスに属する前記学習済みデータの特徴量を用いて、前記正クラス及び前記負クラスにおける前記学習済みデータの特徴量の確率分布を表すパラメータを推定し、
　前記ラベル付与候補データの特徴量、及び前記パラメータを用いて、前記ラベル付与候補データ毎に、前記ラベル付与候補データが前記正クラスに属する確率、及び前記負クラスに属する確率をそれぞれ推定し、
　推定対象である事象らしさを推定する前記学習モデルの入出力関係を表す関数の学習済みモデルパラメータ、前記学習済みデータ、及び前記ラベル付与候補データを用いて、前記学習済みデータ及び前記ラベル付与候補データの各々に対して前記学習モデルの出力であるスコアを算出し、
　前記学習済みデータ及び前記ラベル付与候補データの各々に対するスコアを用いて、前記ラベル付与候補データ毎に、前記ラベル付与候補データが前記正クラス及び前記負クラスの何れか一方に属すると仮定した場合のそれぞれの場合におけるＡＵＣスコアの改善幅を算出し、
　前記ラベル付与候補データ毎のＡＵＣスコアの改善幅、前記ラベル付与候補データが前記正クラスに属する確率、及び前記ラベル付与候補データが前記負クラスに属する確率を用いて、前記ラベル付与候補データ毎にＡＵＣスコアの改善幅の期待値を算出し、前記ラベル付与候補データ集合のうち、ＡＵＣスコアの改善幅の期待値が最も高い前記ラベル付与候補データを、前記学習モデルの追加学習データの生成に用いられるラベル付与対象データとして選択する
　ように構成されている学習データ選択装置。 (Appendix 1)
With memory
With at least one processor connected to the memory
Including
The processor
Each feature amount of the trained data included in the trained data set used for training the training model, and each of the labeling candidate data included in the labeling candidate data set prepared for the additional learning of the training model. Extract the feature amount of
Using the feature amount of the trained data belonging to the positive class indicating the event to be estimated in the learning model and the feature amount of the trained data belonging to the negative class indicating the event other than the estimation target in the learning model. Then, a parameter representing the probability distribution of the feature amount of the learned data in the positive class and the negative class is estimated.
Using the feature amount of the label assignment candidate data and the parameter, the probability that the label assignment candidate data belongs to the positive class and the probability that the label assignment candidate data belongs to the negative class are estimated for each label assignment candidate data.
The trained data and the label assignment candidate data are used using the trained model parameters of the function representing the input / output relationship of the learning model for estimating the event-likeness to be estimated, the trained data, and the label assignment candidate data. The score, which is the output of the learning model, is calculated for each of the above.
When it is assumed that the label assignment candidate data belongs to either the positive class or the negative class for each label assignment candidate data by using the scores for each of the trained data and the label assignment candidate data. Calculate the improvement range of the AUC score in each case,
For each label assignment candidate data, the improvement width of the AUC score for each label assignment candidate data, the probability that the label assignment candidate data belongs to the positive class, and the probability that the label assignment candidate data belongs to the negative class are used. The expected value of the improvement range of the AUC score is calculated, and the label assignment candidate data having the highest expected value of the improvement range of the AUC score among the label assignment candidate data sets is used to generate additional training data of the training model. A training data selection device that is configured to be selected as the data to be labeled.
（付記項２）
　学習データ選択処理を実行するようにコンピュータによって実行可能なプログラムを記憶した非一時的記憶媒体であって、
　前記学習データ選択処理は、
　学習モデルの学習に用いた学習済みデータ集合に含まれる学習済みデータの各々の特徴量、及び前記学習モデルの追加学習のために用意されたラベル付与候補データ集合に含まれるラベル付与候補データの各々の特徴量を抽出し、
　前記学習モデルでの推定対象となる事象を示す正クラスに属する前記学習済みデータの特徴量、及び前記学習モデルでの推定対象以外の事象を示す負クラスに属する前記学習済みデータの特徴量を用いて、前記正クラス及び前記負クラスにおける前記学習済みデータの特徴量の確率分布を表すパラメータを推定し、
　前記ラベル付与候補データの特徴量、及び前記パラメータを用いて、前記ラベル付与候補データ毎に、前記ラベル付与候補データが前記正クラスに属する確率、及び前記負クラスに属する確率をそれぞれ推定し、
　推定対象である事象らしさを推定する前記学習モデルの入出力関係を表す関数の学習済みモデルパラメータ、前記学習済みデータ、及び前記ラベル付与候補データを用いて、前記学習済みデータ及び前記ラベル付与候補データの各々に対して前記学習モデルの出力であるスコアを算出し、
　前記学習済みデータ及び前記ラベル付与候補データの各々に対するスコアを用いて、前記ラベル付与候補データ毎に、前記ラベル付与候補データが前記正クラス及び前記負クラスの何れか一方に属すると仮定した場合のそれぞれの場合におけるＡＵＣスコアの改善幅を算出し、
　前記ラベル付与候補データ毎のＡＵＣスコアの改善幅、前記ラベル付与候補データが前記正クラスに属する確率、及び前記ラベル付与候補データが前記負クラスに属する確率を用いて、前記ラベル付与候補データ毎にＡＵＣスコアの改善幅の期待値を算出し、前記ラベル付与候補データ集合のうち、ＡＵＣスコアの改善幅の期待値が最も高い前記ラベル付与候補データを、前記学習モデルの追加学習データの生成に用いられるラベル付与対象データとして選択する
　非一時的記憶媒体。 (Appendix 2)
A non-temporary storage medium that stores a program that can be executed by a computer to perform training data selection processing.
The learning data selection process is
Each feature amount of the trained data included in the trained data set used for training the training model, and each of the labeling candidate data included in the labeling candidate data set prepared for the additional learning of the training model. Extract the feature amount of
Using the feature amount of the trained data belonging to the positive class indicating the event to be estimated in the learning model and the feature amount of the trained data belonging to the negative class indicating the event other than the estimation target in the learning model. Then, a parameter representing the probability distribution of the feature amount of the learned data in the positive class and the negative class is estimated.
Using the feature amount of the label assignment candidate data and the parameter, the probability that the label assignment candidate data belongs to the positive class and the probability that the label assignment candidate data belongs to the negative class are estimated for each label assignment candidate data.
The trained data and the label assignment candidate data are used using the trained model parameters of the function representing the input / output relationship of the learning model for estimating the event-likeness to be estimated, the trained data, and the label assignment candidate data. The score, which is the output of the learning model, is calculated for each of the above.
When it is assumed that the label assignment candidate data belongs to either the positive class or the negative class for each label assignment candidate data by using the scores for each of the trained data and the label assignment candidate data. Calculate the improvement range of the AUC score in each case,
For each label assignment candidate data, the improvement width of the AUC score for each label assignment candidate data, the probability that the label assignment candidate data belongs to the positive class, and the probability that the label assignment candidate data belongs to the negative class are used. The expected value of the improvement range of the AUC score is calculated, and the label assignment candidate data having the highest expected value of the improvement range of the AUC score among the label assignment candidate data sets is used to generate additional training data of the training model. The non-temporary storage medium selected for the data to be labeled.

Claims (7)

  1. 　学習モデルの学習に用いた学習済みデータ集合に含まれる学習済みデータの各々の特徴量、及び前記学習モデルの追加学習のために用意されたラベル付与候補データ集合に含まれるラベル付与候補データの各々の特徴量を抽出するステップと、
    　前記学習モデルでの推定対象となる事象を示す正クラスに属する前記学習済みデータの特徴量、及び前記学習モデルでの推定対象以外の事象を示す負クラスに属する前記学習済みデータの特徴量を用いて、前記正クラス及び前記負クラスにおける前記学習済みデータの特徴量の確率分布を表すパラメータを推定するステップと、
    　前記ラベル付与候補データの特徴量、及び前記パラメータを用いて、前記ラベル付与候補データ毎に、前記ラベル付与候補データが前記正クラスに属する確率、及び前記負クラスに属する確率をそれぞれ推定するステップと、
    　推定対象である事象らしさを推定する前記学習モデルの入出力関係を表す関数の学習済みモデルパラメータ、前記学習済みデータ、及び前記ラベル付与候補データを用いて、前記学習済みデータ及び前記ラベル付与候補データの各々に対して前記学習モデルの出力であるスコアを算出するステップと、
    　前記学習済みデータ及び前記ラベル付与候補データの各々に対するスコアを用いて、前記ラベル付与候補データ毎に、前記ラベル付与候補データが前記正クラス及び前記負クラスの何れか一方に属すると仮定した場合のそれぞれの場合におけるＡＵＣスコアの改善幅を算出するステップと、
    　前記ラベル付与候補データ毎のＡＵＣスコアの改善幅、前記ラベル付与候補データが前記正クラスに属する確率、及び前記ラベル付与候補データが前記負クラスに属する確率を用いて、前記ラベル付与候補データ毎にＡＵＣスコアの改善幅の期待値を算出し、前記ラベル付与候補データ集合のうち、ＡＵＣスコアの改善幅の期待値が最も高い前記ラベル付与候補データを、前記学習モデルの追加学習データの生成に用いられるラベル付与対象データとして選択するステップと、
    　を含む学習データ選択方法。     Each feature amount of the trained data included in the trained data set used for training the training model, and each of the labeling candidate data included in the labeling candidate data set prepared for the additional learning of the training model. Steps to extract the feature amount of
    Using the feature amount of the trained data belonging to the positive class indicating the event to be estimated in the learning model and the feature amount of the trained data belonging to the negative class indicating the event other than the estimation target in the learning model. Then, the step of estimating the parameter representing the probability distribution of the feature amount of the learned data in the positive class and the negative class, and
    A step of estimating the probability that the label assignment candidate data belongs to the positive class and the probability that the label assignment candidate data belongs to the negative class for each label assignment candidate data using the feature amount of the label assignment candidate data and the parameter. ,
    The trained data and the label assignment candidate data are used using the trained model parameters of the function representing the input / output relationship of the learning model for estimating the event-likeness to be estimated, the trained data, and the label assignment candidate data. And the step of calculating the score which is the output of the learning model for each of
    When it is assumed that the label assignment candidate data belongs to either the positive class or the negative class for each label assignment candidate data by using the scores for each of the trained data and the label assignment candidate data. Steps to calculate the improvement range of AUC score in each case,
    For each label assignment candidate data, the improvement width of the AUC score for each label assignment candidate data, the probability that the label assignment candidate data belongs to the positive class, and the probability that the label assignment candidate data belongs to the negative class are used. The expected value of the improvement range of the AUC score is calculated, and the label assignment candidate data having the highest expected value of the improvement range of the AUC score among the label assignment candidate data sets is used to generate additional training data of the training model. The steps to be selected as the data to be labeled and
    Training data selection method including.
  2. 　前記ラベル付与候補データ集合から前記ラベル付与対象データを選択するステップにおいて、前記ラベル付与候補データが前記正クラスに属する確率と前記正クラスに属するとのラベルを付与した場合の前記ラベル付与候補データにおけるＡＵＣスコアの改善幅との積に、前記ラベル付与候補データが前記負クラスに属する確率と前記負クラスに属するとのラベルを付与した場合の前記ラベル付与候補データにおけるＡＵＣスコアの改善幅との積を加算することで、前記ラベル付与候補データにおけるＡＵＣスコアの改善幅の期待値を算出する
    　請求項１記載の学習データ選択方法。     In the label assignment candidate data when the probability that the label assignment candidate data belongs to the positive class and the label that the label assignment candidate data belongs to the positive class are given in the step of selecting the label assignment target data from the label assignment candidate data set. The product of the product of the improvement width of the AUC score and the improvement width of the AUC score in the label assignment candidate data when the label that the label assignment candidate data belongs to the negative class is given and the label belongs to the negative class. The training data selection method according to claim 1, wherein the expected value of the improvement range of the AUC score in the label assignment candidate data is calculated by adding.
  3. 　前記正クラス及び前記負クラスを、各クラスに対応した前記確率分布に含まれるクラスと前記確率分布に含まれないクラスの集合として扱い、
    　前記ラベル付与候補データが前記正クラスに属する確率を、前記ラベル付与候補データが前記正クラスにおいて前記確率分布に含まれるクラスから発生する確率と前記正クラスにおいて前記確率分布外となるクラスから発生する確率の和によって推定し、
    　前記ラベル付与候補データが前記負クラスに属する確率を、前記ラベル付与候補データが前記負クラスにおいて前記確率分布に含まれるクラスから発生する確率と前記負クラスにおいて前記確率分布外となるクラスから発生する確率の和によって推定する
    　請求項１又は請求項２に記載の学習データ選択方法。     The positive class and the negative class are treated as a set of classes included in the probability distribution and classes not included in the probability distribution corresponding to each class.
    The probability that the label assignment candidate data belongs to the positive class is generated from the probability that the label assignment candidate data is generated from the class included in the probability distribution in the positive class and the probability that the label assignment candidate data is generated from the class outside the probability distribution in the positive class. Estimated by the sum of probabilities,
    The probability that the label assignment candidate data belongs to the negative class is generated from the probability that the label assignment candidate data is generated from the class included in the probability distribution in the negative class and the probability that the label assignment candidate data is generated from the class outside the probability distribution in the negative class. The training data selection method according to claim 1 or claim 2, which is estimated by the sum of probabilities.
  4. 　学習モデルの学習に用いた学習済みデータ集合に含まれる学習済みデータの各々の特徴量、及び前記学習モデルの追加学習のために用意されたラベル付与候補データ集合に含まれるラベル付与候補データの各々の特徴量を抽出する特徴量抽出部と、
    　前記学習モデルでの推定対象となる事象を示す正クラスに属する、前記特徴量抽出部で抽出された前記学習済みデータの特徴量、及び前記学習モデルでの推定対象以外の事象を示す負クラスに属する、前記特徴量抽出部で抽出された前記学習済みデータの特徴量を用いて、前記正クラス及び前記負クラスにおける前記学習済みデータの特徴量の確率分布を表すパラメータを推定する分布推定部と、
    　前記特徴量抽出部で抽出された前記ラベル付与候補データの特徴量、及び前記分布推定部で推定された前記パラメータを用いて、前記ラベル付与候補データ毎に、前記ラベル付与候補データが前記正クラスに属する確率、及び前記負クラスに属する確率をそれぞれ推定する確率推定部と、
    　推定対象である事象らしさを推定する前記学習モデルの入出力関係を表す関数の学習済みモデルパラメータ、前記学習済みデータ、及び前記ラベル付与候補データを用いて、前記学習済みデータ及び前記ラベル付与候補データの各々に対して前記学習モデルの出力であるスコアを算出するスコア算出部と、
    　前記スコア算出部で算出された前記学習済みデータ及び前記ラベル付与候補データの各々に対するスコアを用いて、前記ラベル付与候補データ毎に、前記ラベル付与候補データが前記正クラス及び前記負クラスの何れか一方に属すると仮定した場合のそれぞれの場合におけるＡＵＣスコアの改善幅を算出するＡＵＣ改善幅算出部と、
    　前記ＡＵＣ改善幅算出部で算出された前記ラベル付与候補データ毎のＡＵＣスコアの改善幅、前記確率推定部で推定された前記ラベル付与候補データが前記正クラスに属する確率、及び前記確率推定部で推定された前記ラベル付与候補データが前記負クラスに属する確率を用いて、前記ラベル付与候補データ毎にＡＵＣスコアの改善幅の期待値を算出し、前記ラベル付与候補データ集合のうち、ＡＵＣスコアの改善幅の期待値が最も高い前記ラベル付与候補データを、前記学習モデルの追加学習データの生成に用いられるラベル付与対象データとして選択する選択部と、
    　を含む学習データ選択装置。     Each feature of the trained data included in the trained data set used for training the training model, and each of the labeling candidate data included in the labeling candidate data set prepared for the additional learning of the training model. The feature amount extraction unit that extracts the feature amount of
    The feature amount of the trained data extracted by the feature amount extraction unit, which belongs to the positive class indicating the event to be estimated by the learning model, and the negative class indicating the event other than the estimation target by the learning model. A distribution estimation unit that estimates a parameter representing the probability distribution of the feature amount of the learned data in the positive class and the negative class by using the feature amount of the learned data extracted by the feature amount extraction unit to which the feature amount belongs. ,
    Using the feature amount of the label assignment candidate data extracted by the feature amount extraction unit and the parameter estimated by the distribution estimation unit, the label assignment candidate data is the positive class for each label assignment candidate data. A probability estimation unit that estimates the probability of belonging to the negative class and the probability of belonging to the negative class, respectively.
    The trained data and the label assignment candidate data are used using the trained model parameters of the function representing the input / output relationship of the learning model for estimating the event-likeness to be estimated, the trained data, and the label assignment candidate data. A score calculation unit that calculates the score that is the output of the learning model for each of
    Using the scores for each of the learned data and the label assignment candidate data calculated by the score calculation unit, the label assignment candidate data is either the positive class or the negative class for each label assignment candidate data. The AUC improvement width calculation unit that calculates the improvement width of the AUC score in each case assuming that it belongs to one of them,
    The improvement width of the AUC score for each label assignment candidate data calculated by the AUC improvement width calculation unit, the probability that the label assignment candidate data estimated by the probability estimation unit belongs to the positive class, and the probability estimation unit. Using the probability that the estimated label assignment candidate data belongs to the negative class, the expected value of the improvement range of the AUC score is calculated for each label assignment candidate data, and the AUC score of the label assignment candidate data set is calculated. A selection unit that selects the label assignment candidate data having the highest expected value of improvement as the label assignment target data used for generating additional training data of the training model.
    Training data selection device including.
  5. 　前記選択部は、前記ラベル付与候補データが前記正クラスに属する確率と前記正クラスに属するとのラベルを付与した場合の前記ラベル付与候補データにおけるＡＵＣスコアの改善幅との積に、前記ラベル付与候補データが前記負クラスに属する確率と前記負クラスに属するとのラベルを付与した場合の前記ラベル付与候補データにおけるＡＵＣスコアの改善幅との積を加算することで、前記ラベル付与候補データにおけるＡＵＣスコアの改善幅の期待値を算出し、前記ラベル付与候補データ集合のうち、ＡＵＣスコアの改善幅の期待値が最も高い前記ラベル付与候補データを、前記学習モデルの追加学習データの生成に用いられるラベル付与対象データとして選択する
    　請求項４記載の学習データ選択装置。     The selection unit assigns the label to the product of the probability that the label assignment candidate data belongs to the positive class and the improvement range of the AUC score in the label assignment candidate data when the label that the label assignment candidate data belongs to the positive class is given. By adding the product of the probability that the candidate data belongs to the negative class and the improvement range of the AUC score in the label assignment candidate data when the label that the candidate data belongs to the negative class is given, the AUC in the label assignment candidate data is added. The expected value of the improvement range of the score is calculated, and the label assignment candidate data having the highest expected value of the improvement range of the AUC score among the label assignment candidate data sets is used to generate additional training data of the training model. The training data selection device according to claim 4, which is selected as the data to be labeled.
  6. 　前記確率推定部は、前記正クラス及び前記負クラスを、各クラスに対応した前記確率分布に含まれるクラスと前記確率分布に含まれないクラスの集合として扱い、
    　前記ラベル付与候補データが前記正クラスに属する確率を、前記ラベル付与候補データが前記正クラスにおいて前記確率分布に含まれるクラスから発生する確率と前記正クラスにおいて前記確率分布外となるクラスから発生する確率の和によって推定し、
    　前記ラベル付与候補データが前記負クラスに属する確率を、前記ラベル付与候補データが前記負クラスにおいて前記確率分布に含まれるクラスから発生する確率と前記負クラスにおいて前記確率分布外となるクラスから発生する確率の和によって推定する
    　請求項４又は請求項５に記載の学習データ選択装置。     The probability estimation unit treats the positive class and the negative class as a set of classes included in the probability distribution corresponding to each class and classes not included in the probability distribution.
    The probability that the label assignment candidate data belongs to the positive class is generated from the probability that the label assignment candidate data is generated from the class included in the probability distribution in the positive class and the probability that the label assignment candidate data is generated from the class outside the probability distribution in the positive class. Estimated by the sum of probabilities,
    The probability that the label assignment candidate data belongs to the negative class is generated from the probability that the label assignment candidate data is generated from the class included in the probability distribution in the negative class and the probability that the label assignment candidate data is generated from the class outside the probability distribution in the negative class. The learning data selection device according to claim 4 or claim 5, which is estimated by the sum of probabilities.
  7. 　コンピュータを、請求項４～請求項６の何れか１項に記載の学習データ選択装置の各部として機能させるための学習データ選択プログラム。 A learning data selection program for making a computer function as each part of the learning data selection device according to any one of claims 4 to 6.
PCT/JP2020/023371 2020-06-15 2020-06-15 Learning data selection method, learning data selection device, and learning data selection program WO2021255778A1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
PCT/JP2020/023371 WO2021255778A1 (en) 2020-06-15 2020-06-15 Learning data selection method, learning data selection device, and learning data selection program

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2020/023371 WO2021255778A1 (en) 2020-06-15 2020-06-15 Learning data selection method, learning data selection device, and learning data selection program

Publications (1)

Publication Number Publication Date
WO2021255778A1 true WO2021255778A1 (en) 2021-12-23

Family

ID=79268617

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2020/023371 WO2021255778A1 (en) 2020-06-15 2020-06-15 Learning data selection method, learning data selection device, and learning data selection program

Country Status (1)

Country Link
WO (1) WO2021255778A1 (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2024057599A1 (en) * 2022-09-13 2024-03-21 株式会社日立国際電気 Classification device, classification method, and classification program

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2012117966A1 (en) * 2011-02-28 2012-09-07 日本電気株式会社 Data discrimination device, method, and program
WO2020090413A1 (en) * 2018-10-31 2020-05-07 日本電信電話株式会社 Classification device, classification method, and classification program

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2012117966A1 (en) * 2011-02-28 2012-09-07 日本電気株式会社 Data discrimination device, method, and program
WO2020090413A1 (en) * 2018-10-31 2020-05-07 日本電信電話株式会社 Classification device, classification method, and classification program

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2024057599A1 (en) * 2022-09-13 2024-03-21 株式会社日立国際電気 Classification device, classification method, and classification program

Similar Documents

Publication Publication Date Title
JP6889728B2 (en) Structural learning in convolutional neural networks
CN109271521B (en) Text classification method and device
CN110633786A (en) Techniques for determining artificial neural network topology
JP6973106B2 (en) Learning programs, learning methods and learning devices
JP6506360B2 (en) Method of generating teacher data, method of generating learned model, learned model, computer and program
JP6633476B2 (en) Attribute estimation device, attribute estimation method, and attribute estimation program
JP2020046883A (en) Classification device, classification method, and program
WO2021240707A1 (en) Data classification system, data classification method, and recording medium
JP2019159576A (en) Learning program, learning method and learning device
WO2016112782A1 (en) Method and system of extracting user living range
JP2015169951A (en) information processing apparatus, information processing method, and program
Nguyen et al. Incomplete label multiple instance multiple label learning
JP2015036939A (en) Feature extraction program and information processing apparatus
JPWO2017183548A1 (en) Information processing system, information processing method, and recording medium
WO2021255778A1 (en) Learning data selection method, learning data selection device, and learning data selection program
JP6535134B2 (en) Creation device, creation program, and creation method
JP2019160236A (en) Learning data generation method, learning data generation program and data structure
CN114238764A (en) Course recommendation method, device and equipment based on recurrent neural network
JP2015130093A (en) image recognition algorithm combination selection device
JP7031686B2 (en) Image recognition systems, methods and programs, as well as parameter learning systems, methods and programs
CN111666965A (en) Multi-level depth feature and multi-matcher fusion for improved image recognition
JP5633424B2 (en) Program and information processing system
JP7283554B2 (en) LEARNING DEVICE, LEARNING METHOD, AND PROGRAM
Has Consensual Aggregation on Random Projected High-dimensional Features for Regression
JP6988991B2 (en) Semantic estimation system, method and program

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20941012

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20941012

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: JP