WO2021044459A1 - 学習装置、予測システム、方法およびプログラム - Google Patents

学習装置、予測システム、方法およびプログラム Download PDF

Info

Publication number
WO2021044459A1
WO2021044459A1 PCT/JP2019/034345 JP2019034345W WO2021044459A1 WO 2021044459 A1 WO2021044459 A1 WO 2021044459A1 JP 2019034345 W JP2019034345 W JP 2019034345W WO 2021044459 A1 WO2021044459 A1 WO 2021044459A1
Authority
WO
WIPO (PCT)
Prior art keywords
worker
data
input
model
answer
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2019/034345
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
邦紘 竹岡
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
NEC Corp
Original Assignee
NEC Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by NEC Corp filed Critical NEC Corp
Priority to JP2021543615A priority Critical patent/JP7283548B2/ja
Priority to US17/638,984 priority patent/US20220269953A1/en
Priority to PCT/JP2019/034345 priority patent/WO2021044459A1/ja
Publication of WO2021044459A1 publication Critical patent/WO2021044459A1/ja
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00—Machine learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00—Machine learning
    • G06N20/10—Machine learning using kernel methods, e.g. support vector machines [SVM]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N5/00—Computing arrangements using knowledge-based models
    • G06N5/02—Knowledge representation; Symbolic representation
    • G06N5/022—Knowledge engineering; Knowledge acquisition

Definitions

  • the present invention is a learning device for learning a model for prediction, a learning method and a learning program, and a prediction system for making predictions using the model, using the answer results of workers obtained by cloud sourcing or the like. Regarding prediction methods and prediction programs.
  • Supervised learning represented by regression / classification is used for various analytical processes such as demand forecasting of products in retail stores and classification of objects in images.
  • supervised learning given an input-output pair, it learns the relationship between the input and the output, and when an input with an unknown output is given, it predicts an appropriate output based on the learned relationship. To do.
  • Non-Patent Document 1 and Non-Patent Document 2 have been proposed.
  • work is requested to an unspecified number of people by crowdsourcing or the like that can collect a large number of input / output pairs at low cost, and the classifier is learned by using the answer results. Is described.
  • Non-Patent Document 2 describes a technique for learning a classifier from the response results collected by crowdsourcing or the like.
  • the technique described in Non-Patent Document 2 is different from the technique described in Non-Patent Document 1 in that a classifier (called a worker model) corresponding to each worker is estimated and a prediction model is obtained from the worker model. To construct.
  • a classifier called a worker model
  • Non-Patent Document 2 can estimate the worker's answer in more detail by assuming a worker model corresponding to the worker, thus improving the prediction accuracy of the trained classifier. Can be done. Moreover, since a prediction model is prepared separately from the worker model, the estimation cost at the time of prediction can be reduced.
  • Non-Patent Document 1 and Non-Patent Document 2 it is not allowed that the answer result of the worker includes an answer other than the output candidate. If an answer other than the output candidate is included, processing such as removing it in advance is performed.
  • Non-Patent Document 1 and Non-Patent Document 2 when the number of responses from workers is small and there is a worker who gives an answer other than the label that is an output candidate, the worker model is estimated with high accuracy. This is difficult, and there is a problem that the prediction accuracy of the worker model is lowered.
  • answers other than labels that are output candidates are referred to as "unknown" answers.
  • Non-Patent Document 2 requires a set of input data and answers sufficient for learning the worker model, and if the number of answers is small, learning becomes difficult. Further, in the technique described in Non-Patent Document 2, since the answer not included in the output candidate label cannot be used, the worker model must be learned from a smaller number of input data and answer sets.
  • the present invention provides a learning device, a learning method and a learning program capable of learning a worker model for predicting a worker's answer with high accuracy, and a prediction system, a prediction method and a prediction program for making a prediction using the model.
  • the purpose is.
  • the learning device is a model that predicts an answer to new input data by using an input unit that accepts the input of answer data in which an answer is attached to the input data by each worker and the input answer data.
  • the first answer is that the input unit is provided with a learning unit that learns a certain worker model for each worker, and the input unit is assigned a label included in the output candidate label data indicating a candidate label assigned to the input data.
  • the learning department accepts the input of both the data and the second answer data in which the label not included in the output candidate label data is attached to the input data, and the learning department receives either the first answer data or the second answer data. It is characterized by learning a worker model using answer data.
  • the prediction system includes the above-mentioned learning device, a test data input unit that accepts input of test data, and a prediction unit that predicts the output of a worker with respect to the test data using a worker model learned by the learning device. It is characterized by that.
  • the learning method according to the present invention is a worker model that accepts input of answer data in which an answer is attached to input data by each worker and predicts an answer to new input data using the input answer data. Is learned for each worker, and when accepting the input of the answer data, the first answer data in which the label included in the output candidate label data indicating the candidate of the label given to the input data is given to the input data and its It accepts the input of both answer data of the second answer data with the label not included in the output candidate label data attached to the input data, and uses both the answer data of the first answer data and the second answer data to create a worker model. It is characterized by learning.
  • the prediction method according to the present invention is characterized in that a learning process based on the above learning method is performed, an input of test data is accepted, and a worker output with respect to the test data is predicted by using the worker model learned by the above learning process. To do.
  • the learning program receives an input process of answer data in which an answer is attached to the input data by each worker to a computer, and uses the input answer data to give an answer to new input data.
  • a learning process for learning a worker model which is a model to be predicted, is executed for each worker, and in the input process, a label included in the output candidate label data indicating the candidate of the label given to the input data is given to the input data.
  • the input of both the first answer data and the second answer data in which the label not included in the output candidate label data is attached to the input data is accepted, and in the learning process, the first answer data and the second answer data are accepted. It is characterized in that a worker model is trained using any of the answer data of the answer data.
  • the prediction program according to the present invention is a worker for test data using a test data input process that causes a computer to execute the above training program and further accepts input of test data, and a worker model learned by executing the training program. It is characterized in that a prediction process for predicting the output of is executed.
  • a worker model that predicts a worker's answer can be learned with high accuracy.
  • annotation that is, the work of labeling each data
  • crowdsourcing service each entity that performs annotation
  • each entity that performs annotation is described as a worker (worker)
  • a model that predicts the response of each worker to new input data is described as a worker model. That is, in the present embodiment, annotation is performed by each worker, and a worker model that predicts the answer of each annotated worker is learned.
  • annotation is not limited to the case where the annotation is performed by the crowdsourcing service.
  • Annotation may be performed by any person in charge.
  • FIG. 1 is a block diagram showing a configuration example of the learning device according to the first embodiment of the present invention.
  • the learning device 1 of the present embodiment includes a data input unit 2, a processing unit 3, a storage unit 4, and a result output unit 5.
  • the unidirectional arrow shown in FIG. 1 simply indicates the direction of information flow, and does not exclude bidirectionality.
  • the input data is unlabeled data, for example, data to be labeled (annotated) by a worker.
  • the output candidate label data is a label candidate given to the input data, and is predetermined according to the target to be given.
  • the output candidate label data may be referred to as label data.
  • the response data may be referred to as an annotation result.
  • Input data, output candidate label data, and response data each include multiple records.
  • the ID of each record of the input data is referred to as an input ID
  • the ID of the worker for distinguishing the worker in the response data is referred to as a worker ID.
  • the ID of each record of the output candidate label data is described as a label ID or a label.
  • an input ID and an input attribute corresponding to the input ID are associated with each other.
  • the input ID, the worker ID, and the corresponding answer are associated with each other.
  • the answer corresponding to the input ID and the worker ID is one of the label IDs or a label indicating an "unknown" answer. That is, the label indicating the answer of "unknown” is a label indicating the answer not included in the output candidate label data.
  • FIG. 2 is an explanatory diagram showing an example of input data.
  • the input data shown in FIG. 2 exemplifies "product name” and "price” as attributes corresponding to the product ID (input ID).
  • the attributes of the input data may be converted in advance into a numerical vector or the like that becomes a feature amount.
  • the input data illustrated in FIG. 2 can be said to be product data.
  • FIG. 3 is an explanatory diagram showing an example of output candidate label data.
  • a label ID and a corresponding attribute “name” are illustrated for each record of output candidate label data.
  • the labels of the non-luxury product and the luxury product are shown as output candidate label data, and the label IDs are "0" and "1", respectively.
  • FIG. 4 is an explanatory diagram showing an example of response data.
  • FIG. 4 shows an example of the product ID corresponding to the input ID of the input data and the answer data indicating the answer corresponding to the worker ID.
  • the worker specified by the worker ID responds to the product specified by the product ID (input ID).
  • the answer is indicated by the label ID or "?” Of the output candidate label data illustrated in FIG.
  • "?” Indicates an answer of "unknown".
  • Classification is one of supervised learning and is a task of predicting the relationship between an input and a finite number of output candidate labels. The classification assumes that the same label is output for data with similar properties.
  • Classifier learning is to estimate the parameters of a classifier by optimizing the parameters of the classifier for some index using a given set of inputs and outputs (learning data set). For example, a loss function is defined as an index for optimization, and the parameters of the classifier that minimizes the loss function are estimated. At the time of prediction, the classifier is operated based on the learned parameters for the newly given input, and the prediction result corresponding to the input is output.
  • the loss function is a function that outputs how close the prediction result of the classifier with the current parameter value is to the output of the learning data set.
  • the learning process when the input data, the output candidate label data, and the answer data as described above are given will be described.
  • the outline of the learning device 1 of the first embodiment will be described.
  • the learning device 1 inputs input data, output candidate label data, and answer data.
  • the response data includes the response data of one or more workers.
  • some records of the answer data include labels that are not included in the output candidate label data as the answer. As described above, here, the label not included in the output candidate label data is referred to as an "unknown" answer.
  • the input data exemplified in FIG. 2 the output candidate label data exemplified in FIG. 3, and the response data exemplified in FIG. 4 are referred to.
  • the input data may show attributes other than those illustrated in FIG. Further, the value of the attribute may be an image, sound, or the like. Further, although the product data is illustrated in FIG. 2, the input data is not limited to the product data.
  • the output candidate label data may show attributes other than those illustrated in FIG.
  • the number of records of the output candidate label data is not limited to 2, and may be 3 or more. That is, the classified classes may be multi-class.
  • the response data is data indicating the relationship between the input ID and the label ID, and is data indicating what kind of response the worker responded to the data of which input ID.
  • the response data used in the present invention it is assumed that some records of the response data include “unknown" answers.
  • FIG. 4 illustrates answer data including “?”, Which is an answer of “0”, “1”, and “unknown”.
  • the value of the attribute of the input data corresponding to the answer of "unknown" is distributed near the determination boundary of the worker model, so that the worker by the worker model is used. Improve the prediction accuracy of the answer.
  • the decision boundary of the worker model can be said to be the separation boundary that separates the input data group by the worker. This will be described in detail below with reference to the drawings.
  • FIG. 5 is an explanatory diagram schematically showing the true decision boundary assumed by the worker and the attributes of the input data in the vector space.
  • the asterisk illustrated in FIG. 5 indicates the input data that the worker answered "unknown”. Further, the circles or triangles illustrated in FIG. 5 indicate the input data answered in any of the answers indicated by the output candidate labels.
  • the worker model to be estimated that is, the true decision boundary assumed by the worker, is assumed to be distributed near the attributes of the input data corresponding to the data answered as "unknown".
  • the learning device 1 of the present embodiment uses this to learn a worker model that can improve the prediction accuracy.
  • the following steps S101 to S103 are performed.
  • the learning device 1 prepares a worker model corresponding to the worker ID of the answer data, and initializes the parameters of the worker model (step S101).
  • the learning device 1 is based on a loss function based on some or all of the answer data, currently defined parameters of the worker model, and a loss function that introduces a term that explicitly handles the worker's "unknown" answer.
  • the parameters of the worker model are updated so that the value of is small (step S102).
  • the learning device 1 repeats the process of step S102 until the condition of the end determination is satisfied, and outputs a worker model including the learned parameter when the condition is satisfied (step S103).
  • the data input unit 2 accepts the input of the answer data in which the answer is attached to the input data by each worker. Specifically, the data input unit 2 accepts the input of the data group used for learning the worker model and the set value of the worker model.
  • the worker model is a model that predicts the worker's response.
  • the setting values of the worker model include, for example, the attributes used as explanatory variables in the worker model and the type of the worker model. Examples of the types of worker models include support vector machines and logistic regression. One of various models is specified as the type of the worker model in the setting value of the worker model.
  • the data input unit 2 may access, for example, an external device (not shown) to acquire the data group and the set value of the worker model. Further, the data input unit 2 may be an input interface for inputting the data group and the set value of the worker model.
  • the data group used for learning the worker model includes input data (for example, product data exemplified in FIG. 2), predetermined output candidate label data (for example, output candidate label data exemplified in FIG. 3), and answers. Includes data (eg, response data illustrated in FIG. 4).
  • the answer data in some records, the answer value includes a value (“unknown” answer) that is not included in the output candidate label.
  • the data input unit 2 includes the response data in which the label included in the output candidate label data is attached to the input data (hereinafter, may be referred to as the first response data) and the label not included in the output candidate label data. Accepts the input of both answer data of the answer data given to the input data (hereinafter, may be referred to as the second answer data).
  • Non-Patent Document 1 does not allow a value other than the output candidate label to exist in the response value in the response data. Therefore, it can be said that the present embodiment is different from the technique described in Non-Patent Document 1 in that it includes an answer (that is, an answer of "unknown") that is not included in the output candidate label in some records.
  • the processing unit 3 performs processing for learning the worker model. Specifically, the processing unit 3 learns the worker model of the worker for the input worker's response data.
  • the processing unit 3 includes an initialization unit 31 and a worker model generation unit 32.
  • the initialization unit 31 receives the input data, the output candidate label data, the answer data, and the set value of the worker model from the data input unit 2, and stores them in the storage unit 4. In addition, the initialization unit 31 initializes various parameters used for learning the worker model. The initialization unit 31 may initialize various parameters according to the learning method of the worker model.
  • the worker model generation unit 32 learns the worker model by iterative processing. Hereinafter, the processing performed by each unit of the worker model generation unit 32 will be described.
  • the worker model generation unit 32 has a worker model update unit 321 and an end determination unit 322.
  • the worker model update unit 321 updates the parameters of the worker model based on the input data, the output candidate label data, the answer data, the parameters of the currently set worker model, and the specified loss function. At this time, the worker model update unit 321 may use a part or all of the answer data.
  • the answer data with a label (label of "unknown") not included in the output candidate label is used. That is, the worker model update unit 321 learns the worker model using both the answer data of the first answer data and the second answer data described above.
  • the loss function is included in the output candidate label, for example, a loss term calculated using a pair of answer data (ie, first answer data) for which one of the output candidate labels is the answer and the corresponding input data. Includes a loss term calculated using no "unknown" answer data (ie, second answer data).
  • the worker model update unit 321 may update the parameters of the worker model by using a known method.
  • the worker model update unit 321 updates the above parameters for the response data and input data corresponding to the worker IDs and the worker model.
  • the worker model update unit 321 may update the parameters of the worker model using, for example, Equation 1 shown below.
  • Equation 1 D j represents a set of sets of input data corresponding to the answers of the labels included in the output candidate label data among the answers by the worker j, and U j corresponds to the answer of "unknown".
  • g j is a worker model corresponding to the worker j, and its parameter is ⁇ j .
  • L is a loss function and is represented by, for example, Equation 2 shown below.
  • Equation 2 x i represents the i-th data, and y ij represents the worker j's answer to the i-th input data.
  • l (a, b) is a function for calculating the loss when predicted as b when the true output is a
  • L (g, D, U) is the model g in the response data D, U.
  • ⁇ ( ⁇ ) represents the loss function related to the answer of “unknown”
  • ⁇ and ⁇ are hyperparameters when updating the parameter and calculating the loss function.
  • the loss function may include a loss term that evaluates the closeness between the second answer data and the separation boundary that separates the input data group by the worker.
  • the worker model update unit 321 may learn the worker model based on the loss function to which the loss term for evaluating the output of the worker model with respect to the second answer data (input data included in the data) is added.
  • the end determination unit 322 determines whether or not to end the repetition of the parameter update process by the worker model update unit 321. The end determination unit 322 determines that the repetition of the above series of processes is completed when the end condition is satisfied, and determines that the repetition is continued if the end condition is not satisfied.
  • the number of repetitions of the above series of processes may be defined in the set value of the worker model.
  • the end determination unit 322 may determine that the repetition is completed when the number of repetitions of the above series of processes reaches a predetermined number of times.
  • the storage unit 4 is a storage device that stores various data acquired by the data input unit 2 and various data obtained by the processing of the processing unit 3.
  • the storage unit 4 may be the main storage device of the computer or the secondary storage device.
  • the worker model generation unit 32 can interrupt the process in the middle and store the data in the middle in the storage unit 4, and then restart the process.
  • the storage unit 4 may be divided into a main storage device and a secondary storage device.
  • the processing unit 3 may store a part of the data in the main storage device and store other data in the secondary storage device.
  • the storage unit 4 is realized by, for example, a magnetic disk or the like.
  • the result output unit 5 outputs the result of processing by the worker model generation unit 32. Specifically, the result output unit 5 outputs the worker model and the learned parameters stored in the storage unit 4 as a result of the processing.
  • the mode in which the result output unit 5 outputs the result is not particularly limited.
  • the result output unit 5 may output the result to another device (not shown), or may display the result on the display device, for example.
  • the worker model generation unit 32 having the worker model update unit 321 and the end determination unit 322, the data input unit 2, the initialization unit 31, and the result output unit 5 are, for example, a processor of a computer that operates according to a program (learning program). (For example, it is realized by CPU (Central Processing Unit), GPU (Graphics Processing Unit)).
  • the processor reads a program from a program recording medium such as a computer program storage device (not shown), and according to the program, the data input unit 2, the initialization unit 31, and the worker model generation unit 32 (more specifically). May operate as a worker model update unit 321 and an end determination unit 322), and a result output unit 5.
  • the function of the learning device 1 may be provided in the SaaS (Software as a Service) format.
  • the data input unit 2, the initialization unit 31, the worker model generation unit 32 (more specifically, the worker model update unit 321 and the end determination unit 322), and the result output unit 5 are each realized by dedicated hardware. You may be. Further, a part or all of each component of each device may be realized by a general-purpose or dedicated circuit (circuitry), a processor, or a combination thereof. These may be composed of a single chip or may be composed of a plurality of chips connected via a bus. A part or all of each component of each device may be realized by a combination of the above-mentioned circuit or the like and a program.
  • each component of the learning device 1 when a part or all of each component of the learning device 1 is realized by a plurality of information processing devices and circuits, the plurality of information processing devices and circuits may be centrally arranged or distributed. It may be arranged.
  • the information processing device, the circuit, and the like may be realized as a form in which each of the client-server system, the cloud computing system, and the like is connected via a communication network.
  • FIG. 6 is a flowchart showing an operation example of the learning device 1 of the first embodiment.
  • the data input unit 2 accepts the input of the data group (input data, output candidate label data and answer data) used for learning the worker model, and the set value of the worker model (step S1).
  • the initialization unit 31 stores the input data, the output candidate label data, the answer data, and the set value of the worker model in the storage unit 4. Further, the initialization unit 31 sets initial values for the parameters of the worker model, and stores the initial values in the storage unit 4 (step S2).
  • step S2 the initialization unit 31 may arbitrarily set an initial value or may set it by a random number.
  • the worker model generation unit 32 repeats the processes of steps S3 and S4 until the end condition is satisfied.
  • steps S3 and S4 will be described.
  • the worker model update unit 321 refers to the information stored in the storage unit 4 and learns a worker model that predicts an answer corresponding to the worker ID based on the input data and the answer data. Then, the worker model update unit 321 stores each worker model obtained by learning in the storage unit 4 (step S3).
  • step S4 determines whether or not the end condition is satisfied. If the end condition is not satisfied (No in step S4), the end determination unit 322 determines that step S3 is repeated. Then, the worker model generation unit 32 executes the processes of steps S3 and S4 again.
  • step S4 determines that the end condition is satisfied (Yes in step S4).
  • the end determination unit 322 determines that the repetition of step S3 is completed.
  • the result output unit 5 outputs the result of the processing by the worker model generation unit 32 at that time, and the processing by the learning device 1 ends.
  • the data input unit 2 accepts the input of the answer data in which the input data is given an answer by each worker, and the worker model generation unit 32 uses the input answer data. Then, learn the worker model for each worker. At that time, the data input unit 2 accepts the input of both the first answer data and the second answer data, and the worker model generation unit 32 receives both the first answer data and the second answer data. Use to learn a worker model. Therefore, the worker model that predicts the worker's answer can be learned with high accuracy.
  • the worker model update unit 321 refers to the input data, the output candidate label data, and the answer data to generate the worker model, the answer of "unknown” among the answer data And the corresponding input data are used for training the worker model. Therefore, it is possible to utilize that the input data corresponding to the answer of "unknown” is near the determination boundary of the worker model, and the prediction accuracy of the worker model can be further improved.
  • Embodiment 2 Next, a second embodiment of the present invention will be described.
  • the prediction system of the present embodiment generates a worker model by repeating the processes of steps S3 and S4, and describes the generated worker model and given new input data (hereinafter referred to as test data). Predict the worker's response to the test data using (and).
  • FIG. 7 is a block diagram showing a configuration example of the prediction system of the second embodiment according to the present invention.
  • the same components as those in the first embodiment are designated by the same reference numerals as those in FIG. 1, and the description thereof will be omitted.
  • the prediction system 1a of the second embodiment in addition to the data input unit 2, the processing unit 3, the storage unit 4, and the result output unit 5, the test data input unit 6 and the prediction unit 7 are further added. It includes a prediction result output unit 8.
  • processing unit 3 has completed the learning process described in the first embodiment and the worker model has been generated.
  • the test data input unit 6 accepts the input of test data.
  • the worker ID may be included in the input of the test data.
  • the prediction result output unit 8 described later may output the result of predicting the worker's answer corresponding to all the worker IDs in the answer data.
  • the test data input unit 6 may, for example, access an external device (not shown) to acquire test data. Further, the test data input unit 6 may be an input interface for inputting test data.
  • the test data includes the input ID and the value of each attribute as well as the input data.
  • the test data has the same format as the input data. For example, when the worker model is trained using the data illustrated in FIG. 2 as input data and all attributes as explanatory variables, the test data also requires the same attributes “product name” and “price” as the data illustrated in FIG. It is assumed that the values of each attribute of the test data are defined in the same manner as the input data.
  • the prediction unit 7 predicts the worker's response to the test data and the designated worker ID by using the worker model corresponding to the worker ID.
  • the prediction unit 7 uses the worker model corresponding to the specified worker ID to answer the worker corresponding to the worker ID. Predict.
  • the predicted worker's answer may be one of the labels in the output candidate label data.
  • the prediction unit 7 may output the probability for each of the output candidate labels (hereinafter, may be referred to as belonging probability) as the prediction of the worker's answer output based on the test data and the worker model. Good.
  • FIG. 8 is an explanatory diagram showing an example of a prediction result corresponding to one data included in the test data.
  • FIG. 8A shows an example of outputting the most suitable label among the labels that are output candidates.
  • FIG. 8B shows an example of a value indicating the affiliation probability of all the labels included in the output candidate label data, that is, how much the data matches each label.
  • FIG. 8C shows an example of the affiliation probability for each worker.
  • the prediction result output unit 8 outputs the value predicted by the prediction unit 7.
  • the mode in which the prediction result output unit 8 outputs the predicted value is not particularly limited.
  • the prediction result output unit 8 may output the predicted value to another device (not shown), or may display the predicted value on the display device, for example.
  • test data input unit 6, the prediction unit 7, and the prediction result output unit 8 are also realized by, for example, a computer processor that operates according to a program (prediction program).
  • FIG. 9 is a flowchart showing an operation example of the prediction system of the second embodiment.
  • the process up to the generation of the worker model is the same as the process from step S1 to step S4 illustrated in FIG.
  • the test data input unit 6 accepts the input of test data (step S5).
  • the prediction unit 7 predicts the output of the worker with respect to the test data using the trained worker model (step S6). Then, the prediction result output unit 8 outputs the value predicted by the prediction unit 7 (step S7).
  • the test data input unit 6 accepts the input of the test data
  • the prediction unit 7 predicts the output of the worker with respect to the test data using the learned worker model. Therefore, in addition to the effect of the first embodiment, the response to the worker's test data can be predicted. That is, it is possible to predict the answer to the test data of the worker corresponding to the worker ID for the given test data and the designated worker ID.
  • Embodiment 3 Next, a third embodiment of the present invention will be described.
  • a method of learning a worker model that predicts each worker's answer has been described.
  • a method of learning a model that predicts the answer of an arbitrary user hereinafter, simply referred to as a prediction model
  • the learning device of the present embodiment learns the worker model and the prediction model at the same time when the input data, the output candidate label data, and the answer data are given.
  • FIG. 10 is a block diagram showing a configuration example of the learning device according to the third embodiment of the present invention.
  • the learning device 11 of the third embodiment includes a data input unit 12, a processing unit 13, a storage unit 14, and a result output unit 15.
  • the unidirectional arrow shown in FIG. 10 simply indicates the direction of information flow, and does not exclude bidirectionality.
  • the learning device 11 holds a worker model corresponding to each worker ID and a learned prediction model.
  • Worker models and predictive models typically use the same type of classifier model, but do not necessarily have to be the same type of model.
  • the learning device 11 inputs input data, output candidate label data, and answer data including an answer of "unknown” in some records, and corresponds to each worker ID included in the answer data. Holds a worker model and a predictive model.
  • the “unknown” answer is used to generate the worker model, thereby improving the prediction accuracy of the worker model answer.
  • the learning device 11 calculates the importance of the worker by using the “unknown” answer tendency of the worker included in the answer data in addition to the information of the worker model. By updating the prediction model using the importance of this worker, the prediction accuracy of the prediction model is improved.
  • the learning device 11 prepares a worker model corresponding to the worker ID for each worker ID included in the answer data record and initializes the parameters thereof, as in the first embodiment.
  • the learning device 11 also initializes the parameters of the prediction model (step S210).
  • step S220 the following processes from step S221 to step S223 are performed.
  • the learning device 11 updates the parameters of the worker model with reference to the input data, the output candidate label data, and the answer data, as in the first embodiment. Further, the learning device 11 may use the information of the prediction model for updating the parameters of the worker model as in the method described in Non-Patent Document 2, for example (step S221).
  • the learning device 11 updates the importance of the worker based on the information of the worker model and the answer data.
  • the worker model information includes the parameters of the worker model and the like.
  • the response data includes the result of each worker answering which input. For example, if the worker of interest answers "unknown" even though other workers answer other than "unknown", the learning device 11 determines the importance of the worker of interest. May be updated to lower. Further, the learning device 11 may calculate the importance of the worker by using, for example, the distance between the result of estimating the worker's answer using the information of the worker model and the result of the majority vote of the answers of other workers. .. In this case, the learning device 11 is updated so that the shorter the distance, the higher the importance of the worker.
  • the learning device 11 may refer to the information of the prediction model for updating the importance of the worker.
  • the learning device 11 may calculate the importance of the worker by using, for example, the difference (distance) between the result estimated using the information of the prediction model and the result estimated using the information of the worker model. In this case, the learning device 11 is updated so that the shorter the distance, the higher the importance of the worker. (Step S222).
  • the learning device 11 updates the prediction model based on the input data, the answer data, the worker model, and the importance of the worker. For example, when the worker model and the prediction model are logistic regressions, the learning device 11 may update the parameters of the prediction model by the weighted sum of the worker models. Further, for example, the prediction model may be realized by the weighted sum of the worker models (step S223).
  • the learning device 11 repeats the process of step S220 until the condition of the end determination is satisfied, and outputs the learned worker model and the prediction model when the condition is satisfied (step S230).
  • the data input unit 12 accepts the input of the data group used for learning the worker model and the prediction model and the set values of the worker model and the prediction model.
  • the data input unit 12 may access an external device (not shown) to acquire the data group and the set values of the worker model and the prediction model.
  • the data input unit 12 may be an input interface for inputting the data group and the set values of the worker model and the prediction model.
  • the content of the data group is the same as that of the first embodiment. That is, the data group includes input data, output candidate label data, and response data.
  • the answer data in some records, the answer value includes a value (“unknown” answer) that is not included in the output candidate label.
  • the set values of the worker model and the prediction model include, for example, the attribute used as the explanatory variable in the worker model, the attribute used as the explanatory variable in the prediction model, the type of the worker model, and the type of the prediction model.
  • the prediction model is a model used to predict the output corresponding to the input data, and like the worker model in the first embodiment, the prediction model is one of various prediction models. Is specified.
  • the processing unit 13 performs processing for learning the worker model and the prediction model.
  • the processing unit 13 includes an initialization unit 131 and a model learning unit 132.
  • the initialization unit 131 receives the input data, the response data, and the set values of the worker model and the prediction model from the data input unit 12, and stores them in the storage unit 14. In addition, the initialization unit 131 initializes various parameters used for learning the worker model and the prediction model. The initialization unit 131 may initialize various parameters according to the learning method of the worker model and the prediction model.
  • the model learning unit 132 learns the worker model and the prediction model by iterative processing. Hereinafter, the processing performed by each unit of the model learning unit 132 will be described.
  • the model learning unit 132 includes a worker model generation unit 1321, a worker importance calculation unit 1322, a prediction model update unit 1323, and an end determination unit 1325.
  • the worker model generation unit 1321 learns a worker model that outputs the answer of the corresponding worker ID by inputting the attribute of the input data for each worker ID of the answer data that generates each worker model.
  • the method in which the worker model generation unit 1321 generates the worker model is the same as that in the first embodiment.
  • the worker model generation unit 1321 may use the prediction model for learning the worker model.
  • the worker importance calculation unit 1322 calculates the importance of the worker model for each worker ID included in the response data and the corresponding worker model.
  • each worker has a different specialty, so treating the worker model equally causes a decrease in the prediction accuracy of the prediction model.
  • the worker importance is calculated more accurately by using the “unknown” answer that is not included in the output candidate label data.
  • the worker importance calculation unit 1322 may calculate the worker importance so that the smaller the second response data is, the higher the worker importance is.
  • the worker importance calculation unit 1322 refers to the worker model information and the response data, and calculates the prediction accuracy of each worker using the worker model information. Specifically, the worker importance calculation unit 1322 calculates the worker importance for each worker according to the number of responses of the second response data by the worker. In addition, the worker importance calculation unit 1322 uses the first response data to determine the worker importance for each worker according to the ratio between the number of responses in the worker's first response data and the number of responses in the second response data. May be calculated. In this case, the worker importance calculation unit 1322 calculates the worker importance higher as the number of responses in the first response data increases. Similarly, the worker importance calculation unit 1322 may calculate the worker importance using the degree of agreement between the result estimated using the parameters of the worker model and the first response data of the worker.
  • the worker importance calculation unit 1322 calculates the worker importance higher as the degree of agreement is higher. Further, the worker importance calculation unit 1322 may estimate the prediction accuracy of the worker model by referring to the information of the worker model and the first response data, and use it as the worker importance. This makes it possible to estimate the reliability of the worker model itself. In addition, by referring to the response data, it is possible to calculate the importance of the worker using the number of answers of "unknown" described above. In addition, the information of the prediction model may be used to calculate the importance of the worker. When using the information of the prediction model, the worker importance calculation unit 1322 predicts the worker's response to the given data by using the information of the worker model, and measures the degree of agreement with the prediction result of the prediction model. You may use it to calculate the importance of the worker. In this case, the worker importance calculation unit 1322 calculates the worker importance higher as the degree of agreement is higher.
  • the worker importance calculation unit 1322 may calculate the importance of the worker j using, for example, Equation 3 shown below.
  • Equation 3 w j represents the importance of the worker j
  • P j represents the accuracy of the worker model with respect to the response of the output candidate label data in the response data.
  • U j is a set of sets of input data corresponding to the "unknown" answer of the worker j in the response data.
  • the importance of the worker is calculated based on the number of times the worker answers "unknown” and the accuracy of the response data.
  • the prediction model update unit 1323 refers to the learned worker model and the calculated worker importance, and updates the prediction model stored in the storage unit 4.
  • the predictive model is determined by the worker model and its importance.
  • the prediction model update unit 1323 may weight the worker model according to the corresponding worker importance and generate a prediction model using the weighted worker model. That is, the prediction model update unit 1323 may update the parameters of the prediction model with a weighted average that takes into account the importance of the corresponding worker for the parameters of the worker model, for example.
  • the model learning unit 132 repeats the processing by the worker model generation unit 1321, the processing by the worker importance calculation unit 1322, and the processing by the prediction model update unit 1323.
  • the end determination unit 1325 determines whether or not to end the repetition of the parameter update process by the model learning unit 132. The end determination unit 1325 determines that the repetition of the above series of processes is completed when the end condition is satisfied, and determines that the repetition is continued if the end condition is not satisfied.
  • the number of repetitions of the above series of processes may be defined in the set value of the prediction model.
  • the end determination unit 1325 may determine that the repetition is completed when the number of repetitions of the above series of processes reaches a predetermined number of times.
  • the end determination unit 1325 may make an end determination according to the amount of change in the parameter update.
  • the contents of the storage unit 14 and the result output unit 15 are the same as those of the storage unit 4 and the result output unit 5 in the first embodiment and the second embodiment.
  • the result output unit 15 of the present embodiment outputs a part or all of the worker model and the prediction model obtained as a result of the processing.
  • the result output unit 15 is realized by, for example, a computer processor that operates according to a program (learning program).
  • FIG. 11 is a flowchart showing an operation example of the learning device 11 of the third embodiment.
  • the data input unit 12 receives the input of the data group (input data and answer data) used for learning the worker model and the prediction model, and the set values of the worker model and the prediction model (step S11).
  • the initialization unit 131 stores the input data, the response data, and the set values of the worker model and the prediction model in the storage unit 14. Further, the initialization unit 131 sets initial values for the parameters of the worker model, the importance of the worker, and the parameters of the prediction model, and stores the initial values in the storage unit 14 (step S12).
  • the initialization unit 131 may arbitrarily set an initial value, or may determine a random number for each worker and use it as the initial value of the parameter. For example, the initialization unit 131 may divide the number of responses of each worker by the number of records of the response data and set the value as the initial value of the importance of the worker. Further, the initialization unit 131 may determine, for example, the initial value of the parameter of the prediction model by a random number.
  • step S12 the model learning unit 132 repeats the processes of steps S13 to S17 until the end condition is satisfied.
  • steps S13 to S17 will be described.
  • the worker model generation unit 1321 refers to the information stored in the storage unit 14, and learns a worker model that predicts the answer result of the worker for each worker based on the input data and the answer data. Then, the worker model generation unit 1321 stores each worker model obtained by learning in the storage unit 14 (step S13).
  • the worker importance calculation unit 1322 updates the importance of each worker stored in the storage unit 14 (step S14). Specifically, in step S14, the worker importance calculation unit 1322 reads the worker model information and the response data stored in the storage unit 14, and newly determines the importance of each worker based on them. If the importance of the worker model is not set, the worker importance calculation unit 1322 does not have to perform the process of step S14. Then, the worker importance calculation unit 1322 stores the calculated worker importance in the storage unit 14.
  • the prediction model update unit 1323 updates the prediction model by referring to the worker model of each worker ID and the worker importance of each worker ID. Specifically, the prediction model update unit 1323 updates the model information of the prediction model stored in the storage unit 14 with the updated model information of the prediction model (step S15).
  • step S16 determines whether or not the end condition is satisfied. If the end condition is not satisfied (No in step S16), the end determination unit 1325 determines that steps S13 to S16 are repeated. Then, the model learning unit 132 executes the processes of steps S13 to S16 again.
  • the end determination unit 1325 determines that the repetition of steps S13 to S16 is completed. In this case, the result output unit 15 outputs the result of the processing by the model learning unit 132 at that time, and the processing by the learning device ends.
  • the worker importance calculation unit 1322 calculates the worker importance for each worker according to the number of responses of the second response data by the worker, and the prediction model update unit 1323 is the worker model. Generate a predictive model based on the calculated worker importance. Therefore, in addition to the effect of the first embodiment, it is possible to learn a prediction model that does not depend on the annotated worker with high accuracy.
  • the worker model generation unit 1321 learns the worker model corresponding to each worker ID by referring to the input data and the answer data.
  • the record of the answer of "unknown” and the corresponding input data are used for learning the worker model.
  • a large amount of input data can be used as compared with the case of learning the worker model except for the answer of "unknown”.
  • the prediction accuracy of the worker model can be further improved.
  • the worker importance calculation unit 1322 adjusts the worker importance by referring to the response data and the information of the worker model. As a result, it is possible to give a higher degree of importance to an appropriate worker as compared with the case where the worker model is handled uniformly.
  • the prediction model update unit 1323 can further improve the prediction accuracy of the prediction model.
  • Embodiment 4 Next, a fourth embodiment of the present invention will be described.
  • the prediction system of the present embodiment generates a worker model by repeating the processes of steps S13 to S16, and also generates a prediction model. Then, when the test data is input, the prediction system predicts the output value corresponding to the test data.
  • the prediction system of the fourth embodiment of the present invention also outputs the predicted value of the answer corresponding to the test data by the worker of the designated worker ID when the test data and the worker ID are input.
  • FIG. 12 is a block diagram showing a configuration example of the prediction system according to the fourth embodiment of the present invention.
  • the same components as those in the third embodiment are designated by the same reference numerals as those in FIG. 10, and the description thereof will be omitted.
  • the prediction system 11a of the third embodiment includes a data input unit 12, a processing unit 13, a storage unit 14, a result output unit 15, and a test data input unit 16 and a prediction unit 17 for prediction. It includes a result output unit 18.
  • processing unit 13 has completed the learning process described in the third embodiment, and the worker model and the prediction model have been generated.
  • the test data input unit 16 accepts the input of test data.
  • the worker ID may be included in the input of the test data.
  • the prediction result output unit 18 described later may output the result of predicting the worker's answer corresponding to all the worker IDs in the answer data.
  • the test data input unit 16 may, for example, access an external device (not shown) to acquire test data. Further, the test data input unit 16 may be an input interface for inputting test data.
  • the content of the input test data is the same as that of the second embodiment.
  • the prediction unit 17 predicts the output value of the new input data included in the test data by using the prediction model or the worker model corresponding to the designated worker ID.
  • the prediction unit 17 may refer to the information of the learned worker model. For example, the prediction unit 17 may predict the output value by weighting and adding the classifiers for each worker and taking a majority vote.
  • the prediction unit 17 may output one of the output candidate labels or may output the belonging probability for each output candidate label, as in the prediction system of the second embodiment.
  • the prediction result output unit 18 outputs the value predicted by the prediction unit 17 as in the prediction system of the second embodiment.
  • the mode in which the prediction result output unit 18 outputs the predicted value is not particularly limited.
  • test data input unit 16 the prediction unit 17, and the prediction result output unit 18 are also realized by, for example, a computer processor that operates according to a program (prediction program).
  • FIG. 13 is a flowchart showing an operation example of the prediction system of the fourth embodiment.
  • the processing until the worker model and the prediction model are generated is the same as the processing from step S11 to step S16 illustrated in FIG.
  • the test data input unit 16 accepts the input of test data (step S17).
  • the prediction unit 17 predicts the output for the test data using the trained worker model or the prediction model (step S18). Then, the prediction result output unit 18 outputs the value predicted by the prediction unit 17 (step S19).
  • the test data input unit 16 receives the input of the test data, and the prediction unit 17 predicts the output for the test data using the worker model or the prediction model. Therefore, in addition to the effect of the third embodiment, the response to the test data can be predicted.
  • the output corresponding to the given test data can be predicted with high accuracy by using the prediction model. Further, according to the present embodiment, based on the given test data and the designated worker ID, the answer of the worker corresponding to the worker ID can be predicted with high accuracy by using the worker model.
  • FIG. 14 is a block diagram showing an outline of the learning device according to the present invention.
  • the learning device 80 (learning device 1, learning device 11) according to the present invention includes an input unit 81 (for example, a data input unit 2) that accepts input of answer data in which an answer is attached to the input data by each worker. It is provided with a learning unit 82 (for example, a processing unit 3) that learns a worker model, which is a model for predicting an answer to new input data, for each worker using the obtained answer data.
  • the input unit 81 includes first response data in which labels included in the output candidate label data indicating label candidates assigned to the input data are attached to the input data, and labels not included in the output candidate label data ( For example, "Unknown") is accepted for input of both answer data of the second answer data added to the input data, and the learning unit 82 uses both the first answer data and the second answer data as a worker. Learn the model.
  • the learning unit 82 may learn the worker model based on the loss function including the loss term for evaluating the output of the worker model for the second answer data.
  • the learning unit 82 learns the worker model of the worker based on the loss function including the loss term for evaluating the closeness between the second answer data and the separation boundary that separates the input data group by the worker. You may.
  • the learning device 80 (for example, the learning device 11) is a worker importance calculation unit that calculates the worker importance indicating the degree of reliability of the worker model for each worker according to the number of responses of the second answer data by the worker.
  • the worker importance calculation unit 1322 and a prediction model that predicts the output value corresponding to the input data from the output candidates indicated by the output candidate label data based on the worker model and the calculated worker importance. It may include a prediction model generation unit (for example, a prediction model update unit 1323) to be generated.
  • the worker importance calculation unit may calculate the worker importance so that the smaller the second response data, the higher the importance.
  • the prediction model generation unit may weight the worker model according to the corresponding worker importance, and generate the prediction model using the weighted worker model.
  • FIG. 15 is a block diagram showing an outline of the prediction system according to the present invention.
  • the prediction system 90 includes the above-mentioned learning device 80 (for example, learning device 1 and learning device 11) and a test data input unit 91 (for example, test data input unit 6 and test data input unit 11) that accepts test data input. 16) and a prediction unit 92 (for example, a prediction unit 7 and a prediction unit 17) that predict the output of a worker with respect to test data using a worker model learned by the learning device 80.
  • the prediction unit 92 may predict the output of the test data using the worker model of the worker.
  • the prediction unit 92 may predict the output for the test data by using the worker model or the prediction model learned by the learning device 80.
  • FIG. 16 is a schematic block diagram showing a configuration of a computer according to at least one embodiment.
  • the computer 2000 includes a processor 2001, a main storage device 2002, an auxiliary storage device 2003, and an interface 2004.
  • the above-mentioned learning device 80 is mounted on the computer 2000.
  • the operation of each processing unit described above is stored in the auxiliary storage device 2003 in the form of a program (learning program).
  • the processor 2001 reads a program from the auxiliary storage device 2003, deploys it to the main storage device 2002, and executes the above processing according to the program.
  • the auxiliary storage device 2003 is an example of a non-temporary tangible medium.
  • Other examples of non-temporary tangible media include magnetic disks, magneto-optical disks, CD-ROMs (Compact Disc Read-only memory), DVD-ROMs (Read-only memory), which are connected via interface 2004. Examples include semiconductor memory.
  • the distributed computer 2000 may expand the program to the main storage device 2002 and execute the above processing.
  • the program may be for realizing a part of the above-mentioned functions. Further, the program may be a so-called difference file (difference program) that realizes the above-mentioned function in combination with another program already stored in the auxiliary storage device 2003.
  • difference file difference program
  • a worker model that predicts the answer to new input data by using the input unit that accepts the input of the answer data to which the answer is attached to the input data by each worker and the input answer data.
  • the input unit is provided with a learning unit that learns each worker, and the input unit includes the first answer data in which the label included in the output candidate label data indicating the candidate of the label assigned to the input data is attached to the input data.
  • the input of both the answer data of the second answer data in which the label not included in the output candidate label data is attached to the input data is accepted, and the learning unit receives either the first answer data or the second answer data.
  • a learning device characterized in that the worker model is learned by using the answer data of.
  • Appendix 2 The learning device according to Appendix 1 that learns a worker model based on a loss function including a loss term that evaluates the output of the worker model with respect to the second answer data.
  • the learning unit learns the worker model of the worker based on the loss function including the loss term that evaluates the closeness between the second answer data and the separation boundary that separates the input data group by the worker. 2.
  • the learning device according to 2.
  • Appendix 4 A worker importance calculation unit that calculates the worker importance indicating the degree of reliability of the worker model for each worker according to the number of responses of the second response data by the worker, and the worker calculated as the worker model. Any of Appendix 1 to Appendix 3 provided with a prediction model generator that generates a prediction model that predicts the output value corresponding to the input data from the output candidates indicated by the output candidate label data based on the importance.
  • the learning device according to one.
  • the worker importance calculation unit is a learning device according to Appendix 4, which calculates the worker importance so that the smaller the second answer data is, the higher the worker importance is.
  • the prediction model generation unit weights the worker model according to the corresponding worker importance, and generates a prediction model using the weighted worker model. Any one of Appendix 1 to Appendix 5.
  • the prediction unit is the prediction system according to Appendix 7, which predicts the output of test data using the worker model of the worker when the test data input unit receives the input of information for identifying the worker.
  • Appendix 9 Using the learning device according to any one of Appendix 4 to Appendix 6, the test data input unit that accepts the input of test data, and the worker model or prediction model learned by the learning device. , A prediction system including a prediction unit that predicts the output of the test data.
  • Each worker receives the input of the answer data in which the answer is attached to the input data by each worker, and uses the input answer data to predict the answer to the new input data.
  • the first answer data in which the label included in the output candidate label data indicating the candidate of the label given to the input data is given to the input data and the output candidate are given.
  • the worker model accepts the input of both answer data of the second answer data in which the label not included in the label data is attached to the input data, and uses both the answer data of the first answer data and the second answer data.
  • a learning method characterized by learning.
  • Appendix 11 The learning method according to Appendix 10 for learning a worker model based on a loss function including a loss term for evaluating the output of the worker model for the second answer data.
  • Appendix 12 Performs learning processing based on the learning method described in Appendix 10 or Appendix 11, accepts input of test data, and predicts the output of the worker with respect to the test data using the worker model learned in the learning process.
  • a prediction method characterized by doing.
  • Appendix 13 The prediction method according to Appendix 12, wherein when the test data input unit receives an input of information for identifying a worker, the output of test data is predicted using the worker model of the worker.
  • Appendix 14 An input process that accepts the input of answer data in which an answer is given to the input data by each worker, and a model that predicts an answer to new input data using the input answer data.
  • a learning process for learning a worker model is executed for each worker, and in the input process, a label included in the output candidate label data indicating a candidate for a label given to the input data is given to the input data.
  • the input of the answer data of both the one answer data and the second answer data in which the label not included in the output candidate label data is attached to the input data is accepted, and in the learning process, the first answer data and the first answer data and the first answer data are accepted.
  • Appendix 15 The learning program according to Appendix 14, which causes a computer to learn a worker model based on a loss function including a loss term for evaluating the output of the worker model with respect to the second answer data in a learning process.
  • Appendix 16 Using a test data input process that causes a computer to execute the learning program described in Appendix 14 or 15 and further accepts input of test data, and a worker model learned by executing the learning program. , A prediction program for executing a prediction process for predicting the output of a worker with respect to the test data.
  • Appendix 17 The prediction program according to Appendix 16 for predicting the output of test data using a worker model of the worker when the test data input unit receives input of information for identifying a worker in the prediction process.
  • the present invention is suitably applied to a learning device that learns a model for prediction by using the answer results of a worker obtained by crowdsourcing or the like, and a prediction system that makes a prediction using the learned model. ..
  • the present invention can also be applied to a prediction model learning device for predicting labels of data such as images based on answers collected by a crowdsourcing system or the like, and a prediction system based on the learned prediction model.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Software Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Evolutionary Computation (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • Computing Systems (AREA)
  • General Physics & Mathematics (AREA)
  • Mathematical Physics (AREA)
  • Artificial Intelligence (AREA)
  • Medical Informatics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Computational Linguistics (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
PCT/JP2019/034345 2019-09-02 2019-09-02 学習装置、予測システム、方法およびプログラム Ceased WO2021044459A1 (ja)

Priority Applications (3)

Application Number Priority Date Filing Date Title
JP2021543615A JP7283548B2 (ja) 2019-09-02 2019-09-02 学習装置、予測システム、方法およびプログラム
US17/638,984 US20220269953A1 (en) 2019-09-02 2019-09-02 Learning device, prediction system, method, and program
PCT/JP2019/034345 WO2021044459A1 (ja) 2019-09-02 2019-09-02 学習装置、予測システム、方法およびプログラム

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2019/034345 WO2021044459A1 (ja) 2019-09-02 2019-09-02 学習装置、予測システム、方法およびプログラム

Publications (1)

Publication Number Publication Date
WO2021044459A1 true WO2021044459A1 (ja) 2021-03-11

Family

ID=74852588

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2019/034345 Ceased WO2021044459A1 (ja) 2019-09-02 2019-09-02 学習装置、予測システム、方法およびプログラム

Country Status (3)

Country Link
US (1) US20220269953A1 (https=)
JP (1) JP7283548B2 (https=)
WO (1) WO2021044459A1 (https=)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11620157B2 (en) * 2019-10-18 2023-04-04 Splunk Inc. Data ingestion pipeline anomaly detection

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2014136316A1 (ja) * 2013-03-04 2014-09-12 日本電気株式会社 情報処理装置、情報処理方法、及びプログラム
JP2015230570A (ja) * 2014-06-04 2015-12-21 日本電信電話株式会社 学習モデル作成装置、判定システムおよび学習モデル作成方法
WO2017073373A1 (ja) * 2015-10-30 2017-05-04 株式会社モルフォ 学習システム、学習装置、学習方法、学習プログラム、教師データ作成装置、教師データ作成方法、教師データ作成プログラム、端末装置及び閾値変更装置

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2014136316A1 (ja) * 2013-03-04 2014-09-12 日本電気株式会社 情報処理装置、情報処理方法、及びプログラム
JP2015230570A (ja) * 2014-06-04 2015-12-21 日本電信電話株式会社 学習モデル作成装置、判定システムおよび学習モデル作成方法
WO2017073373A1 (ja) * 2015-10-30 2017-05-04 株式会社モルフォ 学習システム、学習装置、学習方法、学習プログラム、教師データ作成装置、教師データ作成方法、教師データ作成プログラム、端末装置及び閾値変更装置

Also Published As

Publication number Publication date
US20220269953A1 (en) 2022-08-25
JP7283548B2 (ja) 2023-05-30
JPWO2021044459A1 (https=) 2021-03-11

Similar Documents

Publication Publication Date Title
EP3467723B1 (en) Machine learning based network model construction method and apparatus
TWI631518B (zh) 具有一或多個計算裝置的電腦伺服系統及訓練事件分類器模型的電腦實作方法
US20220253725A1 (en) Machine learning model for entity resolution
US20220027738A1 (en) Distributed synchronous training architecture using stale weights
JP2019109634A (ja) 学習プログラム、予測プログラム、学習方法、予測方法、学習装置および予測装置
US20200265307A1 (en) Apparatus and method with multi-task neural network
CN113537630A (zh) 业务预测模型的训练方法及装置
CN112801231A (zh) 用于业务对象分类的决策模型训练方法和装置
CN111159241B (zh) 一种点击转化预估方法及装置
WO2014199920A1 (ja) 予測関数作成装置、予測関数作成方法、及びコンピュータ読み取り可能な記録媒体
CN108062448A (zh) 预测边坡稳定性的建模及分析方法、设备和存储介质
US12333467B2 (en) Method for controlling a process for handling a conflict and related electronic device
US20220207420A1 (en) Utilizing machine learning models to characterize a relationship between a user and an entity
CN111160959A (zh) 一种用户点击转化预估方法及装置
Yahaya et al. An enhanced bank customers churn prediction model using a hybrid genetic algorithm and k-means filter and artificial neural network
JP7207540B2 (ja) 学習支援装置、学習支援方法、及びプログラム
CN113822390B (zh) 用户画像构建方法、装置、电子设备和存储介质
WO2019205544A1 (zh) 兼顾公平性的上下文感知学习的结果预测分类器
CN114529191A (zh) 用于风险识别的方法和装置
Al Ali et al. Enhancing financial distress prediction through integrated Chinese whisper clustering and federated learning
CN116975686B (zh) 训练学生模型的方法、行为预测方法和装置
US20220405640A1 (en) Learning apparatus, classification apparatus, learning method, classification method and program
CN112561569B (zh) 基于双模型的到店预测方法、系统、电子设备及存储介质
US20230385664A1 (en) A computer-implemented method for deriving a data processing and inference pipeline
US20210319269A1 (en) Apparatus for determining a classifier for identifying objects in an image, an apparatus for identifying objects in an image and corresponding methods

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19944055

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2021543615

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19944055

Country of ref document: EP

Kind code of ref document: A1