WO2020045341A1 - Dispositif de sélection, dispositif d'apprentissage, procédé de sélection, procédé d'apprentissage et programme - Google Patents

Dispositif de sélection, dispositif d'apprentissage, procédé de sélection, procédé d'apprentissage et programme Download PDF

Info

Publication number
WO2020045341A1
WO2020045341A1 PCT/JP2019/033290 JP2019033290W WO2020045341A1 WO 2020045341 A1 WO2020045341 A1 WO 2020045341A1 JP 2019033290 W JP2019033290 W JP 2019033290W WO 2020045341 A1 WO2020045341 A1 WO 2020045341A1
Authority
WO
WIPO (PCT)
Prior art keywords
evaluator
learning
value
evaluation
data
Prior art date
Application number
PCT/JP2019/033290
Other languages
English (en)
Japanese (ja)
Inventor
歩相名 神山
厚志 安藤
亮 増村
哲 小橋川
Original Assignee
日本電信電話株式会社
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 日本電信電話株式会社 filed Critical 日本電信電話株式会社
Publication of WO2020045341A1 publication Critical patent/WO2020045341A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00—Machine learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00—Administration; Management
    • G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
    • G06Q50/10—Services
    • G06Q50/20—Education

Definitions

  • the present invention relates to a technique for selecting a specific evaluator from a plurality of evaluators who have evaluated a plurality of evaluation targets.
  • Non-patent document 1 In tests in which conversational skills are considered as one item of the skill test, such as favorable sensitivity of telephone voice (Non-patent document 1) and good pronunciation and fluency of foreign languages (Non-patent document 2), the quantitative Impressive impression values (for example, five-grade evaluation from good to bad, 5 to low 1 with high preference, 1 to 5 low with naturalness, etc.).
  • a machine learning model may be learned from impression value data and features of the data.
  • impressions have different criteria for different people, the impression values may be different for different persons even with the same data.
  • an average impression can be learned by giving an impression value to one data by a large number of people and using the average value of the impression values.
  • the impression value should be given by as many people as possible. For example, in the impression data created in Non-Patent Document 3, impression values are assigned to one voice data by ten people.
  • the impression value is at most about one or two persons per data.
  • the standard of impression may be different depending on the evaluator, and may be significantly different from the value obtained by the average value of a large number of people.
  • an averaged label may not be able to be correctly learned due to a difference in evaluation criteria.
  • the above-described problem is not limited to the case where the evaluation target is a voice, and may be a problem that can occur when the evaluation target is evaluated by a plurality of evaluators in various fields.
  • the present invention has been made in view of the above points, and provides a technology that enables an evaluator to perform an average evaluation to be selected even when the number of evaluators per evaluation target is small. Aim.
  • a selection device that selects a specific evaluator from a plurality of evaluators who have performed evaluations on a plurality of evaluation targets
  • ⁇ a coincidence counting unit that calculates the degree of coincidence of the evaluation value with another evaluator for each evaluator based on the input data including the evaluator and the evaluation value for each evaluation target;
  • a probability value calculation unit that calculates a probability value indicating whether the evaluation value significantly matches other evaluators
  • an evaluator selection unit that selects an evaluator whose evaluation value is significantly consistent with another evaluator based on the probability value.
  • a technology that enables an evaluator to perform an average evaluation even when the number of evaluators per evaluation target is small.
  • FIG. 1 shows a functional configuration diagram of the learning device 100 according to the present embodiment.
  • the learning device 100 includes a number-of-matches counting unit 110, a probability value calculation unit 120, an evaluator selection unit 130, and a learning unit 140.
  • a learning label data DB 150 and a learning feature data DB 160 which are databases for storing data used in the learning device 100.
  • the learning label data DB 150 and the learning feature amount data DB 160 may be provided as a storage unit inside the learning device 100, or may be connected to the outside of the learning device 100 via a network.
  • the number of matches counted by the number-of-matches counting unit 110 is an example of the degree of match. More generally, the number-of-matches counting section 110 may be referred to as a matching degree counting section 110.
  • a device including the number-of-matches totaling unit 110, the probability value calculating unit 120, the evaluator selecting unit 130, and the learning unit 140 may be referred to as a selecting device.
  • the learning apparatus 100 further includes a matching number counting section 110, a probability value calculating section 120, an evaluator selecting section 130, and a learning section 140.
  • the matching number counting section 110, the probability value calculating section 120, and the evaluator selecting section 130 are provided.
  • the part may be referred to as a selection device.
  • the learning device 100 and the selection device described above can be realized by, for example, causing a computer to execute a program describing processing contents described in the present embodiment.
  • the device can be realized by executing a program corresponding to a process performed by the device using hardware resources such as a CPU and a memory built in the computer.
  • the above-mentioned program can be recorded on a computer-readable recording medium (a portable memory or the like) and can be stored or distributed. Further, the above program can be provided through a network such as the Internet or electronic mail.
  • FIG. 2 is a diagram illustrating an example of a hardware configuration of the computer according to the present embodiment.
  • the computer in FIG. 2 includes a drive device 170, an auxiliary storage device 172, a memory device 173, a CPU 174, an interface device 175, a display device 176, an input device 177, and the like, which are interconnected by a bus B.
  • the program for realizing the processing in the computer is provided by a recording medium 171 such as a CD-ROM or a memory card.
  • a recording medium 171 such as a CD-ROM or a memory card.
  • the program is installed from the recording medium 171 to the auxiliary storage device 172 via the drive device 170.
  • the program need not always be installed from the recording medium 171 and may be downloaded from another computer via a network.
  • the auxiliary storage device 172 stores installed programs and also stores necessary files and data.
  • the memory device 173 reads out the program from the auxiliary storage device 172 and stores it when there is an instruction to start the program.
  • the CPU 174 implements functions of the memory device 173 according to a program stored in the device.
  • the interface device 175 is used as an interface for connecting to a network.
  • the display device 176 displays a GUI (Graphical User Interface) or the like by a program.
  • the input device 177 includes a keyboard, a mouse, buttons, a touch panel, and the like, and is used to input various operation instructions.
  • the learning label data DB150 the learning label data stored in the learning feature data DB160, the learning feature data, and the operation of each unit will be described in detail.
  • FIG. 3 shows an example of the learning label data in this embodiment
  • FIG. 4 shows an example of the learning feature amount data.
  • the data number y (i, 0) is y (i, 0) ⁇ 0,1,2, ..., J ⁇ , and is a number indicating the data number j of the learning feature amount data.
  • the evaluator number is the number y (i, 1) ⁇ 1, 2, 3,..., K ⁇ of the evaluator who evaluated the data.
  • the impression value label is an impression value (evaluation value) for the data. In the present embodiment, it is assumed that binary values are assigned as 0 and 1.
  • a plurality of impression labels are given to one learning feature amount data by a plurality of persons.
  • the learning feature amount data is, for example, an audio signal or a value such as a vector obtained by extracting a feature from the audio signal.
  • the learning label data is input from the learning label data DB 150 to the number-of-matches counting section 110.
  • the number-of-matches counting section 110 Based on the input learning label data, the number-of-matches counting section 110 counts the number of matches and the number of mismatches for each evaluator, and calculates the number-of-matches total data C (k, m) and the evaluation. The evaluation number N (k) of the person k is obtained. Note that the coincidence count data C (k, m) may be referred to as coincidence.
  • the learning label data is converted into a function based on the following rules 1 and 2.
  • the probability value calculation unit 120 obtains a probability value for calculating whether or not the number of the evaluators that match the other evaluators is significantly high. The details are as follows.
  • the probability value calculation unit 120 performs a binomial test according to the following equation to obtain a significance probability P (k), assuming that the number of C (k, 0) and C (k, 1) follows a binomial distribution.
  • M Normcdf in the above equation is a standard normal cumulative distribution function.
  • the evaluator selection unit 130 selects an evaluator whose evaluation value is significantly consistent with another using the threshold ⁇ based on P (k) obtained by the probability value calculation unit 120, and Select only the data evaluated by.
  • the evaluator selecting unit 130 calculates and outputs a set I ′ of label data of the selected evaluator according to the following equation.
  • an evaluator k that satisfies “P (k) ⁇ C (k, 1)> C (k, 0)” may be selected, and the selected evaluator may be output.
  • the learning unit 140 may extract the label data evaluated by the selected evaluator.
  • the learning unit 140 learns a machine learning model using the label data selected by the evaluator selection unit 130 and the corresponding learning feature data as teacher data.
  • a general method for example, SVM, neural network, or the like.
  • the label data only the label data set I ′ selected by the evaluator selection unit 130 is used. Specifically, sets X and Y of the learning feature amount data and the label data are obtained as described below.
  • the learning unit 140 performs learning using X and Y as teacher data. For example, the model is learned using X as the input of the model and Y as the correct data. The model obtained by learning is used, for example, for automatic evaluation of speech.
  • the label has two values (0 or 1).
  • a stable evaluator can be obtained even if the value is more than two steps, for example, five values, ten values, or the like. Can be selected.
  • a description will be given of processing contents that can be applied even when values of more than two levels are used as labels.
  • the learning label data at this time is such that the number of labels (the number of rows) per one data number (y (i, 0)) is the number of stages (not limited to two). It is.
  • the number-of-matches totaling unit 110 in the modified example totals differences between evaluators.
  • C (k, m) indicates the number of times that evaluator k agrees with another at any stage
  • m ⁇ 0, C (k, m) indicates that evaluator k is another
  • the views indicate the number of times there was an m-step difference.
  • the operation consists of the following (1) and (2).
  • the learning label data is converted into a function based on the following rules 1 and 2.
  • the probability value calculation unit 120 in the modification will be described.
  • the label is binary
  • the probability value is calculated based on the binomial distribution. Calculated.
  • P (k) is obtained on the assumption that C (k, s) follows a polynomial distribution.
  • the probability difference of the evaluation of both a m specifically When p m and calculates the square value ⁇ as follows seek P (k).
  • C Chi_p in the above equation is a function for calculating the P value of the ⁇ square value.
  • p m When the evaluation of the M phase, when the probability that a score m is given as q v, for example p m can be set as follows.
  • the above formula means that the difference between two evaluators is calculated as shown in the table of FIG. 5 and the value obtained by summing the probability values that the difference between the scores is the same is obtained.
  • an impression value is assumed, but the present invention can be extended to a case where evaluation of multiple items is performed.
  • a P value is obtained for each of the multiple items in the same manner as in the modification, and an evaluator is selected for each item.
  • an evaluator having a low P value for all items may be selected.
  • an evaluator having a low P value for all items.
  • one of two patterns shown in the following (1) and (2) is used. Either or both can be used.
  • the P value of item A for a certain evaluator is PA
  • the P value of item B is PB
  • the value of item C is
  • the P value of the evaluator is calculated as (PA + PB + PC) / 3.
  • the number of evaluation steps is set so that the P value when matching in seven levels of evaluation is evaluated higher than when matching in two levels of evaluation.
  • Weights can be assigned according to the size so that the stability evaluation when the evaluations match is higher.
  • the weighting according to the magnitude of the number of evaluation steps is, for example, in the case of the number of evaluation steps M set for an arbitrary evaluation item, a value of 1 / M is used as a weight, and the product of each P value and the weight is used.
  • the average obtained by adding is used for evaluator selection.
  • the number of stages of items A, B, and C is MA, MB, and MC, respectively. If the P value is PA, the P value of item B is PB, and the P value of item C is PC, the P value of the evaluator is ((PA / MA) + (PB / MB) + (PC / MC)) / 3.
  • an evaluator capable of giving an average impression value label is selected based on the probability distribution based on the number of evaluations, the number of matches, and the like. (1) Even when the number of evaluators per data is small (for example, two), non-average evaluators can be excluded, and evaluators performing average evaluation can be selected. Further, an average impression value can be learned.
  • a selection device for selecting a specific evaluator from a plurality of evaluators who have performed evaluations on a plurality of evaluation targets, and an evaluator and an evaluation value for each evaluation target.
  • a coincidence counting unit that calculates the degree of coincidence of the evaluation value with another evaluator for each evaluator, and for each evaluator, based on the degree of coincidence for each evaluator,
  • a probability value calculation unit that calculates a probability value indicating whether the value significantly matches another evaluator, and an evaluation that selects an evaluator whose evaluation value significantly matches the other evaluator based on the probability value
  • a selection device comprising: a selection unit.
  • the evaluation value is, for example, a value set for each evaluation target, and is a multi-step value of two or more values for one or a plurality of items, or a continuous value. Further, the probability value calculation unit may calculate the probability value assuming that the degree of coincidence follows a predetermined probability distribution.
  • an evaluation target and an evaluation value evaluated by the evaluator selected by the selection device are input as teacher data, and learning of a machine learning model is performed using the teacher data.
  • a learning device comprising a learning unit is provided.
  • REFERENCE SIGNS LIST 100 learning device 110 number of matches totaling unit 120 probability value calculating unit 130 evaluator selecting unit 140 learning unit 150 learning label data DB 160 learning feature data DB 170 Drive device 171 Recording medium 172 Auxiliary storage device 173 Memory device 174 CPU 175 Interface device 176 Display device 177 Input device

Landscapes

  • Engineering & Computer Science (AREA)
  • Business, Economics & Management (AREA)
  • Theoretical Computer Science (AREA)
  • Strategic Management (AREA)
  • General Physics & Mathematics (AREA)
  • Human Resources & Organizations (AREA)
  • Economics (AREA)
  • Tourism & Hospitality (AREA)
  • Physics & Mathematics (AREA)
  • Software Systems (AREA)
  • General Business, Economics & Management (AREA)
  • Marketing (AREA)
  • Entrepreneurship & Innovation (AREA)
  • Educational Administration (AREA)
  • Quality & Reliability (AREA)
  • Data Mining & Analysis (AREA)
  • Development Economics (AREA)
  • Operations Research (AREA)
  • Health & Medical Sciences (AREA)
  • Game Theory and Decision Science (AREA)
  • Primary Health Care (AREA)
  • Artificial Intelligence (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Health & Medical Sciences (AREA)
  • Evolutionary Computation (AREA)
  • Medical Informatics (AREA)
  • Educational Technology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Mathematical Physics (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)
  • Complex Calculations (AREA)

Abstract

Cette invention concerne un dispositif de sélection qui sélectionne un évaluateur spécifique parmi une pluralité d'évaluateurs qui ont évalué une pluralité de sujets à évaluer, ledit dispositif comprenant : une partie d'agrégation de niveau de correspondance pour, sur la base de données d'entrée comprenant des évaluateurs et des valeurs d'évaluation pour chaque sujet pour une évaluation, calculer le niveau de correspondance dans les valeurs d'évaluation entre l'un des évaluateurs et les autres évaluateurs ; une partie de calcul de valeur de probabilité pour calculer une valeur de probabilité pour chacun des évaluateurs sur la base du niveau de correspondance relatif à l'évaluateur, la valeur de probabilité indiquant si le niveau de correspondance entre la valeur d'évaluation pour l'évaluateur et les valeurs d'évaluation pour les autres évaluateurs est significatif ; et une partie de sélection d'évaluateur pour, sur la base des valeurs de probabilité, sélectionner l'évaluateur pour lequel le niveau de correspondance dans les valeurs d'évaluation avec les autres évaluateurs est significatif.
PCT/JP2019/033290 2018-08-27 2019-08-26 Dispositif de sélection, dispositif d'apprentissage, procédé de sélection, procédé d'apprentissage et programme WO2020045341A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2018-158652 2018-08-27
JP2018158652A JP2020035018A (ja) 2018-08-27 2018-08-27 選定装置、学習装置、選定方法、学習方法、及びプログラム

Publications (1)

Publication Number Publication Date
WO2020045341A1 true WO2020045341A1 (fr) 2020-03-05

Family

ID=69643025

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2019/033290 WO2020045341A1 (fr) 2018-08-27 2019-08-26 Dispositif de sélection, dispositif d'apprentissage, procédé de sélection, procédé d'apprentissage et programme

Country Status (2)

Country Link
JP (1) JP2020035018A (fr)
WO (1) WO2020045341A1 (fr)

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2004227208A (ja) * 2003-01-22 2004-08-12 Matsushita Electric Ind Co Ltd ユーザ適応型行動決定装置および行動決定方法
JP2018063536A (ja) * 2016-10-12 2018-04-19 株式会社野村総合研究所 家計簿管理支援システム

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2004227208A (ja) * 2003-01-22 2004-08-12 Matsushita Electric Ind Co Ltd ユーザ適応型行動決定装置および行動決定方法
JP2018063536A (ja) * 2016-10-12 2018-04-19 株式会社野村総合研究所 家計簿管理支援システム

Non-Patent Citations (4)

* Cited by examiner, † Cited by third party
Title
ANDO, ATSUSHI: "Speech emotion classification based on soft target learning using ambiguous emotion utterances", LECTURE PROCEEDINGS OF 2018 SPRING RESEARCH CONFERENCE OF THE ACOUSTICAL SOCIETY OF JAPAN CD- ROM, 27 February 2018 (2018-02-27), pages 41 - 42 *
NEYATANI, TAKUROU: "Optimization of KANSEI Retrieval Agent Using the Neural Network", PROCEEDINGS OF THE 2012 GENERAL CONFERENCE OF IEICE: INFORMATION AND SYSTEM 1, 6 March 2012 (2012-03-06), pages 20 *
SUZUKI, YOSUKE: "Stacked Denoising Autoencoder-based Deep Collaborative Filtering using the change of similarity", DOCUMENTS OF SPECIAL INTEREST GROUP ON KNOWLEDGE-BASED SYSTEMS, 1 November 2016 (2016-11-01), pages 7 - 12, XP033099453 *
TAKAKI, SHINJI: "Unsupervised speaker adaptation based on speaker similarity for DNN- based speech synthesis", IPSJ SIG TECHNICAL REPORTS, SPOKEN LANGUAGE PROCESSING (SLP) 2017-SLP- 118, 6 October 2017 (2017-10-06), pages 1 - 6, XP033525977 *

Also Published As

Publication number Publication date
JP2020035018A (ja) 2020-03-05

Similar Documents

Publication Publication Date Title
Wang et al. Super-naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks
Rodrigues et al. Sequence labeling with multiple annotators
US9141622B1 (en) Feature weight training techniques
Wang et al. Cost-effective quality assurance in crowd labeling
Bentivogli et al. SICK through the SemEval glasses. Lesson learned from the evaluation of compositional distributional semantic models on full sentences through semantic relatedness and textual entailment
US20180225581A1 (en) Prediction system, method, and program
JP6977901B2 (ja) 学習材推薦方法、学習材推薦装置および学習材推薦プログラム
WO2020082734A1 (fr) Procédé et appareil de reconnaissance d'émotions dans un texte, dispositif électronique, et support d'enregistrement non volatil lisible par ordinateur
US20190026655A1 (en) Machine Learning System for Patient Similarity
US20190012573A1 (en) Co-clustering system, method and program
US11669740B2 (en) Graph-based labeling rule augmentation for weakly supervised training of machine-learning-based named entity recognition
US20200250574A1 (en) Techniques for bimodal learning in a financial context
Kong et al. Eliciting expertise without verification
US20130004931A1 (en) Computer-Implemented Systems and Methods for Determining Content Analysis Metrics for Constructed Responses
US9348810B2 (en) Model learning method
CN113011689B (zh) 软件开发工作量的评估方法、装置及计算设备
US20100296728A1 (en) Discrimination Apparatus, Method of Discrimination, and Computer Program
WO2020045341A1 (fr) Dispositif de sélection, dispositif d'apprentissage, procédé de sélection, procédé d'apprentissage et programme
Scharl et al. Leveraging the wisdom of the crowds for the acquisition of multilingual language resources
Javid et al. Developing Novel T-Swish Activation Function in Deep Learning
JP6306951B2 (ja) ソーシャルネットワークサービスにおけるノイズ投稿の分類方法およびシステム
Liu et al. Transfer learning robustness in multi-class categorization by fine-tuning pre-trained contextualized language models
Bujnowski et al. An approach to intuitionistic fuzzy decision trees
CN115048525B (zh) 基于多元组的文本分类、文本分类模型训练的方法及装置
Huang et al. ChatGPT Rates Natural Language Explanation Quality Like Humans: But on Which Scales?

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19855792

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19855792

Country of ref document: EP

Kind code of ref document: A1