WO2020143301A1 - 一种训练样本有效性检测方法、计算机设备及计算机非易失性存储介质 - Google Patents

一种训练样本有效性检测方法、计算机设备及计算机非易失性存储介质 Download PDF

Info

Publication number
WO2020143301A1
WO2020143301A1 PCT/CN2019/117090 CN2019117090W WO2020143301A1 WO 2020143301 A1 WO2020143301 A1 WO 2020143301A1 CN 2019117090 W CN2019117090 W CN 2019117090W WO 2020143301 A1 WO2020143301 A1 WO 2020143301A1
Authority
WO
WIPO (PCT)
Prior art keywords
question
preset
extended
questions
score
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2019/117090
Other languages
English (en)
French (fr)
Inventor
许开河
杜尧鑫
王少军
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Ping An Technology Shenzhen Co Ltd
Original Assignee
Ping An Technology Shenzhen Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Ping An Technology Shenzhen Co Ltd filed Critical Ping An Technology Shenzhen Co Ltd
Publication of WO2020143301A1 publication Critical patent/WO2020143301A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
    • G06F16/33—Querying
    • G06F16/332—Query formulation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
    • G06F16/35—Clustering; Classification

Definitions

  • the present application relates to the field of artificial intelligence technology, and in particular, to a method for detecting the validity of training samples, computer equipment, and computer non-volatile storage media.
  • the embodiments of the present application provide a training sample validity detection method, a computer device, and a computer non-volatile storage medium to solve the problem of low efficiency of training sample validity detection in the prior art.
  • a method for detecting the validity of training samples includes:
  • Step S101 Acquire multiple extended questions, where each extended question is associated with a corresponding preset standard question
  • Step S102 Randomly divide the plurality of extended questions into a sample set of a preset number of copies, and divide the sample set of the preset number of shares into a training set and a cross-validation set according to a preset ratio;
  • Step S103 using the training set to train a classification model
  • Step S104 adopting the cross-validation method and using the classification model to mark multiple extended questions in the cross-validation set until all the extended question labels are completed;
  • Step S105 Obtain all the labeling results of the extended question output by the classification model
  • Step S106 an abnormal expansion question is obtained according to the labeling result, and the labeling result of the abnormal expansion question is different from the associated preset standard question.
  • a training sample validity detection device includes:
  • a first acquiring unit is used to acquire multiple extended questions, wherein each of the extended questions is associated with a corresponding preset standard question; a production unit is used to randomly divide the multiple extended questions into preset copies The sample set of the preset number is divided into a training set and a cross-validation set according to a preset ratio; a training unit is used to train the classification model using the training set; a labeling unit is used to adopt cross-validation Method, and use the classification model to label multiple extension questions in the cross-validation set until all the extension question labels are completed; a second acquisition unit, configured to acquire all the extensions output by the classification model The marked result of the question; a first processing unit, configured to obtain an abnormal expansion question based on the marked result, and the marked result of the abnormal expanded question is different from the associated preset standard question.
  • a computer non-volatile storage medium includes a stored program, and when the program is running, the device on which the storage medium is located is controlled to perform the above-mentioned training The steps of the sample validity test method.
  • a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing When the computer program is described, the steps of the above-mentioned training sample validity detection method are realized.
  • the cross-validation method and classification model are used to mark the extended question, so as to match the preset standard question associated with the extended question according to the labeling result, the abnormal extended question is obtained, and the abnormal extended question is quickly screened out, which improves the training.
  • the efficiency of the sample is tested to avoid some low-quality extended questions to train the intelligent response model.
  • FIG. 1 is a flowchart of a training sample validity detection method according to an embodiment of the present application
  • FIG. 2 is a schematic diagram of a training sample validity detection device according to an embodiment of the present application.
  • FIG. 3 is a schematic diagram of a computer device according to an embodiment of the present application.
  • first, second, third, etc. may be used to describe the terminals in the embodiments of the present application, these terminals should not be limited to these terms. These terms are only used to distinguish the terminals from each other.
  • first acquisition unit may also be called a second second acquisition unit, and similarly, the second acquisition unit may also be called a first acquisition unit.
  • the word “if” as used herein may be interpreted as “when” or “when” or “in response to determination” or “in response to detection”.
  • the phrases “if determined” or “if detected (statement or event stated)” may be interpreted as “when determined” or “in response to determination” or “when detected (statement or event stated) )” or “in response to detection (statement or event stated)”.
  • FIG. 1 is a flowchart of a training sample validity detection method according to an embodiment of the present application. As shown in FIG. 1, the method includes:
  • Step S101 Acquire multiple extended questions, where each extended question is associated with a corresponding preset standard question.
  • Step S102 Randomly divide the multiple extended questions into a sample set with a preset number of copies, and divide the sample set with a preset number of copies into a training set and a cross-validation set according to a preset ratio.
  • step S103 a classification model is trained using the training set.
  • Step S104 adopting the cross-validation method and using the classification model to mark multiple extended questions in the cross-validation set until all the extended questions are marked.
  • Step S105 Obtain the labeling results of all extended questions output by the classification model.
  • step S106 an abnormal expansion question is obtained according to the labeling result.
  • the labeling result of the abnormal expansion question is different from the associated preset standard question.
  • the cross-validation method and classification model are used to mark the extended question, so as to match the preset standard question associated with the extended question according to the labeling result, the abnormal extended question is obtained, and the abnormal extended question is quickly screened out, which improves the training.
  • the efficiency of the sample is tested to avoid some low-quality extended questions to train the intelligent response model.
  • the preset standard question is “What should I do if the credit card is overdue?”
  • its extended question 1 is "My credit card is overdue, what should I do with it”
  • the extended question 2 is "Can the credit card be overdue?”
  • the preset standard questions associated with Extended Question 1, Extended Question 2, and Extended Question 3 are all "What about credit card overdue?”
  • the training set is used to train the classification sample set of the classification model, so that the trained classification model can identify standard questions that the user may desire to consult based on questions input by the user.
  • the extended question in the training set is the data to be classified, and its category is its associated preset question.
  • the classification model includes an input layer, an output layer, and a fusion layer.
  • the input layer includes word feature input layer and question feature input layer.
  • the preset standard question associated with each extended question is used as a question feature.
  • the question feature input layer is used to obtain the characteristics of the preset standard question associated with the input extended question
  • the word feature input layer is used to obtain the vocabulary obtained after the extended question segmentation process Characteristics.
  • the classification model can classify a new extended question according to the learned vocabulary characteristics and question features, so that the new extended question can be classified to an accurate standard question as much as possible.
  • the cross-validation method is any one of a 10-fold cross-validation method or a 5-fold cross-validation method.
  • a five-fold cross-validation method is adopted, specifically, multiple expansion questions are randomly divided into 10 parts, and two of them are taken as the cross-validation set each time, and the remaining 8 parts are taken as the training set.
  • the extended question in the cross-validation set is marked according to the classification results of the classification model, for example, extended question 4 "My credit card is overdue, what should I do” is classified to the preset criteria and asked "What to do if the credit card is overdue”
  • extended question 4 "My credit card is overdue, what should I do" is classified to the preset criteria and asked "What to do if the credit card is overdue”
  • the correlation standard question of extended question 4 is the classification result of the classification model.
  • the multiple extended questions associated with each preset standard question are randomly divided into 10 copies, and one copy is taken from each of the 10 extended questions associated with each preset standard question as cross-validation Set, the remaining 9 copies are used as training sets.
  • there are 10 preset standard questions, namely A1, A2...A10 each preset standard question is associated with multiple extended questions, so the multiple extended questions associated with each preset standard question are randomly divided into 10, It can ensure that each preset standard can be effectively trained during cross-validation.
  • step S105 the method further includes:
  • the labeling results of all the expanded questions of the M times are counted.
  • the lower the score of the expanded question the higher the probability that the associated preset criterion question is wrong.
  • the extended question has a score of 1; when the labeling result is different from the preset standard question, the extended question has a score of -1, and the labeling is repeated 10 times, of which 6 times the score is 1 , The score of 4 times is -1, then the score of the extended question is the sum average of 10 times of the labeling results, which is 0.2 points.
  • the first preset threshold is 0.1 points, it means that all the extended questions whose scores are less than 0.1 points are abnormal extended questions.
  • the classification model may use a variety of machine learning algorithms, which may be Naive Bayes algorithm, nearest neighbor rule, linear discriminant analysis, support vector machine or decision tree algorithm.
  • machine learning algorithms used by the classification model may be the same or different. Using different machine learning algorithms can reduce the accidental error of labeling errors, mark the cross-validation set multiple times through the classification model, and compare the labeling results with the preset standard questions, which can effectively enhance the recognition accuracy of abnormal expansion questions. .
  • the method further includes:
  • the current score of the standard question consisting of the labeling result and the preset standard question is scored as the first preset score; or, when the labeling result is different from the preset standard question , The current score of the standard question is divided into the second preset score; the second cumulative score of each standard question is calculated; the standard question of the second cumulative score less than the second preset threshold is selected; the selected standard Ask the right question as a confusing standard, and output.
  • step S105 the method further includes:
  • the methods of calculating semantic similarity include cosine similarity calculation method, BM25 algorithm and so on.
  • the method for calculating the semantic similarity includes: acquiring multiple extended questions associated with the same preset standard question, performing word segmentation processing on each extended question and extracting keywords, wherein an open source tool for word segmentation may be used, for example ICTCLAS (Institute of Computing Technology, Chinese Lexical Analysis System), SCWS (Simple Chinese WordsSegmentation), etc., can also directly use the self-developed word segmentation interface to segment each sub-segment text;
  • the word vector representation model obtains a vector representation of each keyword, where the word vector representation model may be, for example, a neural network model such as Word2vec.
  • the cosine similarity formula is used to calculate the similarity of any two extended questions, and multiple extended questions whose similarity is greater than a preset threshold are identified as multiple extended questions that are substantially the same, such as extended questions that only differ in punctuation.
  • the method further includes: modifying the preset criterion question associated with the abnormal expansion question to the correct preset criterion question.
  • the method further includes: correcting the abnormal expansion question, and replacing the original expansion question with the modified abnormal expansion question; training the classification model with the replaced multiple expansion questions.
  • An embodiment of the present application provides a training sample validity detection device.
  • the training sample validity detection device is used to perform the above training sample validity detection method. As shown in FIG. 2, the device includes: a first acquisition unit 10, Unit 20, training unit 30, labeling unit 40, second acquisition unit 50, first processing unit 60.
  • the first acquiring unit 10 is configured to acquire multiple extended questions, where each extended question is associated with a corresponding preset standard question;
  • the production unit 20 is used to randomly divide multiple extended questions into a sample set with a preset number of copies, and divide the sample set with a preset number into a training set and a cross-validation set according to a preset ratio;
  • the training unit 30 is used to train a classification model using the training set
  • the labeling unit 40 is used to adopt the cross-validation method and use the classification model to label multiple extended questions in the cross-validation set until all the extended questions are marked;
  • the second obtaining unit 50 is used to obtain all the labeling results of the extended questions output by the classification model
  • the first processing unit 60 is configured to obtain an abnormal expansion question according to the labeling result.
  • the labeling result of the abnormal expansion question is different from the associated preset standard question.
  • the cross-validation method and classification model are used to mark the extended question, so as to match the preset standard question associated with the extended question according to the labeling result, the abnormal extended question is obtained, and the abnormal extended question is quickly screened out, which improves the training.
  • the efficiency of the sample is tested to avoid some low-quality extended questions to train the intelligent response model.
  • the preset standard question is “What should I do if the credit card is overdue?”
  • its extended question 1 is "My credit card is overdue, what should I do with it”
  • the extended question 2 is "Can the credit card be overdue?”
  • the preset standard questions associated with Extended Question 1, Extended Question 2, and Extended Question 3 are all "What about credit card overdue?”
  • the training set is used to train the classification sample set of the classification model, so that the trained classification model can identify the standard questions that the user may desire to consult based on the questions input by the user.
  • the extended question in the training set is the data to be classified, and its category is its associated preset question.
  • the classification model includes an input layer, an output layer, and a fusion layer.
  • the input layer includes word feature input layer and question feature input layer.
  • the preset standard question associated with each extended question is used as a question feature.
  • the question feature input layer is used to obtain the characteristics of the preset standard question associated with the input extended question
  • the word feature input layer is used to obtain the vocabulary obtained after the extended question segmentation process Characteristics.
  • the classification model can classify a new extended question according to the learned vocabulary characteristics and question features, so that the new extended question can be classified to an accurate standard question as much as possible.
  • the cross-validation method is any one of a 10-fold cross-validation method or a 5-fold cross-validation method.
  • a five-fold cross-validation method is adopted, specifically, multiple expansion questions are randomly divided into 10 parts, and two of them are taken as the cross-validation set each time, and the remaining 8 parts are taken as the training set.
  • the classification model will expand the question 4 "My credit card is overdue, what should I do” to the preset criteria and ask "How is the credit card overdue?" Under the "management", then the correlation standard question of extended question 4 is the classification result of the classification model.
  • the multiple extended questions associated with each preset standard question are randomly divided into 10 copies, and one copy is taken from each of the 10 extended questions associated with each preset standard question as cross-validation Set, the remaining 9 copies are used as training sets.
  • there are 10 preset standard questions, namely A1, A2...A10 each preset standard question is associated with multiple extended questions, so the multiple extended questions associated with each preset standard question are randomly divided into 10, It can ensure that each preset standard can be effectively trained during cross-validation.
  • the device further includes a repeating unit, a second obtaining unit, a scoring unit, a calculating unit, a screening unit, and an output unit.
  • a repeating unit is used to control the production unit 20, the training unit 30 and the labeling unit 40 to repeat the operation M times, where M is a natural number greater than 1, and the classification model is initialized before each repeated operation;
  • Second acquisition unit also used to obtain the M-time labeling results of all the extended questions output by the classification model;
  • the scoring unit used to score the current question of the extended question when the labeled result of the extended question is the same as the preset standard question Is the first preset score; or, when the labeling result of the extended question is different from the preset standard question, the current question of the extended question is divided into the second preset score;
  • the calculation unit is used to calculate the first question of each extended question A cumulative score; a screening unit for screening out extended questions whose first cumulative score is less than a first preset threshold; an output unit for outputting the filtered expanded questions as abnormal expanded questions.
  • the labeling results of all the expanded questions of the M times are counted.
  • the lower the score of the expanded question the higher the probability that the associated preset criterion question is wrong.
  • the extended question has a score of 1; when the labeling result is different from the preset standard question, the extended question has a score of -1, and the labeling is repeated 10 times, of which 6 times the score is 1 , The score of 4 times is -1, then the score of the extended question is the sum average of 10 times of the labeling results, which is 0.2 points.
  • the first preset threshold is 0.1 points, it means that all the extended questions whose scores are less than 0.1 points are abnormal extended questions.
  • the classification model may use a variety of machine learning algorithms, which may be Naive Bayes algorithm, nearest neighbor rule, linear discriminant analysis, support vector machine or decision tree algorithm.
  • machine learning algorithm used by the classification model may be the same or different.
  • the use of different machine learning algorithms can reduce the accidental error of labeling errors.
  • the classification model can be used to label the cross-validation set multiple times and compare the labeling results with the preset standard questions, which can effectively enhance the recognition accuracy of abnormal expansion questions.
  • the scoring unit is also used to score the first preset score for the current standard question composed of the marked result and the preset standard question when the tag result is the same as the preset standard question; or , When the labeling result is different from the preset standard question, the current score of the standard question is the second preset score; the calculation unit is also used to calculate the second cumulative score of each standard question; the screening unit is also used It is used to filter out standard question pairs whose second cumulative score is less than the second preset threshold; the output unit is also used to output the selected standard question pairs as confusing standard question pairs.
  • the device further includes a third acquisition unit, a statistical unit, a second processing unit, and a confirmation unit.
  • the third obtaining unit is used to obtain the labeling results of multiple substantially identical extended questions, and the semantic similarity of the substantially identical multiple extended questions is greater than a preset value; the statistical unit is used to count the labels of multiple substantially identical extended questions Results; the second processing unit is used to take the labeling result with the most repetitions as the correct preset standard question corresponding to substantially the same multiple extended questions; the confirmation unit is used to extend the labeled result differently from the correct preset standard question Confirmed as abnormal expansion.
  • the methods of calculating semantic similarity include cosine similarity calculation method, BM25 algorithm and so on.
  • the method for calculating the semantic similarity includes: acquiring multiple extended questions associated with the same preset standard question, performing word segmentation processing on each extended question and extracting keywords, wherein an open source tool for word segmentation may be used, for example ICTCLAS, SCWS, etc., can also use the self-developed word segmentation interface to segment each sub-segment text; get the vector representation of each keyword through the word vector representation model, where the word vector representation model can be, for example, a neural network model such as WORD2VEC .
  • the cosine similarity formula is used to calculate the similarity of any two extended questions, and multiple extended questions whose similarity is greater than a preset threshold are identified as multiple extended questions that are substantially the same, such as extended questions that only differ in punctuation.
  • the device further includes a modification unit for modifying the preset standard question associated with the abnormal expansion question to the correct preset standard question.
  • the device further includes a correction unit and a replacement unit;
  • the correction unit is used to correct the abnormal expansion question and replace the original expanded question with the corrected abnormal expansion question;
  • the replacement unit is used to train the classification model with the replaced multiple expansion questions.
  • An embodiment of the present application provides a computer non-volatile storage medium, where the storage medium includes a stored program, wherein, when the program is running, the device where the storage medium is located is controlled to perform the following steps:
  • Step S101 Acquire multiple extended questions, where each extended question is associated with a corresponding preset standard question; Step S102, randomly divide the multiple extended questions into a sample set of preset number of copies, and divide The sample set of the preset number of copies is divided into a training set and a cross-validation set; step S103, the training set is used to train the classification model; step S104, the cross-validation method is used, and the classification model is used to mark multiple extended questions in the cross-validation set until All the expansion questions are marked; step S105, the labeling results of all the expansion questions output by the classification model are obtained; step S106, the abnormal expansion questions are obtained according to the labeling results, and the abnormal expansion questions are different from the associated preset standard questions.
  • Steps S102 to S104 are repeated M times, where M is a natural number greater than 1, and the classification model is initialized before each repeated operation; Get the M-time labeling results of all extended questions output by the classification model; when the labeled results of the extended question are the same as the preset standard question, the current question of the extended question is scored as the first preset score; or, when the extended question When the labeling result is different from the preset standard question, the current question of the extended question is divided into the second preset score; the first cumulative score of each extended question is calculated; the extensions whose first cumulative score is less than the first preset threshold are selected Question: The filtered expanded question is regarded as an abnormal expanded question and output.
  • the device where the storage medium is located performs M times of labeling results of all extended questions that obtain the output of the classification model, and further includes: when the labeling result is the same as the preset standard question, the labeling result and The current score of the standard question pair composed of the preset standard question is divided into the first preset score; or, when the marked result is different from the preset standard question, the current score of the standard question is divided into the second preset score Calculate the second cumulative score of each standard question pair; filter out the standard question pairs whose second cumulative score is less than the second preset threshold; use the filtered standard question pairs as confusing standard question pairs and output them.
  • controlling the device where the storage medium is located while the program is running further includes:
  • the method further includes:
  • FIG. 3 is a schematic diagram of a computer device provided by an embodiment of the present application.
  • the computer device 100 of this embodiment includes: a processor 101, a memory 102, and a computer program 103 stored in the memory 102 and executable on the processor 101.
  • the computer program 103 is executed by the processor 101
  • the method for detecting the validity of the training samples in the embodiment is implemented. To avoid repetition, details are not described here one by one.
  • the computer program is executed by the processor 101, the functions of each model/unit in the apparatus for detecting the validity of the training samples in the embodiment are implemented. To avoid repetition, they are not described here one by one.
  • the computer device 100 may be a computing device such as a desktop computer, a notebook, a palmtop computer, and a cloud server.
  • the computer device may include, but is not limited to, the processor 101 and the memory 102.
  • FIG. 3 is only an example of the computer device 100, and does not constitute a limitation on the computer device 100, and may include more or less components than shown, or combine certain components, or different components.
  • computer equipment may also include input and output devices, network access devices, buses, and so on.
  • the so-called processor 101 may be a central processing unit (Central Processing Unit, CPU), or other general-purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), Field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.
  • the general-purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
  • the memory 102 may be an internal storage unit of the computer device 100, such as a hard disk or a memory of the computer device 100.
  • the memory 102 may also be an external storage device of the computer device 100, for example, a plug-in hard disk equipped on the computer device 100, a smart memory card (Smart) Card (SMC), a secure digital (SD) card, and a flash memory card (Flash Card) etc.
  • the memory 102 may also include both an internal storage unit of the computer device 100 and an external storage device.
  • the memory 102 is used to store computer programs and other programs and data required by computer devices.
  • the memory 102 may also be used to temporarily store data that has been output or will be output.
  • terminals involved in the embodiments of the present application may include, but are not limited to, personal computers (Personal Computers, PCs), personal digital assistants (Personal Digital Assistants, PDAs), wireless handheld devices, tablet computers (Tablet Computers), Mobile phones, MP3 players, MP4 players, etc.
  • the application may be an application (native) installed on the terminal, or may also be a web application (web App) of the browser on the terminal, which is not limited in this embodiment of the application.
  • the disclosed system, device, and method may be implemented in other ways.
  • the device embodiments described above are only schematic.
  • the division of the unit is only a logical function division, and there may be other divisions in actual implementation, for example, multiple units or components may be combined Or it can be integrated into another system, or some features can be ignored or not implemented.
  • the displayed or discussed mutual coupling or direct coupling or communication connection may be indirect coupling or communication connection through some interfaces, devices or units, and may be in electrical, mechanical, or other forms.
  • the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
  • the above integrated unit implemented in the form of a software functional unit may be stored in a computer-readable storage medium.
  • the above software functional unit is stored in a storage medium, and includes several instructions to enable a computer device (which may be a personal computer, server, or network device, etc.) or processor (Processor) to perform the methods described in the embodiments of the present application Partial steps.
  • the foregoing storage media include: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disk and other media that can store program code .

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Mathematical Physics (AREA)
  • Computational Linguistics (AREA)
  • Electrically Operated Instructional Devices (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

一种训练样本有效性检测方法、计算机设备及计算机非易失性存储介质，涉及人工智能技术领域，方法包括：获取多个扩展问，其中，每个扩展问与对应的一个预设标准问关联(S101)；将多个扩展问随机分为预设份数的样本集，并根据预设比例将预设份数的样本集分为训练集和交叉验证集(S102)；利用训练集训练分类模型(S103)；采用交叉验证方法，并利用分类模型对交叉验证集中的多个扩展问标注，直至所有的扩展问标注完成(S104)；获取分类模型输出的所有的扩展问的标注结果(S105)；根据标注结果得到异常扩展问，异常扩展问的标注结果与所关联的预设标准问不同(S106)。能够解决现有技术中训练样本有效性检测效率低的问题。

Description

一种训练样本有效性检测方法、计算机设备及计算机非易失性存储介质
本申请要求于2019年01月10日提交中国专利局、申请号为201910022265.4、申请名称为“一种训练样本有效性检测方法及装置”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
【技术领域】
本申请涉及人工智能技术领域，尤其涉及一种训练样本有效性检测方法、计算机设备及计算机非易失性存储介质。
【背景技术】
目前，客服机器人发展趋于成熟，根据所需要实现的业务场景，需要大量的训练语料对基于机器学习智能应答模型进行训练，然而，现实中训练语料往往质量参差不齐，不好的语料会导致应答模型的回答效果产生负面影响，因此如何提高训练样本的有效性检测效率成为目前亟待解决的问题。
【申请内容】
有鉴于此，本申请实施例提供了一种训练样本有效性检测方法、计算机设备及计算机非易失性存储介质，用以解决现有技术中训练样本的有效性检测效率低的问题。
为了实现上述目的，根据本申请的一个方面，提供了一种训练样本有效性检测方法，所述方法包括：
步骤S101，获取多个扩展问，其中，每个所述扩展问与对应的一个预设标准问关联；
步骤S102，将多个所述扩展问随机分为预设份数的样本集，并根据预设比例将所述预设份数的样本集分为训练集和交叉验证集；
步骤S103，利用所述训练集训练分类模型；
步骤S104，采用交叉验证方法，并利用所述分类模型对所述交叉验证集中的多个扩展问标注，直至所有的所述扩展问标注完成；
步骤S105，获取所述分类模型输出的所有的所述扩展问的标注结果；
步骤S106，根据所述标注结果得到异常扩展问，所述异常扩展问的标注结果与所关联的预设标准问不同。
为了实现上述目的，根据本申请的一个方面，提供了一种训练样本有效性检测装置，所述装置包括：
第一获取单元，用于获取多个扩展问，其中，每个所述扩展问与对应的一个预设标准问关联；制作单元，用于将多个所述扩展问随机分为预设份数的样本集，并根据预设比例将所述预设份数的样本集分为训练集和交叉验证集；训练单元，用于利用所述训练集训练分类模型；标注单元，用于采用交叉验证方法，并利用所述分类模型对所述交叉验证集中的多个扩展问标注，直至所有的所述扩展问标注完成；第二获取单元，用于获取所述分类模型输出的所有的所述扩展问的标注结果；第一处理单元，用于根据所述标注结果得到异常扩展问，所述异常扩展问的标注结果与所关联的预设标准问不同。
为了实现上述目的，根据本申请的一个方面，提供了一种计算机非易失性存储介质，所述存储介质包括存储的程序，在所述程序运行时控制所述存储介质所在设备执行上述的训练样本有效性检测方法的步骤。
为了实现上述目的，根据本申请的一个方面，提供了一种计算机设备，包括存储器、处理器以及存储在所述存储器中并可在所述处理器上运行的计算机程序，所述处理器执行所述计算机程序时实现上述的训练样本有效性检测方法的步骤。
在本方案中，利用交叉验证方法和分类模型来对扩展问进行标注，从而根据标注结果与扩展问关联的预设标准问进行匹配，得到异常扩展问，快速筛选出异常扩展问，提高了训练样本的有效性检测效率，从而避免一些质量低的扩展问来训练智能应答模型。
【附图说明】
为了更清楚地说明本申请实施例的技术方案，下面将对实施例中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本申请的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动性的前提下，还可以根据这些附图获得其它的附图。
图1是根据本申请实施例的一种训练样本有效性检测方法的流程图；
图2是根据本申请实施例的一种训练样本有效性检测装置的示意图；
图3是根据本申请实施例的一种计算机设备的示意图。
【具体实施方式】
为了更好的理解本申请的技术方案，下面结合附图对本申请实施例进行详细描述。
应当明确，所描述的实施例仅仅是本申请一部分实施例，而不是全部的实施例。基于本申请中的实施例，本领域普通技术人员在没有作出创造性劳动前提下所获得的所有其它实施例，都属于本申请保护的范围。
在本申请实施例中使用的术语是仅仅出于描述特定实施例的目的，而非旨在限制本申请。在本申请实施例和所附权利要求书中所使用的单数形式的“一种”、“所述”和“该”也旨在包括多数形式，除非上下文清楚地表示其他含义。
应当理解，本文中使用的术语“和/或”仅仅是一种描述关联对象的关联关系，表示可以存在三种关系，例如，A和/或B，可以表示：单独存在A，同时存在A和B，单独存在B这三种情况。另外，本文中字符“/”，一般表示前后关联对象是一种“或”的关系。
应当理解，尽管在本申请实施例中可能采用术语第一、第二、第三等来描述终端，但这些终端不应限于这些术语。这些术语仅用来将终端 彼此区分开。例如，在不脱离本申请实施例范围的情况下，第一获取单元也可以被称为第二二获取单元，类似地，第二获取单元也可以被称为第一获取单元。
取决于语境，如在此所使用的词语“如果”可以被解释成为“在……时”或“当……时”或“响应于确定”或“响应于检测”。类似地，取决于语境，短语“如果确定”或“如果检测(陈述的条件或事件)”可以被解释成为“当确定时”或“响应于确定”或“当检测(陈述的条件或事件)时”或“响应于检测(陈述的条件或事件)”。
图1是根据本申请实施例的一种训练样本有效性检测方法的流程图，如图1所示，该方法包括：
步骤S101，获取多个扩展问，其中，每个扩展问与对应的一个预设标准问关联。
步骤S102，将多个扩展问随机分为预设份数的样本集，并根据预设比例将预设份数的样本集分为训练集和交叉验证集。
步骤S103，利用训练集训练分类模型。
步骤S104，采用交叉验证方法，并利用分类模型对交叉验证集中的多个扩展问标注，直至所有的扩展问标注完成。
步骤S105，获取分类模型输出的所有的扩展问的标注结果。
步骤S106，根据标注结果得到异常扩展问，异常扩展问的标注结果与所关联的预设标准问不同。
在本方案中，利用交叉验证方法和分类模型来对扩展问进行标注，从而根据标注结果与扩展问关联的预设标准问进行匹配，得到异常扩展问，快速筛选出异常扩展问，提高了训练样本的有效性检测效率，从而避免一些质量低的扩展问来训练智能应答模型。
例如，预设标准问例如为“信用卡逾期怎么办？”，它的扩展问1为“我的信用卡逾期了，需要怎么处理”，扩展问2为“信用卡逾期可以不还吗”，扩展问3为“过了信用卡还款日，怎么办”。可以理解地，扩展问1、扩展问2、扩展问3所关联的预设标准问皆为“信用卡逾期怎么办？”
在本实施例中，训练集用于对分类模型进行训练样本集，为了使训练后的分类模型能够基于用户输入的问题识别出用户可能期望咨询的标准问。训练集中的扩展问为待分类的数据，其类别即为其关联的预设标准问。具体地，分类模型包括输入层、输出层及融合层。其中，输入层包括词语特征输入层和问题特征输入层。每个扩展问所关联的预设标准问作为问题特征，问题特征输入层用于获取输入的扩展问关联的预设标准问的特征，词语特征输入层用于获取扩展问分词处理后得到的词汇的特征。分类模型在经过训练后，能够根据学习到的词汇特性及问题特征来对一个新的扩展问进行分类，使得新的扩展问尽可能地分类至一个准确的标准问下。
可选地，交叉验证方法为十折交叉验证方法或五折交叉验证方法中的任意一种。在一种实施方式中，采用五折交叉验证方法，具体地，将多个扩展问随机分成10份，每次取其中2份作为交叉验证集，其余8份作为训练集。训练时，先用其中的8份对初始化后的分类模型进行训练，然后对2份交叉验证集进行分类标注，以此重复训练及验证过程5次，每次选取的交叉验证集不同，直至所有的扩展问都被标注一遍。可以理解地，根据分类模型的分类结果对交叉验证集中的扩展问进行标注，例如将扩展问4“我的信用卡要逾期了，我该怎么处理”分类至预设标准问“信用卡逾期怎么办”下，那么扩展问4的关联标准问就是分类模型的分类结果。
在另一种实施方式中，将每个预设标准问所关联的多个扩展问随机分成10份，每次从每个预设标准问所关联的10份扩展问中抽取1份作为交叉验证集，其余9份作为训练集。例如：预设标准问有10个，分别为A1，A2…A10，每个预设标准问关联多个扩展问，因此将每个预设标准问所关联的多个扩展问随机分成10份，可以保证交叉验证时每个预设标准问都能得到有效的训练。
可选地，在步骤S105之后，方法还包括：
重复操作M次的步骤S102至步骤S104，其中，M为大于1的自然数，每次重复操作前对分类模型进行初始化设置；获取分类模型的输出的所有的扩展问的M次标注结果；当标注结果与预设标准问相同时，扩展问的当次 计分为第一预设分值；或，当标注结果与预设标准问不同时，扩展问的当次计分为第二预设分值；计算各个扩展问的第一累计得分；筛选出第一累计得分小于第一预设阈值的扩展问；将筛选出的扩展问作为异常扩展问，并输出。
具体地，分类模型标注M次后，对M次的所有扩展问的标注结果进行统计，扩展问的得分越低，其关联的预设标准问是错误的可能性越高。例如：当标注结果与预设标准问相同时，扩展问的得分为1；当标注结果与预设标准问不同时，扩展问的得分为-1，重复标注10次，其中6次得分为1，4次得分为-1，则扩展问的得分为10次标注结果的加和平均值，为0.2分。当第一预设阈值为0.1分时，则表示多个扩展问的分值小于0.1分的皆为异常扩展问。
可选地，所述分类模型可以采用多种机器学习算法，可以是朴素贝叶斯算法、最近邻法则、线性判别分析、支持向量机或决策树算法。在重复操作M次的步骤S102至步骤S104中，所述分类模型所采用的机器学习算法可以相同，也可以不同。采用不同的机器学习算法，能够减小标注错误的偶然误差，通过分类模型对交叉验证集进行多次标注，并将标注结果和预设标准问进行对比，可以有效增强异常扩展问的识别准确性。
可选地，在获取分类模型的输出的所有的扩展问的M次标注结果之后，方法还包括：
当标注结果与预设标准问相同时，由标注结果与预设标准问所组成的标准问对的当次计分为第一预设分值；或，当标注结果与预设标准问不同时，标准问对的当次计分为第二预设分值；计算各个标准问对的第二累计得分；筛选出第二累计得分小于第二预设阈值的标准问对；将筛选出的标准问对作为易混淆标准问对，并输出。
例如，分类模型标注10次后，10个预设标准问(A1，A2…A10)，10种标注结果(a1，a2…a10)，就会有100种可能的标注问对，如：标准问对(A1，a1)。当扩展问的标注结果a1与预设标准问A1相同时，标准问对(A1，a1)的得分为1，不同则得分-1；分类模型标注10次后，标准问对(A1，a1)的得分越低，预设标准问A1与标注结果a1之间越容 易混淆。将易混淆标准问对输出，有利于开发人员修正标注结果，精确分类，降低混淆概率。
可选地，在步骤S105之后，方法还包括：
获取实质相同的多个扩展问的标注结果，实质相同的多个扩展问的语义相似度大于预设值；统计实质相同的多个扩展问的标注结果；将重复次数最多的标注结果作为实质相同的多个扩展问对应的正确预设标准问；将标注结果与正确预设标准问不同的扩展问确认为异常扩展问。
其中，计算语义相似度的方法有余弦相似度计算法、BM25算法等。
在一种实施方式中，计算语义相似度的方法包括：获取同一预设标准问关联的多个扩展问，将每个扩展问进行分词处理并提取关键词，其中，可以采用分词开源工具，例如ICTCLAS(Institute of Computing Technology,Chinese Lexical Analysis System，汉语词法分析系统)、SCWS(Simple Chinese Words Segmentation，简易中文分词系统)等，也可以直接使用自行开发的分词接口对每个子段文本进行分词；通过词向量表示模型获取每个关键词的向量表示，其中，词向量表示模型例如可以是Word2vec等神经网络模型。采用余弦相似度公式分别计算任意两个扩展问的相似度，将相似度大于预设阈值的多个扩展问确认为实质相同的多个扩展问，例如仅标点符号差异的扩展问。
可选地，在将标注结果与正确预设标准问不同的扩展问确认为异常扩展问之后，方法还包括：将异常扩展问所关联的预设标准问修改为正确预设标准问。
可选地，方法还包括：修正异常扩展问，并用修正后的异常扩展问替换原扩展问；用替换后的多个扩展问训练分类模型。
可以理解地，用修正后的异常扩展问替换原扩展问，使得分类模型的训练集中的样本精准度更高，从而进一步保障分类模型输出的标注更加精确。
本申请实施例提供了一种训练样本有效性检测装置，该训练样本有效性检测装置用于执行上述训练样本有效性检测方法，如图2所示，该装置包括：第一获取单元10、制作单元20、训练单元30、标注单元40、第二 获取单元50、第一处理单元60。
第一获取单元10，用于获取多个扩展问，其中，每个所述扩展问与对应的一个预设标准问关联；
制作单元20，用于将多个扩展问随机分为预设份数的样本集，并根据预设比例将预设份数的样本集分为训练集和交叉验证集；
训练单元30，用于利用训练集训练分类模型；
标注单元40，用于采用交叉验证方法，并利用分类模型对交叉验证集中的多个扩展问标注，直至所有的扩展问标注完成；
第二获取单元50，用于获取分类模型输出的所有的扩展问的标注结果；
第一处理单元60，用于根据标注结果得到异常扩展问，异常扩展问的标注结果与所关联的预设标准问不同。
在本方案中，利用交叉验证方法和分类模型来对扩展问进行标注，从而根据标注结果与扩展问关联的预设标准问进行匹配，得到异常扩展问，快速筛选出异常扩展问，提高了训练样本的有效性检测效率，从而避免一些质量低的扩展问来训练智能应答模型。
例如，预设标准问例如为“信用卡逾期怎么办？”，它的扩展问1为“我的信用卡逾期了，需要怎么处理”，扩展问2为“信用卡逾期可以不还吗”，扩展问3为“过了信用卡还款日，怎么办”。可以理解地，扩展问1、扩展问2、扩展问3所关联的预设标准问皆为“信用卡逾期怎么办？”
在本实施例中，训练集用于对分类模型进行训练样本集，为了使训练后的分类模型能够基于用户输入的问题识别出用户可能期望咨询的标准问。训练集中的扩展问为待分类的数据，其类别即为其关联的预设标准问。具体地，分类模型包括输入层、输出层及融合层。其中，输入层包括词语特征输入层和问题特征输入层。每个扩展问所关联的预设标准问作为问题特征，问题特征输入层用于获取输入的扩展问关联的预设标准问的特征，词语特征输入层用于获取扩展问分词处理后得到的词汇的特征。分类模型在经过训练后，能够根据学习到的词汇特性及问题特征来对一个新的扩展问进行分类，使得新的扩展问尽可能地分 类至一个准确的标准问下。
可选地，交叉验证方法为十折交叉验证方法或五折交叉验证方法中的任意一种。在一种实施方式中，采用五折交叉验证方法，具体地，将多个扩展问随机分成10份，每次取其中2份作为交叉验证集，其余8份作为训练集。训练时，先用其中的8份对初始化后的分类模型进行训练，然后对2份交叉验证集进行分类标注，以此重复训练及验证过程5次，每次选取的交叉验证集不同，直至所有的扩展问都被标注一遍。可以理解地，根据分类模型的分类结果对交叉验证集中的扩展问进行标注，例如分类模型将扩展问4“我的信用卡要逾期了，我该怎么处理”分类至预设标准问“信用卡逾期怎么办”下，那么扩展问4的关联标准问就是分类模型的分类结果。
在另一种实施方式中，将每个预设标准问所关联的多个扩展问随机分成10份，每次从每个预设标准问所关联的10份扩展问中抽取1份作为交叉验证集，其余9份作为训练集。例如：预设标准问有10个，分别为A1，A2…A10，每个预设标准问关联多个扩展问，因此将每个预设标准问所关联的多个扩展问随机分成10份，可以保证交叉验证时每个预设标准问都能得到有效的训练。
可选地，装置还包括重复单元、第二获取单元、计分单元、计算单元、筛选单元、输出单元。
重复单元，用于控制所述制作单元20、所述训练单元30及所述标注单元40重复操作M次，其中，M为大于1的自然数，每次重复操作前对分类模型进行初始化设置；第二获取单元，还用于获取分类模型的输出的所有的扩展问的M次标注结果；计分单元，用于当扩展问的标注结果与预设标准问相同时，扩展问的当次计分为第一预设分值；或，当扩展问的标注结果与预设标准问不同时，扩展问的当次计分为第二预设分值；计算单元，用于计算各个扩展问的第一累计得分；筛选单元，用于筛选出第一累计得分小于第一预设阈值的扩展问；输出单元，用于将筛选出的扩展问作为异常扩展问，并输出。
具体地，分类模型标注M次后，对M次的所有扩展问的标注结果进行统计，扩展问的得分越低，其关联的预设标准问是错误的可能性越高。例 如：当标注结果与预设标准问相同时，扩展问的得分为1；当标注结果与预设标准问不同时，扩展问的得分为-1，重复标注10次，其中6次得分为1，4次得分为-1，则扩展问的得分为10次标注结果的加和平均值，为0.2分。当第一预设阈值为0.1分时，则表示多个扩展问的分值小于0.1分的皆为异常扩展问。
可选地，所述分类模型可以采用多种机器学习算法，可以是朴素贝叶斯算法、最近邻法则、线性判别分析、支持向量机或决策树算法。在重复操作M次时，所述分类模型所采用的机器学习算法可以相同，也可以不同。采用不同的机器学习算法，能够减小标注错误的偶然误差，通过分类模型对交叉验证集进行多次标注并将标注结果和预设标准问进行对比，可以有效增强异常扩展问的识别准确性。
可选地，计分单元，还用于当标注结果与预设标准问相同时，由标注结果与预设标准问所组成的标准问对的当次计分为第一预设分值；或，当标注结果与预设标准问不同时，标准问对的当次计分为第二预设分值；计算单元，还用于计算各个标准问对的第二累计得分；筛选单元，还用于筛选出第二累计得分小于第二预设阈值的标准问对；输出单元，还用于将筛选出的标准问对作为易混淆标准问对，并输出。
例如，分类模型标注10次后，10个预设标准问(A1，A2…A10)，10种标注结果(a1，a2…a10)，就会有100种可能的标注问对，如：标准问对(A1，a1)。当扩展问的标注结果a1与预设标准问A1相同时，标准问对(A1，a1)的得分为1，不同则得分-1；分类模型标注10次后，标准问对(A1，a1)的得分越低，预设标准问A1与标注结果a1之间越容易混淆。将易混淆标准问对输出，有利于开发人员修正标注结果，精确分类，降低混淆概率。
可选地，装置还包括第三获取单元、统计单元、第二处理单元、确认单元。
第三获取单元，用于获取实质相同的多个扩展问的标注结果，实质相同的多个扩展问的语义相似度大于预设值；统计单元，用于统计实质相同的多个扩展问的标注结果；第二处理单元，用于将重复次数最多的标注结 果作为实质相同的多个扩展问对应的正确预设标准问；确认单元，用于将标注结果与正确预设标准问不同的扩展问确认为异常扩展问。
其中，计算语义相似度的方法有余弦相似度计算法、BM25算法等。
在一种实施方式中，计算语义相似度的方法包括：获取同一预设标准问关联的多个扩展问，将每个扩展问进行分词处理并提取关键词，其中，可以采用分词开源工具，例如ICTCLAS、SCWS等，也可以直接使用自行开发的分词接口对每个子段文本进行分词；通过词向量表示模型获取每个关键词的向量表示，其中，词向量表示模型例如可以是WORD2VEC等神经网络模型。采用余弦相似度公式分别计算任意两个扩展问的相似度，将相似度大于预设阈值的多个扩展问确认为实质相同的多个扩展问，例如仅标点符号差异的扩展问。
可选地，装置还包括修改单元，用于将异常扩展问所关联的预设标准问修改为正确预设标准问。
可选地，装置还包括修正单元和替换单元；
修正单元，用于修正异常扩展问，并用修正后的异常扩展问替换原扩展问；替换单元，用于用替换后的多个扩展问训练分类模型。
可以理解地，用修正后的异常扩展问替换原扩展问，使得分类模型的训练集中的样本精准度更高，从而进一步保障分类模型输出的标注更加精确。
本申请实施例提供了一种计算机非易失性存储介质，存储介质包括存储的程序，其中，在程序运行时控制存储介质所在设备执行以下步骤：
步骤S101，获取多个扩展问，其中，每个扩展问与对应的一个预设标准问关联；步骤S102，将多个扩展问随机分为预设份数的样本集，并根据预设比例将预设份数的样本集分为训练集和交叉验证集；步骤S103，利用训练集训练分类模型；步骤S104，采用交叉验证方法，并利用分类模型对交叉验证集中的多个扩展问标注，直至所有的扩展问标注完成；步骤S105，获取分类模型输出的所有的扩展问的标注结果；步骤S106，根据标注结果得到异常扩展问，异常扩展问的标注结果与所关联的预设标准问不同。
可选地，在程序运行时控制存储介质所在设备还执行以下步骤：重复操作M次的步骤S102至步骤S104，其中，M为大于1的自然数，每次重复操作前对分类模型进行初始化设置；获取分类模型的输出的所有的扩展问的M次标注结果；当扩展问的标注结果与预设标准问相同时，扩展问的当次计分为第一预设分值；或，当扩展问的标注结果与预设标准问不同时，扩展问的当次计分为第二预设分值；计算各个扩展问的第一累计得分；筛选出第一累计得分小于第一预设阈值的扩展问；将筛选出的扩展问作为异常扩展问，并输出。
可选地，在程序运行时控制存储介质所在设备在执行获取分类模型的输出的所有的扩展问的M次标注结果之后，还包括：当标注结果与预设标准问相同时，由标注结果与预设标准问所组成的标准问对的当次计分为第一预设分值；或，当标注结果与预设标准问不同时，标准问对的当次计分为第二预设分值；计算各个标准问对的第二累计得分；筛选出第二累计得分小于第二预设阈值的标准问对；将筛选出的标准问对作为易混淆标准问对，并输出。
可选地，在程序运行时控制存储介质所在设备在执行步骤S105之后，还包括：
获取实质相同的多个扩展问的标注结果，实质相同的多个扩展问的语义相似度大于预设值；统计实质相同的多个扩展问的标注结果；将重复次数最多的标注结果作为实质相同的多个扩展问对应的正确预设标准问；将标注结果与正确预设标准问不同的扩展问确认为异常扩展问。
可选地，在程序运行时控制存储介质所在设备在执行将标注结果与正确预设标准问不同的扩展问确认为异常扩展问之后，还包括：
将异常扩展问所关联的预设标准问修改为正确预设标准问。
图3是本申请实施例提供的一种计算机设备的示意图。如图3所示，该实施例的计算机设备100包括：处理器101、存储器102以及存储在存储器102中并可在处理器101上运行的计算机程序103，该计算机程序103被处理器101执行时实现实施例中的训练样本有效性检测方法，为避免重复，此处不一一赘述。或者，该计算机程序被处理器101执行时实 现实施例中训练样本有效性检测装置中各模型/单元的功能，为避免重复，此处不一一赘述。
计算机设备100可以是桌上型计算机、笔记本、掌上电脑及云端服务器等计算设备。计算机设备可包括，但不仅限于，处理器101、存储器102。本领域技术人员可以理解，图3仅仅是计算机设备100的示例，并不构成对计算机设备100的限定，可以包括比图示更多或更少的部件，或者组合某些部件，或者不同的部件，例如计算机设备还可以包括输入输出设备、网络接入设备、总线等。
所称处理器101可以是中央处理单元(Central Processing Unit，CPU)，还可以是其他通用处理器、数字信号处理器(Digital Signal Processor，DSP)、专用集成电路(Application Specific Integrated Circuit，ASIC)、现场可编程门阵列(Field-Programmable Gate Array，FPGA)或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件等。通用处理器可以是微处理器或者该处理器也可以是任何常规的处理器等。
存储器102可以是计算机设备100的内部存储单元，例如计算机设备100的硬盘或内存。存储器102也可以是计算机设备100的外部存储设备，例如计算机设备100上配备的插接式硬盘，智能存储卡(Smart Media Card,SMC)，安全数字(Secure Digital,SD)卡，闪存卡(Flash Card)等。进一步地，存储器102还可以既包括计算机设备100的内部存储单元也包括外部存储设备。存储器102用于存储计算机程序以及计算机设备所需的其他程序和数据。存储器102还可以用于暂时地存储已经输出或者将要输出的数据。
需要说明的是，本申请实施例中所涉及的终端可以包括但不限于个人计算机(Personal Computer，PC)、个人数字助理(Personal Digital Assistant，PDA)、无线手持设备、平板电脑(Tablet Computer)、手机、MP3播放器、MP4播放器等。
可以理解的是，所述应用可以是安装在终端上的应用程序(native App)，或者还可以是终端上的浏览器的一个网页程序(web App)，本申 请实施例对此不进行限定。
所属领域的技术人员可以清楚地了解到，为描述的方便和简洁，上述描述的系统，装置和单元的具体工作过程，可以参考前述方法实施例中的对应过程，在此不再赘述。
在本申请所提供的几个实施例中，应该理解到，所揭露的系统，装置和方法，可以通过其它的方式实现。例如，以上所描述的装置实施例仅仅是示意性的，例如，所述单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如，多个单元或组件可以结合或者可以集成到另一个系统，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，装置或单元的间接耦合或通信连接，可以是电性，机械或其它的形式。
所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方案的目的。
上述以软件功能单元的形式实现的集成的单元，可以存储在一个计算机可读取存储介质中。上述软件功能单元存储在一个存储介质中，包括若干指令用以使得一台计算机装置(可以是个人计算机，服务器，或者网络装置等)或处理器(Processor)执行本申请各个实施例所述方法的部分步骤。而前述的存储介质包括：U盘、移动硬盘、只读存储器(Read-Only Memory，ROM)、随机存取存储器(Random Access Memory，RAM)、磁碟或者光盘等各种可以存储程序代码的介质。
以上所述仅为本申请的较佳实施例而已，并不用以限制本申请，凡在本申请的精神和原则之内，所做的任何修改、等同替换、改进等，均应包含在本申请保护的范围之内。

Claims (20)

  1. 一种训练样本有效性检测方法，其特征在于，所述方法包括：
    步骤S101，获取多个扩展问，其中，每个所述扩展问与对应的一个预设标准问关联；
    步骤S102，将多个所述扩展问随机分为预设份数的样本集，并根据预设比例将所述预设份数的样本集分为训练集和交叉验证集；
    步骤S103，利用所述训练集训练分类模型；
    步骤S104，采用交叉验证方法，并利用所述分类模型对所述交叉验证集中的多个扩展问标注，直至所有的所述扩展问标注完成；
    步骤S105，获取所述分类模型输出的所有的所述扩展问的标注结果；
    步骤S106，根据所述标注结果得到异常扩展问，所述异常扩展问的标注结果与所关联的预设标准问不同。
  2. 根据权利要求1所述的方法，其特征在于，在所述步骤S105之后，所述方法还包括：
    重复操作M次的所述步骤S102至所述步骤S104，其中，M为大于1的自然数，每次重复操作前对所述分类模型进行初始化设置；
    获取所述分类模型的输出的所有的所述扩展问的M次标注结果；
    当所述扩展问的标注结果与所述预设标准问相同时，所述扩展问的当次计分为第一预设分值；或，当所述扩展问的标注结果与所述预设标准问不同时，所述扩展问的当次计分为第二预设分值；
    计算各个所述扩展问的第一累计得分；
    筛选出所述第一累计得分小于第一预设阈值的所述扩展问；
    将筛选出的扩展问作为所述异常扩展问，并输出。
  3. 根据权利要求2所述的方法，其特征在于，在所述获取所述分类模型的输出的所有的所述扩展问的M次标注结果之后，所述方法还包括：
    当所述标注结果与所述预设标准问相同时，由所述标注结果与所述预设标准问所组成的标准问对的当次计分为所述第一预设分值；或，当所述标注结果与所述预设标准问不同时，所述标准问对的当次计分为所述第二预设分值；
    计算各个所述标准问对的第二累计得分；
    筛选出所述第二累计得分小于第二预设阈值的所述标准问对；
    将筛选出的标准问对作为易混淆标准问对，并输出。
  4. 根据权利要求1所述的方法，其特征在于，在所述步骤S105之后，所述方法还包括：
    获取实质相同的多个扩展问的标注结果，所述实质相同的多个扩展问的语义相似度大于预设值；
    统计所述实质相同的多个扩展问的标注结果；
    将重复次数最多的标注结果作为所述实质相同的多个扩展问对应的 正确预设标准问；
    将所述标注结果与所述正确预设标准问不同的扩展问确认为所述异常扩展问。
  5. 根据权利要求4所述的方法，其特征在于，在所述将所述标注结果与所述正确预设标准问不同的扩展问确认为所述异常扩展问之后，所述方法还包括：
    将所述异常扩展问所关联的预设标准问修改为所述正确预设标准问。
  6. 根据权利要求1～5中任意一项所述的方法，其特征在于，所述方法还包括：
    修正所述异常扩展问，并用修正后的所述异常扩展问替换原扩展问；
    用替换后的多个所述扩展问训练所述分类模型。
  7. 根据权利要求1所述的方法，其特征在于，所述交叉验证方法为十折交叉验证方法或五折交叉验证方法中的任意一种。
  8. 一种训练样本有效性检测装置，其特征在于，所述装置包括：
    第一获取单元，用于获取多个扩展问，其中，每个所述扩展问与对应的一个预设标准问关联；
    制作单元，用于将多个所述扩展问随机分为预设份数的样本集，并根据预设比例将所述预设份数的样本集分为训练集和交叉验证集；
    训练单元，用于利用所述训练集训练分类模型；
    标注单元，用于采用交叉验证方法，并利用所述分类模型对所述交叉验证集中的多个扩展问标注，直至所有的所述扩展问标注完成；
    第二获取单元，用于获取所述分类模型输出的所有的所述扩展问的标注结果；
    第一处理单元，用于根据所述标注结果得到异常扩展问，所述异常扩展问的标注结果与所关联的预设标准问不同。
  9. 根据权利要求8所述的装置，其特征在于，所述装置还包括：
    重复单元，用于控制所述制作单元、所述训练单元及所述标注单元重复操作M次，其中，M为大于1的自然数，每次重复操作前对所述分类模型进行初始化设置；
    所述第二获取单元，还用于获取所述分类模型的输出的所有的所述扩展问的M次标注结果；
    计分单元，用于当所述扩展问的标注结果与所述预设标准问相同时，所述扩展问的当次计分为第一预设分值；或，当所述扩展问的标注结果与所述预设标准问不同时，所述扩展问的当次计分为第二预设分值；
    计算单元，用于计算各个所述扩展问的第一累计得分；
    筛选单元，用于筛选出所述第一累计得分小于第一预设阈值的扩展问；
    输出单元，用于将筛选出的扩展问作为所述异常扩展问，并输出。
  10. 根据权利要求9所述的装置，其特征在于，
    所述计分单元，还用于当所述标注结果与所述预设标准问相同时，由 所述标注结果与所述预设标准问所组成的标准问对的当次计分为所述第一预设分值；或，当所述标注结果与所述预设标准问不同时，所述标准问对的当次计分为所述第二预设分值；
    所述计算单元，还用于计算各个所述标准问对的第二累计得分；
    所述筛选单元，还用于筛选出所述第二累计得分小于第二预设阈值的所述标准问对；
    所述输出单元，还用于将筛选出的标准问对作为易混淆标准问对，并输出。
  11. 根据权利要求8所述的装置，其特征在于，所述装置还包括：
    第三获取单元，用于获取实质相同的多个扩展问的标注结果，所述实质相同的多个扩展问的语义相似度大于预设值；
    统计单元，用于统计所述实质相同的多个扩展问的标注结果；
    第二处理单元，用于将重复次数最多的标注结果作为所述实质相同的多个扩展问对应的正确预设标准问；
    确认单元，用于将所述标注结果与所述正确预设标准问不同的扩展问确认为所述异常扩展问。
  12. 根据权利要求8～11任一项所述的装置，其特征在于，所述装置还包括：
    修正单元，用于修正异常扩展问，并用修正后的异常扩展问替换原扩展问；
    替换单元，用于用替换后的多个扩展问训练分类模型。
  13. 一种计算机非易失性存储介质，所述存储介质包括存储的程序，其特征在于，在所述程序运行时控制所述存储介质所在设备执行以下步骤：
    步骤S101，获取多个扩展问，其中，每个所述扩展问与对应的一个预设标准问关联；
    步骤S102，将多个所述扩展问随机分为预设份数的样本集，并根据预设比例将所述预设份数的样本集分为训练集和交叉验证集；
    步骤S103，利用所述训练集训练分类模型；
    步骤S104，采用交叉验证方法，并利用所述分类模型对所述交叉验证集中的多个扩展问标注，直至所有的所述扩展问标注完成；
    步骤S105，获取所述分类模型输出的所有的所述扩展问的标注结果；
    步骤S106，根据所述标注结果得到异常扩展问，所述异常扩展问的标注结果与所关联的预设标准问不同。
  14. 根据权利要求13所述的计算机非易失性存储介质，其特征在于，在所述程序运行时控制所述存储介质所在设备执行以下步骤：
    重复操作M次的所述步骤S102至所述步骤S104，其中，M为大于1的自然数，每次重复操作前对所述分类模型进行初始化设置；
    获取所述分类模型的输出的所有的所述扩展问的M次标注结果；
    当所述扩展问的标注结果与所述预设标准问相同时，所述扩展问的当 次计分为第一预设分值；或，当所述扩展问的标注结果与所述预设标准问不同时，所述扩展问的当次计分为第二预设分值；
    计算各个所述扩展问的第一累计得分；
    筛选出所述第一累计得分小于第一预设阈值的所述扩展问；
    将筛选出的扩展问作为所述异常扩展问，并输出。
  15. 根据权利要求14所述的计算机非易失性存储介质，其特征在于，在所述程序运行时控制所述存储介质所在设备在执行所述获取所述分类模型的输出的所有的所述扩展问的M次标注结果之后，还包括：
    当所述标注结果与所述预设标准问相同时，由所述标注结果与所述预设标准问所组成的标准问对的当次计分为所述第一预设分值；或，当所述标注结果与所述预设标准问不同时，所述标准问对的当次计分为所述第二预设分值；
    计算各个所述标准问对的第二累计得分；
    筛选出所述第二累计得分小于第二预设阈值的所述标准问对；
    将筛选出的标准问对作为易混淆标准问对，并输出。
  16. 根据权利要求13所述的计算机非易失性存储介质，其特征在于，在所述程序运行时控制所述存储介质所在设备在执行步骤S105之后，还包括：
    获取实质相同的多个扩展问的标注结果，所述实质相同的多个扩展问的语义相似度大于预设值；
    统计所述实质相同的多个扩展问的标注结果；
    将重复次数最多的标注结果作为所述实质相同的多个扩展问对应的正确预设标准问；
    将所述标注结果与所述正确预设标准问不同的扩展问确认为所述异常扩展问。
  17. 一种计算机设备，包括存储器、处理器以及存储在所述存储器中并可在所述处理器上运行的计算机程序，其特征在于，所述处理器执行所述计算机程序时实现以下步骤：
    步骤S101，获取多个扩展问，其中，每个所述扩展问与对应的一个预设标准问关联；
    步骤S102，将多个所述扩展问随机分为预设份数的样本集，并根据预设比例将所述预设份数的样本集分为训练集和交叉验证集；
    步骤S103，利用所述训练集训练分类模型；
    步骤S104，采用交叉验证方法，并利用所述分类模型对所述交叉验证集中的多个扩展问标注，直至所有的所述扩展问标注完成；
    步骤S105，获取所述分类模型输出的所有的所述扩展问的标注结果；
    步骤S106，根据所述标注结果得到异常扩展问，所述异常扩展问的标注结果与所关联的预设标准问不同。
  18. 根据权利要求17所述的计算机设备，其特征在于，所述处理器 执行所述计算机程序时还实现以下步骤：
    重复操作M次的步骤S102至步骤S104，其中，M为大于1的自然数，每次重复操作前对分类模型进行初始化设置；
    获取分类模型的输出的所有的扩展问的M次标注结果；
    当扩展问的标注结果与预设标准问相同时，扩展问的当次计分为第一预设分值；或，当扩展问的标注结果与预设标准问不同时，扩展问的当次计分为第二预设分值；
    计算各个所述扩展问的第一累计得分；
    筛选出所述第二累计得分小于第二预设阈值的所述标准问对；
    将筛选出的标准问对作为易混淆标准问对，并输出。
  19. 根据权利要求18所述的计算机设备，其特征在于，所述处理器执行所述计算机程序时在实现所述获取所述分类模型的输出的所有的所述扩展问的M次标注结果之后，还实现以下步骤：
    当标注结果与预设标准问相同时，由标注结果与预设标准问所组成的标准问对的当次计分为第一预设分值；或，当标注结果与预设标准问不同时，标准问对的当次计分为第二预设分值；计算各个标准问对的第二累计得分；筛选出第二累计得分小于第二预设阈值的标准问对；将筛选出的标准问对作为易混淆标准问对，并输出。
  20. 根据权利要求17所述的计算机设备，其特征在于，所述处理器执行所述计算机程序时在实现步骤S105之后，还实现以下步骤：
    获取实质相同的多个扩展问的标注结果，所述实质相同的多个扩展问的语义相似度大于预设值；
    统计所述实质相同的多个扩展问的标注结果；
    将重复次数最多的标注结果作为所述实质相同的多个扩展问对应的正确预设标准问；
    将所述标注结果与所述正确预设标准问不同的扩展问确认为所述异常扩展问。
PCT/CN2019/117090 2019-01-10 2019-11-11 一种训练样本有效性检测方法、计算机设备及计算机非易失性存储介质 Ceased WO2020143301A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201910022265.4 2019-01-10
CN201910022265.4A CN109902157A (zh) 2019-01-10 2019-01-10 一种训练样本有效性检测方法及装置

Publications (1)

Publication Number Publication Date
WO2020143301A1 true WO2020143301A1 (zh) 2020-07-16

Family

ID=66943596

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/117090 Ceased WO2020143301A1 (zh) 2019-01-10 2019-11-11 一种训练样本有效性检测方法、计算机设备及计算机非易失性存储介质

Country Status (2)

Country Link
CN (1) CN109902157A (zh)
WO (1) WO2020143301A1 (zh)

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109902157A (zh) * 2019-01-10 2019-06-18 平安科技（深圳）有限公司 一种训练样本有效性检测方法及装置
CN110705225A (zh) * 2019-08-15 2020-01-17 平安信托有限责任公司 一种合同标注方法及装置
CN110797101B (zh) * 2019-10-28 2023-11-03 腾讯医疗健康(深圳)有限公司 医学数据处理方法、装置、可读存储介质和计算机设备
CN111259625B (zh) * 2020-01-16 2023-06-27 平安科技（深圳）有限公司 意图识别方法、装置、设备及计算机可读存储介质
CN113554474B (zh) * 2021-08-11 2024-08-20 上海明略人工智能(集团)有限公司 模型的验证方法及装置、电子设备和计算机可读存储介质

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108009589A (zh) * 2017-12-12 2018-05-08 腾讯科技（深圳）有限公司 样本数据处理方法、装置和计算机可读存储介质
CN108509969A (zh) * 2017-09-06 2018-09-07 腾讯科技（深圳）有限公司 数据标注方法及终端
CN108734296A (zh) * 2017-04-21 2018-11-02 北京京东尚科信息技术有限公司 优化监督学习的训练数据的方法、装置、电子设备和介质
CN109902157A (zh) * 2019-01-10 2019-06-18 平安科技（深圳）有限公司 一种训练样本有效性检测方法及装置

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106202177B (zh) * 2016-06-27 2017-12-15 腾讯科技（深圳）有限公司 一种文本分类方法及装置
CN106649742B (zh) * 2016-12-26 2023-04-18 上海智臻智能网络科技股份有限公司 数据库维护方法和装置
CN108197115B (zh) * 2018-01-26 2022-04-22 上海智臻智能网络科技股份有限公司 智能交互方法、装置、计算机设备和计算机可读存储介质

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108734296A (zh) * 2017-04-21 2018-11-02 北京京东尚科信息技术有限公司 优化监督学习的训练数据的方法、装置、电子设备和介质
CN108509969A (zh) * 2017-09-06 2018-09-07 腾讯科技（深圳）有限公司 数据标注方法及终端
CN108009589A (zh) * 2017-12-12 2018-05-08 腾讯科技（深圳）有限公司 样本数据处理方法、装置和计算机可读存储介质
CN109902157A (zh) * 2019-01-10 2019-06-18 平安科技（深圳）有限公司 一种训练样本有效性检测方法及装置

Also Published As

Publication number Publication date
CN109902157A (zh) 2019-06-18

Similar Documents

Publication Publication Date Title
CN112016304B (zh) 文本纠错方法、装置、电子设备及存储介质
TWI752455B (zh) 圖像分類模型訓練方法、影像處理方法、資料分類模型訓練方法、資料處理方法、電腦設備、儲存媒介
WO2020143301A1 (zh) 一种训练样本有效性检测方法、计算机设备及计算机非易失性存储介质
CN111382248B (zh) 一种问题回复方法、装置、存储介质及终端设备
CN110008342A (zh) 文献分类方法、装置、设备及存储介质
TWI682287B (zh) 知識圖譜產生裝置、方法及其電腦程式產品
CN112035605B (zh) 题目推荐方法、装置、设备及存储介质
CN107491536B (zh) 一种试题校验方法、试题校验装置及电子设备
US20230045330A1 (en) Multi-term query subsumption for document classification
US10147020B1 (en) System and method for computational disambiguation and prediction of dynamic hierarchical data structures
CN107169021A (zh) 用于预测应用功能标签的方法和设备
CN104142912A (zh) 一种精确的语料类别标注方法及装置
CN114385918A (zh) 消息推送方法、装置、计算机设备及存储介质
CN114067343A (zh) 一种数据集的构建方法、模型训练方法和对应装置
CN108776677B (zh) 平行语句库的创建方法、设备及计算机可读存储介质
CN111737472A (zh) 文本分类模型的更新方法及系统、电子设备及存储介质
WO2021174814A1 (zh) 众包任务的答案验证方法、装置、计算机设备及存储介质
CN107357782B (zh) 一种识别用户性别的方法及终端
CN110705281A (zh) 一种基于机器学习的简历信息抽取方法
CN105243053A (zh) 提取文档关键句的方法及装置
CN112560463A (zh) 文本多标注方法、装置、设备及存储介质
CN111914868A (zh) 模型训练方法、异常数据检测方法、装置和电子设备
CN109299007A (zh) 一种缺陷修复者自动推荐方法
CN116186223B (zh) 一种金融文本处理方法、装置、设备和存储介质
CN110717029A (zh) 一种信息处理方法和系统

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19908170

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19908170

Country of ref document: EP

Kind code of ref document: A1