CN114528391A - Method, device and equipment for training question-answer pair scoring model and storage medium - Google Patents
Method, device and equipment for training question-answer pair scoring model and storage medium Download PDFInfo
- Publication number
- CN114528391A CN114528391A CN202210182951.XA CN202210182951A CN114528391A CN 114528391 A CN114528391 A CN 114528391A CN 202210182951 A CN202210182951 A CN 202210182951A CN 114528391 A CN114528391 A CN 114528391A
- Authority
- CN
- China
- Prior art keywords
- question
- answer
- sample
- score
- training
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012549 training Methods 0.000 title claims abstract description 103
- 238000000034 method Methods 0.000 title claims abstract description 42
- 238000013145 classification model Methods 0.000 claims abstract description 25
- 238000013210 evaluation model Methods 0.000 claims abstract description 16
- 238000013528 artificial neural network Methods 0.000 claims description 64
- 239000013598 vector Substances 0.000 claims description 60
- 238000004590 computer program Methods 0.000 claims description 15
- 238000004458 analytical method Methods 0.000 claims description 4
- 230000002708 enhancing effect Effects 0.000 claims description 2
- 238000013473 artificial intelligence Methods 0.000 abstract description 2
- 238000010586 diagram Methods 0.000 description 10
- 238000004364 calculation method Methods 0.000 description 9
- 238000011156 evaluation Methods 0.000 description 3
- 238000012545 processing Methods 0.000 description 3
- PCTMTFRHKVHKIS-BMFZQQSSSA-N (1s,3r,4e,6e,8e,10e,12e,14e,16e,18s,19r,20r,21s,25r,27r,30r,31r,33s,35r,37s,38r)-3-[(2r,3s,4s,5s,6r)-4-amino-3,5-dihydroxy-6-methyloxan-2-yl]oxy-19,25,27,30,31,33,35,37-octahydroxy-18,20,21-trimethyl-23-oxo-22,39-dioxabicyclo[33.3.1]nonatriaconta-4,6,8,10 Chemical compound C1C=C2C[C@@H](OS(O)(=O)=O)CC[C@]2(C)[C@@H]2[C@@H]1[C@@H]1CC[C@H]([C@H](C)CCCC(C)C)[C@@]1(C)CC2.O[C@H]1[C@@H](N)[C@H](O)[C@@H](C)O[C@H]1O[C@H]1/C=C/C=C/C=C/C=C/C=C/C=C/C=C/[C@H](C)[C@@H](O)[C@@H](C)[C@H](C)OC(=O)C[C@H](O)C[C@H](O)CC[C@@H](O)[C@H](O)C[C@H](O)C[C@](O)(C[C@H](O)[C@H]2C(O)=O)O[C@H]2C1 PCTMTFRHKVHKIS-BMFZQQSSSA-N 0.000 description 2
- 238000012360 testing method Methods 0.000 description 2
- 230000002457 bidirectional effect Effects 0.000 description 1
- 230000019771 cognition Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000011176 pooling Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/33—Querying
- G06F16/332—Query formulation
- G06F16/3329—Natural language query formulation or dialogue systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/33—Querying
- G06F16/3331—Query processing
- G06F16/3332—Query translation
- G06F16/3335—Syntactic pre-processing, e.g. stopword elimination, stemming
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/33—Querying
- G06F16/338—Presentation of query results
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/35—Clustering; Classification
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/084—Backpropagation, e.g. using gradient descent
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Mathematical Physics (AREA)
- Databases & Information Systems (AREA)
- Artificial Intelligence (AREA)
- Biomedical Technology (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Health & Medical Sciences (AREA)
- Evolutionary Computation (AREA)
- Biophysics (AREA)
- Software Systems (AREA)
- Life Sciences & Earth Sciences (AREA)
- Health & Medical Sciences (AREA)
- Human Computer Interaction (AREA)
- Electrically Operated Instructional Devices (AREA)
Abstract
The application relates to the field of artificial intelligence, and particularly discloses a method, a device, equipment and a storage medium for training a question-answer pair scoring model, wherein the method comprises the following steps: obtaining a sample question-answer pair, wherein the sample question-answer pair comprises a question and a correct answer corresponding to the question; carrying out data enhancement on the correct answer to obtain an incorrect answer; determining a first score of the sample question-answer pair according to the question, the wrong answer and the correct answer; performing word order scoring on the sample question-answer pairs to obtain second scores of the sample question-answer pairs; and performing iterative training on a classification model according to the first score and the second score, and taking the trained classification model as a question-answer pair evaluation model.
Description
Technical Field
The application relates to the field of artificial intelligence, in particular to a method, a device, equipment and a storage medium for training a question-answer pair scoring model.
Background
The retrieval type question-answering system is one of common question-answering system structures, and has stable and controllable reminding by searching a question which is most similar to an input question in an original question-answering library as a target and returning an answer of the similar question in the existing question-answering library as an answer. However, the quality of the query-based answering system is closely related to the quality of the question-answer library in the query-based answering system, and therefore, the quality of the question-answer pairs in the query-based answering system needs to be measured.
Disclosure of Invention
The application provides a method, a device, equipment and a storage medium for training a question-answer pair scoring model so as to score question-answer pairs in a question-answer library.
In a first aspect, the present application provides a method for training a question-answer pair score model, the method including:
obtaining a sample question-answer pair, wherein the sample question-answer pair comprises a question and a correct answer corresponding to the question;
performing data enhancement on the correct answer to obtain an incorrect answer;
determining a first score of the sample question-answer pair according to the question, the wrong answer and the correct answer;
performing word order scoring on the sample question-answer pairs to obtain second scores of the sample question-answer pairs;
and performing iterative training on a classification model according to the first score and the second score, and taking the trained classification model as a question-answer pair evaluation model.
In a second aspect, the present application further provides a device for training a question-answer pair score model, the device comprising:
the system comprises a sample acquisition module, a sample analysis module and a sample analysis module, wherein the sample acquisition module is used for acquiring a sample question-answer pair, and the sample question-answer pair comprises a question and a correct answer corresponding to the question;
the data enhancement module is used for enhancing the data of the correct answer to obtain an incorrect answer;
a first score module, configured to determine a first score of the sample question-answer pair according to the question, the wrong answer, and the correct answer;
the second score module is used for scoring the word order of the sample question-answer pairs to obtain second scores of the sample question-answer pairs;
and the model training module is used for carrying out iterative training on the classification model according to the first score and the second score and taking the trained classification model as a question-answer pair scoring model.
In a third aspect, the present application further provides a computer device comprising a memory and a processor; the memory is used for storing a computer program; the processor is used for executing the computer program and implementing the training method of the question-answer pair scoring model when the computer program is executed.
In a fourth aspect, the present application further provides a computer-readable storage medium storing a computer program, which when executed by a processor causes the processor to implement the method for training a question-answer pair score model as described above.
The application discloses a method, a device, equipment and a storage medium for training a question-answer pair scoring model. When the question-answer pair evaluation model is trained, the sequence of correct answers in all answers of the question in the sample question-answer pair and the rationalization degree of the word sequence of the sample question-answer pair are combined, so that the comprehensive degree of the obtained question-answer pair evaluation model when the question-answer pair is evaluated is improved, and the accuracy of the evaluation result is improved.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
FIG. 1 is a flow chart illustrating the steps of a method for training a question-answer pair score model according to an embodiment of the present application;
fig. 2 is a schematic flowchart of steps provided in an embodiment of the present application for determining a first score of a sample question-answer pair;
fig. 3 is a schematic structural diagram of a first neural network provided in an embodiment of the present application;
FIG. 4 is a schematic diagram of a training architecture of a question-answer pair scoring model according to an embodiment of the present disclosure;
FIG. 5 is a schematic block diagram of a training apparatus for a question-answer pair score model according to an embodiment of the present application;
fig. 6 is a schematic block diagram of a structure of a computer device according to an embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some, but not all, embodiments of the present application. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
The flow diagrams depicted in the figures are merely illustrative and do not necessarily include all of the elements and operations/steps, nor do they necessarily have to be performed in the order depicted. For example, some operations/steps may be decomposed, combined or partially combined, so that the actual execution sequence may be changed according to the actual situation.
It is to be understood that the terminology used in the description of the present application herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in the specification of the present application and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.
It should also be understood that the term "and/or" as used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.
The embodiment of the application provides a method and a device for training a question-answer pair scoring model, computer equipment and a storage medium.
In order to maintain the robustness and the question-answering quality of a question-answering library in an retrieval type question-answering system, the quality of question-answering pairs in the question-answering library is required to be ensured not to be lower than a certain standard, a large amount of manpower and time are required to be consumed if manual auditing is adopted, and the auditing standard changes along with the cognition of auditors.
The method for training the question-answer pair scoring model can be used for scoring the quality of the question-answer pairs in the question-answer library, so that the manpower input in the management of the question-answer library is reduced, and the quality of the question-answer pairs in the question-answer library can be scored to improve the quality of the question-answer pairs in the question-answer library.
Some embodiments of the present application will be described in detail below with reference to the accompanying drawings. The embodiments described below and the features of the embodiments can be combined with each other without conflict.
Referring to fig. 1, fig. 1 is a schematic flow chart of a method for training a question-answer pair score model according to an embodiment of the present application. According to the method for training the question-answer pair evaluation model, the sample question-answer pairs are scored from different angles, so that the question-answer pair evaluation model is trained according to the scores of the sample question-answer pairs in different aspects, and the trained question-answer pair evaluation model can comprehensively evaluate the question-answer pairs in multiple aspects.
As shown in fig. 1, the method for training the question-answer pair score model specifically includes: step S101 to step S105.
S101, obtaining a sample question-answer pair, wherein the sample question-answer pair comprises a question and a correct answer corresponding to the question.
A part of question-answer pairs can be extracted from the existing question-answer pairs as sample question-answer pairs, and the sample question-answer pairs comprise questions and correct answers corresponding to the questions.
And S102, performing data enhancement on the correct answer to obtain an incorrect answer.
And performing data enhancement on the correct answer in the sample question-answer pair, and taking the correct answer subjected to data enhancement as an error answer. In an embodiment, the data enhancement includes at least one of a synonym replacement, a stop word removal, an insert word, or a delete word.
In addition, data enhancement can also include a mode of disordering sentence sequence or randomly Mask dropping some words in the original text.
S103, determining a first score of the sample question-answer pair according to the question, the wrong answer and the correct answer.
Wherein the first score represents the ranking of the correct answer among all answers to the question. After the wrong answers are obtained, sentence vectors corresponding to the questions, the wrong answers and the correct answers respectively can be obtained through the neural network, and then the first scores of the sample question-answer pairs are determined according to the respective sentence vectors.
In an embodiment, please refer to fig. 2, fig. 2 is a schematic flowchart illustrating steps of determining a first score of a sample question-answer pair according to an embodiment of the present application. As shown in fig. 2, the step of determining the first score of the sample question-answer pair according to the question, the wrong answer and the correct answer may include: s1031, calculating a first similarity between the question and the correct answer and a second similarity between the question and the wrong answer; s1032, determining a first score of the sample question-answer pair according to the first similarity and the second similarity.
The first score may be determined according to a first similarity between the question and the correct answer in the sample question-answer pair and a second similarity between the question and the incorrect answer in the sample question-answer pair, after the first similarity and the second similarity are calculated respectively, the correct answer and the incorrect answer are ranked, and the first score is determined according to the rank of the correct answer in all the answers.
In one embodiment, the step of calculating a first similarity between the question and the correct answer and a second similarity between the question and the wrong answer may include: grouping the questions, the correct answers and the wrong answers to obtain a plurality of sample groups; training a first neural network for multiple times according to the multiple sample groups to obtain a first sentence vector corresponding to the problem predicted by the first neural network, a second sentence vector corresponding to the correct answer and a third sentence vector corresponding to the wrong answer; calculating cosine similarity between the first sentence vector and the second sentence vector to obtain first similarity between the question and the correct answer; and calculating cosine similarity between the first sentence vector and the third sentence vector to obtain a second similarity between the question and the wrong answer.
Since the length of the answer in the question-answer pair is not too long, the first neural network may be a bidirectional long-short term memory network, and certainly, other types of networks may be used as the first neural network to perform the feature extraction of the sentence vector according to the data situation.
For example, as shown in fig. 3, which is a schematic structural diagram of a first neural network, the first neural network may include an input layer, a first hidden layer, a second hidden layer, and an output layer. The first hidden layer is used for forward calculation, the second hidden layer is used for reverse calculation, and the output of the first hidden layer and the output of the second hidden layer are jointly input to the output layer for maximum pooling, so that sentence vectors can be obtained.
In particular implementations, the first neural network may be trained using sample question-and-answer pairs and false answers resulting from data enhancement. When the first neural network is trained by using the sample question-answer pairs and the wrong answers, the sample question-answer pairs and the wrong answers can be equally divided into a plurality of sample groups according to questions in the sample question-answer pairs, one of the sample groups is used as a test set, and other sample groups are used as training sets to train the first neural network.
For example, if there are 1000 sample question-answer pairs, each sample question-answer pair includes a question and a correct answer, then there are 1000 correct answers and 1000 questions, and the data enhancement is performed on the correct answers to obtain 1000 wrong answers. Then, when the time sharing is performed according to the questions, the data may be divided into 5 sample groups of ABCDE, where each sample group includes 200 questions and correct answers and incorrect answers corresponding to the questions, and then the training process of the first neural network may be:
firstly, four ABCD sample groups are used as a training set to train a first neural network for the first time, then the first neural network obtained by the first training is used for predicting an E sample group, and a first sentence vector corresponding to a question in the E sample group, a second sentence vector corresponding to a correct answer and a third sentence vector corresponding to a wrong answer are obtained.
And then, taking the four ABCE sample groups as a training set to carry out second training on the first neural network, and then predicting the D sample group by using the first neural network obtained by the second training to obtain a first sentence vector corresponding to the problem in the D sample group, a second sentence vector corresponding to the correct answer and a third sentence vector corresponding to the wrong answer.
And then, taking four ABDE sample groups as a training set to train the first neural network for the third time, and predicting the C sample group by using the first neural network obtained by the third training to obtain a first sentence vector corresponding to the question in the C sample group, a second sentence vector corresponding to the correct answer and a third sentence vector corresponding to the wrong answer.
And then, taking the four ACDE sample groups as a training set to train the first neural network for the fourth time, and then predicting the B sample group by using the first neural network obtained by the fourth training to obtain a first sentence vector corresponding to the question in the B sample group, a second sentence vector corresponding to the correct answer and a third sentence vector corresponding to the wrong answer.
And then, taking the four sample groups of the BCDE as a training set to train the first neural network for the fifth time, and predicting the sample group A by using the first neural network obtained by the fifth training to obtain a first sentence vector corresponding to the question in the sample group A, a second sentence vector corresponding to the correct answer and a third sentence vector corresponding to the wrong answer.
And splicing the predicted values of each sample group predicted by training the first neural network each time, thereby obtaining first sentence vectors corresponding to all the problems, second sentence vectors corresponding to correct answers and third sentence vectors corresponding to wrong answers.
The first similarity and the second similarity are then calculated based on the first sentence vector, the second sentence vector, and the third sentence vector. In a specific implementation process, the first similarity and the second similarity may be calculated by a cosine function, that is, the first similarity and the second similarity may be calculated by a cosine value between sentence vectors. The larger the value of the first similarity, the more similar the question is to the correct answer, and likewise, the larger the value of the second similarity, the more similar the question is to the wrong answer.
For example, q is a first sentence vector representation of the question, a + is a second sentence vector representation of the correct answer, and a-is a third sentence vector representation of the wrong answer, then the first similarity may be represented as cosine (q, a +), and the second similarity may be represented as cosine (q, a-).
In an embodiment, the step of determining the first score of the sample question-answer pair according to the first similarity and the second similarity may include:
determining a first score of the sample question-answer pair according to the first similarity and the second similarity according to a preset formula; the preset formula comprises:
L＝max{0,M-C0+C1}
wherein L represents a first fraction, M represents a constant, C0Denotes a first degree of similarity, C1Indicating a second degree of similarity.
In calculating the first similarity and the second similarity, the similarity between the question and the correct answer and the similarity between the question and the wrong answer may be calculated by the sentence vector.
The greater the value of the first similarity, the more similar the question is to the correct answer, and the greater the value of the second similarity, the more similar the question is to the wrong answer, so the more similar the question is to the correct answer, the lower the first score and the higher the ranking of the correct answer among all answers to the question, whereas the more similar the question is to the wrong answer, the higher the first score and the higher the ranking of the wrong answer among all answers to the question, the higher the ranking of the correct answer among all answers to the question is.
And S104, performing word order scoring on the sample question-answer pairs to obtain second scores of the sample question-answer pairs.
Wherein the second score represents the degree of plausibility of the word order of the questions and/or correct answers in the sample question-answer pair. When the sample question-answer pairs are subjected to language order evaluation, the question and/or correct answer in the sample question-answer pairs can be subjected to language order rationalization prediction according to the neural network, so that a second score is obtained.
In an embodiment, the step of scoring the sample question-answer pairs includes: performing word order scoring on the questions of the sample question-answer pairs; and/or performing word order scoring on correct answers of the sample question-answer pairs.
In the specific implementation process, the questions in the sample question-answer pairs can be subjected to word order scoring, and whether the word order of the questions is reasonable or not and whether the questions use words correctly or not is judged; the correct answers obtained in the sample question-answer pairs can be scored according to the word order, and whether the word order of the correct answers is reasonable or not and whether the correct answers use words or not can be judged; or the questions in the sample question-answer pairs can be subjected to word order scoring, and the correct answers in the sample question-answer pairs can be subjected to word order scoring.
It can be understood that when the neural network is used to perform the word order scoring on the sample question-answer pair, different neural networks may be trained to perform the word order scoring on the questions and correct answers in the sample question-answer pair.
In an embodiment, the step of scoring the sample question-answer pairs may include: grouping the sample question-answer pairs to obtain a plurality of sample groups; and training a second neural network for multiple times according to the plurality of sample groups to obtain the word order score of the sample question answer predicted by the first neural network.
In particular implementations, the second neural network may be trained using sample question-and-answer. When the second neural network is trained by using the sample question-answer pairs, the sample question-answer pairs can be equally divided into a plurality of sample groups, one of the sample groups is used as a test set, and the other sample groups are used as training sets to train the second neural network.
Taking training a second neural network capable of performing word order scoring on questions in question-answer pairs as an example, if the sample question-answer pairs have 1000 pairs in total, the sample question-answer pairs can be equally divided into 5 sample groups of ABCDE, each sample group includes 200 questions, and then the training process of the second neural network may be:
firstly, four sample groups of ABCD are used as a training set to train a second neural network for the first time, then the second neural network obtained by the training for the first time is used for predicting the E sample group, and the word order score corresponding to the problem in the E sample group is obtained.
And then, taking the four ABCE sample groups as a training set to carry out secondary training on the second neural network, and then predicting the D sample group by using the second neural network obtained by the secondary training to obtain the word order score corresponding to the problem in the D sample group.
And then, taking the four ABDE sample groups as a training set to carry out second training on the second neural network, and then predicting the C sample group by using the second neural network obtained by the second training to obtain the word order score corresponding to the problem in the C sample group.
And secondly, training the second neural network by taking the four ACDE sample groups as a training set, and predicting the B sample group by using the second neural network obtained by the second training to obtain the word order score corresponding to the problem in the B sample group.
And then, taking the four BCDE sample groups as a training set to carry out second training on the second neural network, and then predicting the A sample group by using the second neural network obtained by the second training to obtain the word order score corresponding to the problem in the A sample group.
And splicing the predicted values of the sample groups predicted by training the second neural network each time, thereby obtaining the word order scores corresponding to all the problems.
In a specific implementation process, part of the questions in the sample question-answer pair may also be subjected to disorder processing, and the questions in the training question-answer pair and the questions subjected to disorder processing are respectively input into the second neural network as positive and negative samples, so as to train the second neural network.
Similarly, when performing the word order scoring on the correct answer in the sample question-answer pair, the training process described above may also be used to train the second neural network, so as to obtain the word order scoring of the correct answer predicted by the second neural network.
And S105, carrying out iterative training on the classification model according to the first score and the second score, and taking the trained classification model as a question-answer pair evaluation model.
And then, the first score and the second score of the sample question-answer pair are jointly used as a training set and then input into a classification model for training, and the classification model after training is used as a pre-trained question-answer pair evaluation model. The question-answer pair evaluation model is a two-classification model used for evaluating whether answers are in compliance or not, and the quality of the question-answer pair is evaluated through the output of the question-answer pair evaluation model.
Please refer to fig. 4, which is a schematic diagram of a training architecture of a question-answer pair score model according to an embodiment of the present application. As shown in fig. 4, the training process of the question-answer scoring model in the present application includes:
and performing data enhancement on correct answers in the sample question-answer pairs to obtain wrong answers. And then training the first neural network based on the question, the correct answer and the wrong answer, and obtaining a prediction result output by the first neural network, namely a first score. And training the second neural network based on the sample question-answer, and obtaining a prediction result output by the second neural network, namely a second score.
And then, the obtained first score and the second score are jointly used as the input of the last classification model to train the classification model, so that when the training of the classification model is finished, the trained classification model is used as a pre-trained question-answer pair evaluation model.
The training mode enables the question-answer pair scoring model to be classified into two categories by combining the first score and the second score during training, and considers the sequence of correct answers in all answers of the question in the sample question-answer pair and the word sequence rationalization degree of the sample question-answer pair, so that the comprehensive degree of the obtained question-answer pair scoring model during scoring of the question-answer pair is improved, and the accuracy of scoring results is improved.
The method for training the scoring model of the question-answer pair provided in the above embodiment includes obtaining a sample question-answer pair, where the sample question-answer pair includes a question and a correct answer corresponding to the question, then performing data enhancement on the correct answer to obtain a wrong answer, then determining a first score of the sample question-answer pair according to the question, the wrong answer, and the correct answer, and performing word order scoring on the sample question-answer pair to obtain a second score of the sample question-answer pair, and finally performing iterative training on the classification model according to the first score and the second score, and taking the trained classification model as the scoring model of the question-answer pair. When the question-answer pair evaluation model is trained, the sequence of correct answers in all answers of the question in the sample question-answer pair and the rationalization degree of the word sequence of the sample question-answer pair are combined, so that the comprehensive degree of the obtained question-answer pair evaluation model when the question-answer pair is evaluated is improved, and the accuracy of the evaluation result is improved.
Referring to fig. 5, fig. 5 is a schematic block diagram of a device for training a question-answer pair score model according to an embodiment of the present application, where the device is used to perform the method for training the question-answer pair score model. Wherein, the training device of the question-answer pair scoring model can be configured in a server or a terminal.
The server may be an independent server or a server cluster. The terminal can be an electronic device such as a mobile phone, a tablet computer, a notebook computer, a desktop computer, a personal digital assistant and a wearable device.
As shown in fig. 5, the training apparatus 200 for the question-answer pair score model includes: the system comprises a sample acquisition module 201, a data enhancement module 202, a first sub-module 203, a second sub-module 204 and a model training module 205.
The sample obtaining module 201 is configured to obtain a sample question-answer pair, where the sample question-answer pair includes a question and a correct answer corresponding to the question.
And the data enhancement module 202 is configured to perform data enhancement on the correct answer to obtain an incorrect answer.
A first score module 203, configured to determine a first score of the sample question-answer pair according to the question, the wrong answer, and the correct answer.
In one embodiment, the first score module 203 includes a similarity calculation sub-module 2031 and a score calculation sub-module 2032.
The similarity calculation submodule 2031 is configured to calculate a first similarity between the question and the correct answer and a second similarity between the question and the wrong answer; the score calculating submodule 2032 is configured to determine a first score of the sample question-answer pair according to the first similarity and the second similarity.
And the second score module 204 is configured to perform word order scoring on the sample question-answer pairs to obtain a second score of the sample question-answer pairs.
And the model training module 205 is configured to perform iterative training on a classification model according to the first score and the second score, and use the trained classification model as a question-answer pair scoring model.
In an embodiment, the similarity calculation sub-module may include a sample grouping sub-module, a vector prediction sub-module, and a cosine calculation sub-module.
The sample grouping submodule is used for grouping the questions, the correct answers and the wrong answers to obtain a plurality of sample groups; the vector prediction sub-module is used for training a first neural network for multiple times according to the plurality of sample groups to obtain a first sentence vector corresponding to the problem predicted by the first neural network, a second sentence vector corresponding to the correct answer and a third sentence vector corresponding to the wrong answer; the cosine calculation submodule is used for calculating cosine similarity between the first sentence vector and the second sentence vector to obtain first similarity between the question and the correct answer; and calculating cosine similarity between the first sentence vector and the third sentence vector to obtain a second similarity between the question and the wrong answer.
In one embodiment, the score computation submodule comprises a preset computation submodule.
The preset calculation submodule is used for determining a first score of the sample question-answer pair according to the first similarity and the second similarity according to a preset formula;
the preset formula comprises:
L＝max{0,M-C0+C1}
wherein L represents a first fraction, M represents a constant, C0Denotes a first degree of similarity, C1Indicating a second degree of similarity.
In one embodiment, the second scoring module includes a question scoring sub-module and/or an answer scoring sub-module.
The question scoring submodule is used for scoring the question of the sample question-answer pair in the language order; and the answer scoring submodule is used for carrying out language order scoring on the correct answers of the sample question-answer pairs.
In one embodiment, the second scoring module includes a sample scoring submodule and a scoring predictor submodule.
The sample grouping submodule is used for grouping the sample question-answer pairs to obtain a plurality of sample groups; and the scoring prediction sub-module is used for training a second neural network for multiple times according to the plurality of sample groups so as to obtain the word order scoring of the sample question and answer predicted by the first neural network.
It should be noted that, as will be clearly understood by those skilled in the art, for convenience and brevity of description, the above-described specific working processes of the training device for the question-answer pair scoring model and each module may refer to the corresponding processes in the embodiment of the training method for the question-answer pair scoring model, and are not described herein again.
The above-mentioned training device of the question-answer pair score model may be implemented in the form of a computer program, which may be run on a computer device as shown in fig. 6.
Referring to fig. 6, fig. 6 is a schematic block diagram of a computer device according to an embodiment of the present disclosure. The computer device may be a server or a terminal.
Referring to fig. 6, the computer device includes a processor, a memory, and a network interface connected through a system bus, wherein the memory may include a nonvolatile storage medium and an internal memory.
The non-volatile storage medium may store an operating system and a computer program. The computer program includes program instructions that, when executed, cause a processor to perform any of the methods for training the question-answer pair score model.
The processor is used to provide computing and control capabilities to support the operation of the entire computer device.
The internal memory provides an environment for the execution of a computer program on a non-volatile storage medium, which when executed by the processor, causes the processor to perform any one of the methods for training the question-answer pair score model.
The network interface is used for network communication, such as sending assigned tasks and the like. Those skilled in the art will appreciate that the architecture shown in fig. 6 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects apply, as particular computing devices may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.
It should be understood that the Processor may be a Central Processing Unit (CPU), and the Processor may be other general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic, discrete hardware components, etc. Wherein a general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
Wherein, in one embodiment, the processor is configured to execute a computer program stored in the memory to implement the steps of:
obtaining a sample question-answer pair, wherein the sample question-answer pair comprises a question and a correct answer corresponding to the question;
performing data enhancement on the correct answer to obtain an incorrect answer;
determining a first score of the sample question-answer pair according to the question, the wrong answer and the correct answer;
performing word order scoring on the sample question-answer pairs to obtain second scores of the sample question-answer pairs;
and performing iterative training on a classification model according to the first score and the second score, and taking the trained classification model as a question-answer pair evaluation model.
In one embodiment, the processor, in implementing the determining the first score of the sample question-answer pair from the question, the wrong answer, and the correct answer, is configured to implement:
calculating a first similarity of the question and the correct answer and a second similarity of the question and the wrong answer;
and determining a first score of the sample question-answer pair according to the first similarity and the second similarity.
In one embodiment, the processor, in performing the calculating a first similarity of the question to the correct answer and a second similarity of the question to the wrong answer, is configured to perform:
grouping the questions, the correct answers and the wrong answers to obtain a plurality of sample groups;
training a first neural network for multiple times according to the multiple sample groups to obtain a first sentence vector corresponding to the problem predicted by the first neural network, a second sentence vector corresponding to the correct answer and a third sentence vector corresponding to the wrong answer;
calculating cosine similarity between the first sentence vector and the second sentence vector to obtain first similarity between the question and the correct answer; and
and calculating cosine similarity between the first sentence vector and the third sentence vector to obtain a second similarity between the question and the wrong answer.
In one embodiment, the processor, in implementing the determining the first score of the sample question-and-answer pair according to the first similarity and the second similarity, is configured to implement:
determining a first score of the sample question-answer pair according to the first similarity and the second similarity according to a preset formula;
the preset formula comprises:
L＝max{0,M-C0+C1}
wherein L represents a first fraction, M represents a constant, C0Denotes a first degree of similarity, C1Indicating a second degree of similarity.
In one embodiment, the processor, when implementing the lexical ranking of the sample question-answer pairs, is configured to implement:
performing word order scoring on the questions of the sample question-answer pair; and/or
And carrying out language order scoring on correct answers of the sample question-answer pairs.
In one embodiment, the processor, when implementing the lexical ranking of the sample question-answer pairs, is configured to implement:
grouping the sample question-answer pairs to obtain a plurality of sample groups;
and training a second neural network for multiple times according to the plurality of sample groups to obtain the word order score of the sample question answer predicted by the first neural network.
In one embodiment, the data enhancement includes at least one of a synonym replacement, a stop word removal, an insert word, or a delete word.
In an embodiment of the present application, a computer-readable storage medium is further provided, where a computer program is stored in the computer-readable storage medium, where the computer program includes program instructions, and the processor executes the program instructions to implement any one of the methods for training a question-answer pair score model provided in the embodiments of the present application.
The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiment, for example, a hard disk or a memory of the computer device. The computer readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like provided on the computer device.
While the invention has been described with reference to specific embodiments, the scope of the invention is not limited thereto, and those skilled in the art can easily conceive various equivalent modifications or substitutions within the technical scope of the invention. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.
Claims (10)
1. A method for training a question-answer pair score model, the method comprising:
obtaining a sample question-answer pair, wherein the sample question-answer pair comprises a question and a correct answer corresponding to the question;
performing data enhancement on the correct answer to obtain an incorrect answer;
determining a first score of the sample question-answer pair according to the question, the wrong answer and the correct answer;
performing word order scoring on the sample question-answer pairs to obtain second scores of the sample question-answer pairs;
and performing iterative training on a classification model according to the first score and the second score, and taking the trained classification model as a question-answer pair evaluation model.
2. The method for training a question-answer pair score model according to claim 1, wherein the determining a first score of the sample question-answer pair according to the question, the wrong answer and the correct answer comprises:
calculating a first similarity of the question and the correct answer and a second similarity of the question and the wrong answer;
and determining a first score of the sample question-answer pair according to the first similarity and the second similarity.
3. The method for training a question-answer pair score model according to claim 2, wherein the calculating a first similarity between the question and the correct answer and a second similarity between the question and the wrong answer comprises:
grouping the questions, the correct answers and the wrong answers to obtain a plurality of sample groups;
training a first neural network for multiple times according to the multiple sample groups to obtain a first sentence vector corresponding to the problem predicted by the first neural network, a second sentence vector corresponding to the correct answer and a third sentence vector corresponding to the wrong answer;
calculating cosine similarity between the first sentence vector and the second sentence vector to obtain first similarity between the question and the correct answer; and
and calculating cosine similarity between the first sentence vector and the third sentence vector to obtain a second similarity between the question and the wrong answer.
4. The method for training the question-answer pair score model according to claim 2, wherein the determining the first score of the sample question-answer pair according to the first similarity and the second similarity comprises:
determining a first score of the sample question-answer pair according to the first similarity and the second similarity according to a preset formula;
the preset formula comprises:
L＝max{0,M-C0+C1}
wherein L represents a first scoreM represents a constant, C0Denotes a first degree of similarity, C1Indicating a second degree of similarity.
5. The method for training the question-answer pair scoring model according to claim 1, wherein the step of scoring the sample question-answer pairs in terms of word order comprises:
performing word order scoring on the questions of the sample question-answer pairs; and/or
And carrying out language order scoring on correct answers of the sample question-answer pairs.
6. The method for training the question-answer pair scoring model according to claim 1, wherein the step of scoring the sample question-answer pairs in terms of word order comprises:
grouping the sample question-answer pairs to obtain a plurality of sample groups;
and training a second neural network for multiple times according to the plurality of sample groups to obtain the word order score of the sample question answer predicted by the first neural network.
7. The method of claim 1, wherein the data enhancement comprises at least one of synonym replacement, stop word removal, inserted word, or deleted word.
8. A device for training a question-answer pair score model, comprising:
the system comprises a sample acquisition module, a sample analysis module and a sample analysis module, wherein the sample acquisition module is used for acquiring a sample question-answer pair, and the sample question-answer pair comprises a question and a correct answer corresponding to the question;
the data enhancement module is used for enhancing the data of the correct answer to obtain an incorrect answer;
a first score module, configured to determine a first score of the sample question-answer pair according to the question, the wrong answer, and the correct answer;
the second score module is used for scoring the word order of the sample question-answer pairs to obtain second scores of the sample question-answer pairs;
and the model training module is used for carrying out iterative training on the classification model according to the first score and the second score and taking the trained classification model as a question-answer pair scoring model.
9. A computer device, wherein the computer device comprises a memory and a processor;
the memory is used for storing a computer program;
the processor, configured to execute the computer program and when executing the computer program, implement the method for training a question-answer pair score model according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program which, when executed by a processor, causes the processor to implement the method of training a question-answer pair score model according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210182951.XA CN114528391A (en) | 2022-02-25 | 2022-02-25 | Method, device and equipment for training question-answer pair scoring model and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210182951.XA CN114528391A (en) | 2022-02-25 | 2022-02-25 | Method, device and equipment for training question-answer pair scoring model and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114528391A true CN114528391A (en) | 2022-05-24 |
Family
ID=81624463
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210182951.XA Pending CN114528391A (en) | 2022-02-25 | 2022-02-25 | Method, device and equipment for training question-answer pair scoring model and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114528391A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115129816A (en) * | 2022-07-04 | 2022-09-30 | 北京百度网讯科技有限公司 | Question-answer matching model training method and device and electronic equipment |
| CN115544236A (en) * | 2022-11-16 | 2022-12-30 | 北京红棉小冰科技有限公司 | Model evaluation method and device and electronic equipment |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112487167A (en) * | 2020-12-02 | 2021-03-12 | 广州虎牙科技有限公司 | Training method of automatic question-answering model, and related device and equipment |
| CN112818106A (en) * | 2021-02-10 | 2021-05-18 | 北京工业大学 | Evaluation method of generating type question and answer |
| CN113821623A (en) * | 2021-09-29 | 2021-12-21 | 平安普惠企业管理有限公司 | Model training method, device, equipment and storage medium |
-
2022
- 2022-02-25 CN CN202210182951.XA patent/CN114528391A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112487167A (en) * | 2020-12-02 | 2021-03-12 | 广州虎牙科技有限公司 | Training method of automatic question-answering model, and related device and equipment |
| CN112818106A (en) * | 2021-02-10 | 2021-05-18 | 北京工业大学 | Evaluation method of generating type question and answer |
| CN113821623A (en) * | 2021-09-29 | 2021-12-21 | 平安普惠企业管理有限公司 | Model training method, device, equipment and storage medium |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115129816A (en) * | 2022-07-04 | 2022-09-30 | 北京百度网讯科技有限公司 | Question-answer matching model training method and device and electronic equipment |
| CN115129816B (en) * | 2022-07-04 | 2024-05-14 | 北京百度网讯科技有限公司 | Question-answer matching model training method and device and electronic equipment |
| CN115544236A (en) * | 2022-11-16 | 2022-12-30 | 北京红棉小冰科技有限公司 | Model evaluation method and device and electronic equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11288444B2 (en) | Optimization techniques for artificial intelligence | |
| CN108804641B (en) | Text similarity calculation method, device, equipment and storage medium | |
| CN110390052B (en) | Search recommendation method, training method, device and equipment of CTR (China train redundancy report) estimation model | |
| CN112287670A (en) | Text error correction method, system, computer device and readable storage medium | |
| US11048875B2 (en) | Skimming data sequences using recurrent neural networks | |
| CN111382255A (en) | Method, apparatus, device and medium for question and answer processing | |
| RU2664481C1 (en) | Method and system of selecting potentially erroneously ranked documents with use of machine training algorithm | |
| CN111563158B (en) | Text ranking method, ranking apparatus, server and computer-readable storage medium | |
| CN114528391A (en) | Method, device and equipment for training question-answer pair scoring model and storage medium | |
| CN111767393A (en) | Text core content extraction method and device | |
| CN112101042A (en) | Text emotion recognition method and device, terminal device and storage medium | |
| CN111782826A (en) | Knowledge graph information processing method, device, equipment and storage medium | |
| CN110990533A (en) | Method and device for determining standard text corresponding to query text | |
| CN112287656B (en) | Text comparison method, device, equipment and storage medium | |
| CN112307048B (en) | Semantic matching model training method, matching method, device, equipment and storage medium | |
| CN103744889A (en) | Method and device for clustering problems | |
| CN112380421A (en) | Resume searching method and device, electronic equipment and computer storage medium | |
| CN114021718A (en) | Model behavior interpretability method, system, medium, and apparatus | |
| CN117573985B (en) | Information pushing method and system applied to intelligent online education system | |
| CN113239697B (en) | Entity recognition model training method and device, computer equipment and storage medium | |
| CN110852071A (en) | Knowledge point detection method, device, equipment and readable storage medium | |
| CN112541069A (en) | Text matching method, system, terminal and storage medium combined with keywords | |
| CN112560463A (en) | Text multi-labeling method, device, equipment and storage medium | |
| CN116484829A (en) | Method and apparatus for information processing | |
| CN110895924B (en) | Method and device for reading document content aloud, electronic equipment and readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |