CN112966518B - High-quality answer identification method for large-scale online learning platform - Google Patents

Abstract

A high-quality answer identification method for large-scale online learning platforms. The steps are as follows: (1) Feature vector construction: After preprocessing the acquired data set, manually annotate the data set, and then construct a feature vector; (2) ), use the feature vector constructed in step (1) as input and manually annotated labels as output to build a classification model based on XGBOOOST and train it; (3) for a new question and a series of answers and comments, use The textual content of the new question, the textual content of the answer and the textual content of the comment construct three feature vectors and input them into the model trained in step (2), thereby obtaining a series of classification results as the result of identifying high-quality answers; The present invention uses more information from different angles and makes full use of questions, answers and answer comments to solve the problem of identifying high-quality answers. The prediction results also have a certain improvement in multiple evaluation indicators.

Description

A high-quality answer identification method for large-scale online learning platforms

Technical field

The invention relates to the technical field of natural language processing of artificial intelligence, and in particular to a high-quality answer recognition method for large-scale online learning platforms.

Background technique

With the development of Internet technology, online education has been recognized by the public for its advantages such as no time and location restrictions. More and more people use online learning methods to learn, which has led to the rapid development of online education. Although the Q&A community provided by the large-scale online learning platform provides learners with the opportunity to communicate online, due to the large number of learners, teachers are unable to provide personalized, real-time answers to students' questions. Therefore, intelligent Q&A that can simulate teachers online at any time Technology has become one of the research hotspots in online education. How to quickly select the best answers to learners' questions has become an important issue that needs to be solved in the field of intelligent question answering.

High-quality answer identification and answer sorting are essentially to help users obtain high-quality answers and improve the user experience. The difference between the two is that answer sorting generally uses the number of likes as the goal of model learning, and the number of likes can only indicate the quality of the answer to a certain extent. The number of likes will be affected by factors such as the publication time of the answer. Likes The answer with the highest number does not mean it is the best answer. There are mainly the following methods for sorting answers in community Q&A platforms: by content relevance, by answer length, by answer publication time, by high-quality answers, by the number of comments on the answer, and by the number of likes on the answer. At present, large-scale online learning platforms only provide a way to sort answers according to the number of likes and the time when the answers were published, but do not provide the function of identifying high-quality answers. For large-scale online learning platforms, providing intelligent simulated teachers with online Q&A services at any time is an important way to improve user experience, and identifying high-quality answers is an important technology in intelligent Q&A.

At present, there is relatively little research on high-quality answer identification. The most relevant research is on answer ranking. Many researchers have proposed a variety of answer ranking methods, as listed below:

(1) A method for sorting answers on a community question and answer platform (Applicant: University of Science and Technology of China, Application Number: 201810186972.2);

(2) An answer sorting method for question and answer systems (Applicant: Peking University Shenzhen Graduate School, Application Number: 201810284245.X);

(3) Answer quality determination model training method, answer quality determination method and device (Applicant: Guoxin Youyi Data Co., Ltd., application number: 201811285467.X);

(4) Method to automatically identify correct answers in community question and answer forums based on artificial intelligence (Applicant: Beijing University of Posts and Telecommunications, application number: 201911058818.8).

The above related research mainly uses the number of likes of answers as the learning goal of ranking answer quality. The research focuses on using the correlation between questions and answers, the content attributes of answers, and the time attributes of answers to evaluate answer quality. The evaluation ignores the positive impact of the answer's comment text and the emotional polarity of the comment text on the answer quality evaluation.

Contents of the invention

In order to overcome the above-mentioned shortcomings of the prior art, the purpose of the present invention is to provide a high-quality answer recognition method for large-scale online learning platforms, using more information from different angles, and making full use of questions, answers, and answer comments to solve the problem. For questions that identify high-quality answers, the prediction results also improve to a certain extent on multiple evaluation indicators.

In order to achieve the above objects, the present invention is achieved through the following technical solutions.

A high-quality answer identification method for large-scale online learning platforms. The steps are as follows:

(1) Feature vector construction: After preprocessing the acquired data set, manually annotate the data set, and then construct feature vectors from the following three perspectives: semantic correlation features of questions and answers, and all comments of each answer. document vector features and emotional features of comments; feature acquisition from three perspectives is achieved in the following three ways: (1) Obtain sentence vector representations of questions and answers, and then calculate the similarity of the two semantic vectors based on cosine similarity to obtain the question Semantic relevance to the answer; (2) Use the HAN model to represent the answer comments as document vectors; (3) Use transfer learning to extract emotional features of the comments;

(2) Model construction: Taking the feature vector constructed in step (1) as input and manually annotated labels as output, a classification model based on XGBOOOST is constructed and trained;

(3) For a new question and a series of answers and comments, use the text content of the new question, the text content of the answers, and the text content of the comments to construct the three feature vectors described in step (1) and input them into step ((1)). 2) In the trained model, a series of classification results are obtained as the result of identifying high-quality answers.

The specific operations of manual annotation in step (1) are:

Use crawler technology to crawl website information, store and organize information about questions, answers, answer comments and answer likes, clear the data of questions, answers and comments that are empty, integrate the comments of the same answer under the same question, and The acquired data is stored in the form of questions, answers and integrated comments, and the crawled data sets are manually annotated using the following method:

In the above formula, Flag represents the label of the text pair. If the answer is wrong, it is considered a poor answer, and the text pair is marked as '0'. If the answer is correct but incomplete, it is considered an ordinary answer, and the text pair is marked as '1' ', if the answer is correct and complete, it is considered a high-quality answer, and the text pair is marked as '2'. After completing manual annotation, the final data set contains the following content: questions, answers, integrated answer comments, and labels of text pairs;

The semantic correlation feature extraction operation of the questions and answers is:

(1) Use the BERT model to obtain the sentence vectors of questions and answers, input the question and answer texts into the BERT model and generate sentence vectors, and use the output value of the penultimate layer of the pre-training model as the sentence vectors of questions and answers;

(2) Use the cosine similarity method to calculate the similarity between the question and the answer, and measure the similarity between the two vectors by calculating the cosine value of the angle between them.

The document vector feature extraction operation for the answer comments is:

Use the hierarchical attention network HAN to extract features from multiple comments. The HAN model is divided into two parts. One part is to build sentence vectors based on word vectors, and the other part is to build document vectors based on sentence vectors. The content of the comments in the data set is used as the HAN model. Input, the label of the text pair is used as its output for model training, and the output of the penultimate layer of the model is used as the document vector of the review;

The HAN model is a neural network used for document classification. The model has two characteristics: first, it has a hierarchical structure, and the document vector can be constructed by first constructing the representation of the sentence and then aggregating it into a document representation; the second is A two-level attention mechanism is applied at the word and sentence levels, enabling it to enhance the representation of important content when building document representations;

The extraction operation of the emotional features of the answer comments is:

Since the obtained answer comments do not have relevant emotional labels, and the workload of manual annotation is very heavy, part of the data is randomly labeled with emotional labels, and then a pseudo-label strategy in semi-supervised learning is used to solve the problem of insufficient training data. Question: First, use the emotion classification model to train the already labeled data to obtain the optimal model. Use the optimal model to pseudo-label the unlabeled data, and then use all the data for training to improve the model effect. Specifically:

(1) Train on the already labeled comment data, use the BERT model to obtain the sentence vector of the comment, input the comment text into the BERT model, and use the output value of the penultimate layer of the pre-trained model as the sentence vector of the question and answer, using The fully connected network reduces the dimension of the sentence vector, normalizes the reduced sentence vector through softmax, and uses the result for emotion classification. At the same time, a trained emotion classification model is obtained; the emotion classification model consists of an input layer, a pre-trained It consists of BERT model, fully connected network layer and output layer;

(2) Use the sentiment classification model trained in (1) to analyze the unlabeled review text, represent the unlabeled review text as a sentence vector, use the trained model to perform sentiment feature analysis, and obtain the sentiment features of the review; Then combine the original annotated data with the data generated based on the pseudo-labeling strategy, and continue to train the sentiment analysis model to obtain the optimal model.

Advantages of the present invention: This invention is oriented to the identification of high-quality answers on online education platforms. Feature extraction is performed from three perspectives, namely the correlation features between questions and answers, the comment document vector features of the answers, and the emotional features of the answer comments. Compared with other methods, more information from different angles is used, and the prediction results are also improved in multiple evaluation indicators.

Description of drawings

Figure 1 is an implementation flow chart of an embodiment of the present invention.

Figure 2 is a model diagram of the similarity of question answers.

Figure 3 is a model diagram of the HAN model.

Figure 4 shows the answer comment emotional feature extraction model diagram.

Detailed ways

The present invention will be further described in detail below in conjunction with the accompanying drawings and specific implementation modes.

See Figure 1, a high-quality answer identification method for large-scale online learning platforms. The steps are as follows:

(1) Feature vector construction: After preprocessing the acquired data set (including outlier deletion, format processing, etc.), the data set is manually annotated, and then feature vectors are constructed from the following three perspectives: the semantics of questions and answers Relevance features, document vector features of all comments for each answer, and sentiment features of comments; feature acquisition from three perspectives is achieved in the following three ways: (1) Obtain sentence vector representations of questions and answers, and then based on cosine similarity Calculate the similarity of two semantic vectors to obtain the semantic correlation between the question and the answer; (2) Use the HAN model to represent the answer comments as document vectors; (3) Use transfer learning to extract the emotional features of the comments;

(2) Model construction: Taking the feature vector constructed in step (1) as input and manually annotated labels as output, a classification model based on XGBOOOST is constructed and trained;

(3) For a new question and a series of answers and comments, use the text content of the new question, the text content of the answers, and the text content of the comments to construct the three feature vectors described in step (1) and input them into step ((1)). 2) In the trained model, a series of classification results are obtained as the result of identifying high-quality answers.

The specific operations of manual annotation in step (1) are:

Use crawler technology to crawl website information, store and organize information about questions, answers, answer comments and answer likes, clear the data of questions, answers and comments that are empty, and integrate the comments of the same answer under the same question. The acquired data is stored in the form of questions, answers, and consolidated comments. Use the following method to manually annotate the crawled data set:

In the above formula, Flag represents the label of the text pair. If the answer is wrong, it is considered a poor answer, and the text pair is marked as '0'. If the answer is correct but incomplete, it is considered an ordinary answer, and the text pair is marked as '1' ', if the answer is correct and complete, it is considered a high-quality answer, and the text pair is marked as '2'. After manual annotation, the final dataset contains the following: questions, answers, integrated answer comments, and labels for text pairs.

Referring to Figure 2, the semantic correlation feature extraction operation of the questions and answers is:

(1) Use the BERT model to obtain sentence vectors of questions and answers. The traditional word vector sentence vector generation method has a major drawback. The same word will be represented as the same vector when the semantics are different in different contexts, and BERT It is a large-scale pre-trained model that can solve the problem of polysemy. Using BERT and fine-tuning in specific fields will obtain good experimental results. BERT includes two versions, a 12-layer transformer and a 24-layer transformer. This experiment uses a 12-layer model for the experiment. In theory, the output value of each layer of transformer can be used as a sentence vector. The best sentence can be known by referring to the experimental data. The vector should take the penultimate layer because the value of the last layer is too close to the target and the values of the previous layers have not fully learned the semantic information of the sentence. Input the question and answer text into the BERT model and generate sentence vectors. The output value of the penultimate layer of the pre-trained model is used as the sentence vector of the question and answer.

(2) Use the cosine similarity method to calculate the similarity between the question and the answer, and measure the similarity between the two vectors by calculating the cosine value of the angle between them.

Referring to Figure 3, the document vector feature extraction operation of the answer comments is:

Generally, an answer will have multiple comments. Regarding how to extract multiple comments, existing work is divided into the following two types: one is to splice multiple comments into a longer document, and then perform feature extraction on the document; The other is to model each review individually and then aggregate the modeled features. In the present invention, there is no need to distinguish between single comments, so there is no need to distinguish them. Therefore, the present invention adopts the first method to splice multiple comments into documents, and then uses the method of document vector feature extraction to extract them. Processing, specifically:

Use the hierarchical attention network HAN to extract features from multiple comments. The HAN model is divided into two parts. One part is to build sentence vectors based on word vectors, and the other part is to build document vectors based on sentence vectors. The content of the comments in the data set is used as the HAN model. Input, the label of the text pair is used as its output for model training, and the output of the penultimate layer of the model is used as the document vector of the review;

The HAN model is a neural network used for document classification. The model has two characteristics: first, it has a hierarchical structure, and the document vector can be constructed by first constructing the representation of the sentence and then aggregating it into a document representation; the second is A two-level attention mechanism is applied at the word and sentence levels, enabling it to enhance the representation of important content when building document representations.

Referring to Figure 4, the extraction operation of the emotional features of the answer comments is:

Since the obtained answer comments do not have relevant emotional labels, and the workload of manual annotation is very heavy, part of the data is randomly labeled with emotional labels, and then a pseudo-label strategy in semi-supervised learning is used to solve the problem of insufficient training data. Question: First, use the emotion classification model to train the already labeled data to obtain the optimal model. Use the optimal model to pseudo-label the unlabeled data, and then use all the data for training to improve the model effect. Specifically:

(1) Train on the already labeled comment corpus, use the BERT model to obtain the sentence vector of the comment, input the comment text into the BERT model, use the output value of the penultimate layer of the pre-trained model as the sentence vector of the question and answer, use The fully connected network reduces the dimension of the sentence vector, normalizes the reduced sentence vector through softmax, and uses the result for emotion classification. At the same time, a trained emotion classification model is obtained; the emotion classification model consists of an input layer, a pre-trained It consists of BERT model, fully connected network layer and output layer;

(2) Use the sentiment classification model trained in (1) to analyze the unlabeled comment text, represent the unlabeled comment text as a sentence vector, use the trained model to perform sentiment feature analysis, and obtain the sentiment features of the review.

In summary, based on the three feature extraction methods, the final feature vector format obtained is [similarity of question answers, document vector of comments, and emotional characteristics of comments].

Claims (5)

1. A high-quality answer identification method for a large-scale online learning platform is characterized by comprising the following steps:

construction of feature vectors: after preprocessing the acquired data set, manually labeling the data set, and then constructing the following three-angle feature vectors: semantic relevance features of questions and answers, document vector features of all comments of each answer, and emotion features of comments; the three angles of feature acquisition is realized in three ways: (1) Sentence vector representations of the questions and the answers are obtained, and then similarity of two semantic vectors is calculated based on cosine similarity, so that semantic relevance of the questions and the answers is obtained; (2) Using a HAN model to represent the document vector of the answer comment; (3) Extracting emotion characteristics of the comments by using transfer learning;

and (II) constructing a model: taking the feature vector constructed in the step (one) as input, taking the artificially marked label as output, constructing a XGBOOST-based classification model and training;

and thirdly, for a new question and a series of answers and comments thereof, constructing three feature vectors described in the first step by using the text content of the new question, the text content of the answer and the text content of the comments, and inputting the three feature vectors into the trained model in the second step so as to obtain a series of classification results as a result of identifying high-quality answers.

2. The method for recognizing high-quality answers to a large-scale online learning platform of claim 1, wherein,

the manual labeling specific operation in the step (one) is as follows:

crawling website information by using a crawler technology, storing and sorting the questions, answers, answer comments and answer point approval information, clearing the data of the questions, the answers and the comments which are empty, integrating the comments of the same answer under the same question, and storing the acquired data in the form of the questions, the answers and the integrated comments; manually labeling the crawled data set by using the following method:

in the above formula, flag represents the label of the text pair, if the answer is wrong, it is considered to be a poor answer, the text pair is marked as '0', if the answer is correct but imperfect, it is considered to be a normal answer, the text pair is marked as '1', if the answer is correct and perfect, it is considered to be a good answer, the text pair is marked as '2', and after the manual marking is completed, the final data set contains the following contents: questions, answers, integrated answer comments, and labels for text pairs.

3. The method for recognizing high-quality answers to a large-scale online learning platform of claim 1, wherein,

the semantic relevance feature extraction operation of the questions and answers is as follows:

(1) The method comprises the steps of obtaining sentence vectors of questions and answers by using a BERT model, inputting the texts of the questions and the answers into the BERT model, generating sentence vectors, and taking output values of a penultimate layer of the pre-training model as the sentence vectors of the questions and the answers;

(2) The cosine similarity method is used for calculating the similarity between the questions and the answers, and the cosine value of the included angle of the two vectors is calculated to measure the similarity between the questions and the answers.

4. The method for recognizing high-quality answers to a large-scale online learning platform of claim 1, wherein,

the document vector feature extraction operation of the answer comments comprises the following steps:

extracting characteristics of a plurality of comments by using a hierarchical attention network HAN, wherein the HAN model is divided into two parts, one part is used for constructing sentence vectors according to word vectors, the other part is used for constructing document vectors according to sentence vectors, comment contents in a data set are used as input of the HAN model, labels of text pairs are used as output of the HAN model for model training, and the penultimate layer of the model is used as the document vectors of the comments;

the HAN model is a neural network for document classification, and has two features: firstly, with a hierarchical structure, a document vector can be constructed by first constructing a representation of a sentence and then aggregating it into a document representation; secondly, two levels of attention mechanisms are applied at the word and sentence level, enabling it to strengthen the representation of important content when building a document representation.

5. The method for recognizing high-quality answers to a large-scale online learning platform of claim 1, wherein,

the extraction operation of the emotion characteristics of the answer comments comprises the following steps:

because the obtained answer comment content does not have related emotion labels, and the workload of manual labeling is very large, part of data is randomly labeled with emotion labels, and then a pseudo-label strategy in semi-supervised learning is adopted to solve the problem of insufficient training data: firstly, training the marked data by using an emotion classification model to obtain an optimal model, marking unmarked data by using the optimal model to carry out pseudo-tag marking, and then training all the data to improve the model effect, wherein the method specifically comprises the following steps:

(1) Training on the marked comment data, acquiring a comment sentence vector by using a BERT model, inputting a comment text into the BERT model, taking an output value of a penultimate layer of the pre-training model as a sentence vector of a question and an answer, reducing the dimension of the sentence vector by using a fully connected network, normalizing the sentence vector after the dimension reduction by using softmax, and using the result for emotion classification, thereby obtaining a trained emotion classification model; the emotion classification model consists of an input layer, a pre-trained BERT model, a fully-connected network layer and an output layer;

(2) Analyzing unlabeled comment texts by using the trained emotion classification model in the step (1), expressing the unlabeled comment texts into sentence vectors, and performing emotion feature analysis by using the trained model to obtain the emotion features of comments; and combining the original data with the existing labels and the data generated based on the pseudo tag strategy, and continuing training the emotion analysis model to obtain an optimal model.