CN111241281A - Text similarity-based public opinion topic tracking method - Google Patents

Abstract

The invention discloses a public opinion topic tracking method based on text similarity, which is based on a doc2vec model evolved by a word2vec model, can well obtain the expression of vectors of sentences, paragraphs or documents, and is very suitable for processing public opinion topics, but the model ignores the time characteristic of the public opinion topics. Compared with the prior art, the method has the advantages that the data dimension is relatively low on the expression of the vectors of sentences, paragraphs or documents, the time complexity is reduced, the expression of semantics is relatively more accurate, the text similarity calculation accuracy is improved, the timeliness of topics is ensured by adding the time characteristic on the basis of the existing model, and the method has a good effect on topic tracking through experimental tests.

Description

A public opinion topic tracking method based on text similarity

technical field

The invention belongs to the field of topic tracking in natural language processing, and particularly relates to a research innovation of a topic tracking method based on text similarity.

Background technique

Topic tracking refers to giving one or more reports on a topic and linking the related reports entered with the topic. According to the tracking requirements, the steps can be divided into the following two steps: first, a set of sample reports is given, the topic model is obtained through model training, and then similar or the same topic reports are found in the subsequent reports. The ever-changing topics are gathered and organized to help users discover the relationship between topics, and understand the information of various aspects of public opinion topics as a whole and the connections between topics. With the development and progress of related technologies, topic tracking research goals and processing objects are not limited to media information flow, but are more and more widely used in various fields related to information. The present invention tracks public opinion topics by means of text similarity calculation. At present, in terms of text similarity, there are two mainstream text similarity calculation methods, a string-based method and a corpus-based method.

1 String based

The string-based method starts from the string matching degree, and uses the co-occurrence and repetition degree of strings as the similarity standard to measure. According to the different calculation granularity, this method can be divided into character-based method and word-based method; currently Similarity algorithms considered purely from the perspective of character or word composition include edit distance, Hamming distance, Dice coefficient, cosine similarity, etc. to calculate text similarity. On this basis, the methods of adding character order include Jaro-Winkler and longest. Common string method; based on the above two methods, a set idea is adopted, that is, the string is regarded as a set composed of words, and the co-occurrence of words is calculated by the intersection of the sets. The current main methods are N-gram and Jaccard. .

2 Corpus-based

The corpus-based method uses the information obtained from the corpus to calculate the text similarity, and the corpus-based method can be divided into: bag-of-words model and neural network-based model, and the two methods use the set of documents to be compared as the similarity. Corpus.

1) Based on bag-of-words model

The bag-of-words model is based on the distribution hypothesis, that is, the contexts where words are located are similar, and the semantics are similar. The basic idea of the bag-of-words model is to express the document as a series of words regardless of the order in which the words appear in the document. The combination. According to different semantics, methods based on the bag-of-words model currently mainly include Vector Space Model (VSM), Probabilistic Latent Semantic Analysis (PLSA), Latent Semantic Analysis (LSA) and Latent Semantic Analysis (LSA) Mainstream model methods such as LatentDirichlet Allocation (LDA).

2) Based on neural network model

Generating word vectors based on neural network models to calculate text similarity is a hot area of research in this field in recent years. In this process, many word vector models such as Word2Vec and Glove have been proposed. The essence of word vector is to train a low-dimensional real number vector from unlabeled unstructured text. This expression makes similar words closer in distance, and can also better solve the bag-of-words model due to the independence of words. It brings about the curse of dimensionality and the problem of insufficient semantics.

3 doc2vec algorithm

The doc2vec model algorithm was evolved from Google in 2014 based on the word2vec model. It is an unsupervised algorithm model. Its essence is to learn a representation of the document and obtain the vector representation of the sentence/paragraph/document. It is an extension of word2vec. According to the learned vector, the similarity between sentences/paragraphs/documents can be found by calculating the distance, which can be used for unlabeled text clustering. For labeled data, supervised learning can also be used for text classification. In the process, the paragraph id is added compared to the word2vec model, that is, each sentence in the training corpus has a unique id, and the paragraph id is the same as the ordinary word. First, it is mapped into a vector, that is, the dimension of the paragraph vector and the word vector. Although the same, but from two different vector spaces. In the subsequent calculation, the paragraph vector and the word vector are accumulated or connected as the input of the output layer softmax. During the training process of a sentence or document, the paragraph id remains unchanged and shares the same paragraph vector, which is equivalent to using the semantics of the entire sentence every time the probability of a word is predicted. The doc2vec model framework is shown in Figure 1, and its task is to predict other words in the context given the context. Among them, each word is mapped into the vector space, and the word vector concatenation or summation of the context is used as a feature to predict the next word in the sentence. Its objective function is as formula (1.1), and the task of prediction is a multi- In the classification task, the last layer of the classifier uses softmax, and its calculation formula is as (1.2); in the prediction task, each word is regarded as a predicted task, and each word is regarded as a category, and the calculation formula is as follows (1.3), where U and b are both parameters, and h concatenates or averages w _tk , . . . , w _t+k .

y=b+Uh(ω _tk , . . . , w _t+k ; W) (1.3)

SUMMARY OF THE INVENTION

The invention proposes a public opinion topic tracking method based on text similarity, which is based on the doc2vec model evolved from the word2vec model proposed by Google in 2014. The model is an unsupervised algorithm, which can well obtain sentences, paragraphs or The expression of the vector of the document is very suitable for the processing of public opinion topics, but the model ignores the time characteristics of the public opinion topics. The present invention adds the time characteristics as an important feature to the algorithm to ensure the timeliness of the topic. At the same time, in order to reduce the length of text data To affect the final result, the similarity calculation method of selected text is adopted. The experimental results show that the above method has a good effect on the tracking of public opinion topics.

Step 1 Data preprocessing

1) The text data uses crawler technology. The data acquisition address is Sina News and People’s Daily. The content is to crawl hot public opinion topics and news related to the topic. The main purpose of this crawling method is to obtain high-quality public opinion topics. corpus.

2) Chinese word segmentation is the process of dividing a continuous sequence of words into individual words according to the understanding of Chinese, using jieba word segmentation tool to segment the text, the result after the word segmentation is shown in Figure 3, the sentence has been divided into single word.

3) Normal text or a sentence in Chinese will contain special characters such as commas, commas or periods. After the word segmentation is completed, these special characters will be retained as shown in Figure 2, and these special characters will affect the text similarity calculation. The speed and accuracy of the calculation, so these characters need to be filtered out, in addition to these special characters, such as and, not only, , , etc. also have a similar effect on the calculation of text similarity, and these words have little effect on the final calculation result. , so these words are filtered out in the data preprocessing stage.

Step 2 Text Similarity Calculation

Since the text data is captured from the Internet, the length of the data may be very short after step 1. In order to reduce or eliminate the impact of this short text on the final similarity calculation result, two methods are used to calculate the text similarity. That is, the text with the text length less than 150 adopts the sentence-level calculation method, otherwise, the document-level calculation method is adopted, and the time characteristic is added to the calculation during the calculation process, and the time comparison is performed first. If the number of news is less than 100, the similarity is considered to be low. If the time difference is greater than 30 days and the similarity is greater than or equal to 0.70, the similarity is considered to be high. After the final weighting process, the corresponding text similarity is obtained.

Step 3 Topic Tracking Results

According to the vector expression method of the corresponding text from step 2, in order to better display the calculation result, the present invention uses the k-means algorithm to display the text data in an image, and the result is shown in FIG. 7 .

Compared with the prior art, the present invention has a relatively low data dimension in the expression of the vector of sentences, paragraphs or documents, which reduces the time complexity, the expression of semantics is relatively more accurate, and improves the accuracy of text similarity calculation. The time characteristic is added on the basis of the existing model to ensure the timeliness of the topic, and the present invention has a good effect in topic tracking through experimental tests.

Description of drawings

FIG. 1 is an architecture diagram of the doc2vec model of the present invention.

FIG. 2 is an overall flow chart of the public opinion topic tracking of text similarity according to the present invention.

FIG. 3 is a corpus diagram of the public opinion topic of the present invention.

FIG. 4 is a result diagram after the word segmentation of the present invention is completed.

FIG. 5 is a result diagram after the stop word removal of the present invention is completed.

FIG. 6 is a result diagram after the text similarity calculation of the present invention is completed.

FIG. 7 is a graph of the final topic tracking result of the present invention.

Detailed ways

The embodiments of the invention are described with reference to the accompanying drawings. The public opinion topic tracking of text similarity is mainly divided into the following steps:

Step 1. Text acquisition

The text data uses crawler technology, the data acquisition address is Sina News and Renming.com, and the content is mainly to crawl public opinion topic news and news related to the public opinion topic. The main purpose of this crawling method is to obtain high-quality public opinion topics. corpus.

Step 2, Chinese word segmentation

Chinese word segmentation is the process of dividing a continuous sequence of words into individual words according to the understanding of Chinese. During the implementation process, the jieba word segmentation tool is used to segment the text. The result after the word segmentation is shown in Figure 4. The sentence has been divided into single words.

Step 3. Remove stop words

Normal text or a sentence in Chinese usually contains special characters such as commas, commas or periods. After the word segmentation is completed, these special characters will be retained as shown in Figure 3, and these special characters will affect the calculation of text similarity. speed and precision, so these characters need to be filtered out, in addition to these special characters, such as and, not only, , , etc. also have a similar impact on the calculation of text similarity, and these words have little effect on the final calculation result, So these words are filtered out in the data preprocessing stage.

Step 4. Text similarity calculation

Since the text data is grabbed from the Internet, the length of the data may be very short after steps 1 and 2. In order to reduce or eliminate the impact of this short text on the final similarity calculation result, two methods are used to measure the text similarity. Calculation, that is, the text with text length less than 150 adopts sentence-level calculation method, otherwise, document-level calculation method is adopted, and time characteristics are added to the calculation during the calculation process. First, time comparison is performed. If the time difference is greater than 30 days and the similarity is less than If the number of news with 0.70 is less than 100, the similarity is considered to be low. If the time difference is greater than 30 days and the number of news with a similarity greater than or equal to 0.70 is greater than 100, the similarity is considered to be high. After the final weighting process, the corresponding texts are similar. Spend.

Step 5. Topic tracking results

According to step 4, the vector representation of the corresponding text can be obtained, in order to better display the calculation result, the present invention uses the k-means algorithm to display the text data in an image, and the result is shown in FIG. 7 .

Claims (1)

1. A public opinion topic tracking method based on text similarity is characterized in that: the method comprises the following steps of,

step 1 data preprocessing

1) The text data is obtained by crawling hot public sentiment topics and news related to the topics through a crawler technology, and high-quality public sentiment topic corpora are obtained;

2) the Chinese word segmentation is a process of dividing a continuous word sequence into single words according to the understanding of Chinese, a jieba word segmentation tool is adopted to segment words of a text, and sentences are already divided into the single words;

3) normal text or a sentence in Chinese contains comma, pause or sentence special characters, the special characters are reserved after word segmentation is finished, and the special characters influence the calculation speed and precision when text similarity calculation is carried out, so the characters need to be filtered, except the special characters, the words not only have influence on the calculation of the text similarity, but also do not influence the final calculation result, so the words are filtered in a data preprocessing stage;

step 2 text similarity calculation

Because the text data is the content captured from the internet, the length of the data after the step 1 is possibly very short, two modes are adopted for carrying out text similarity calculation, namely, the text with the text length less than 150 adopts a sentence-level calculation mode, otherwise, a document-level calculation mode is adopted, the time characteristic is added into the calculation in the calculation process, time comparison is firstly carried out, if the time difference is greater than 30 days and the number of the news with the similarity less than 0.70 is less than 100, the similarity is considered to be low, if the time difference is greater than 30 days and the number of the news with the similarity greater than or equal to 0.70 is greater than 100, the similarity is considered to be high, and the corresponding text similarity is obtained through final weighting processing;

step 3 topic tracking results

And (4) displaying the image of the text data by using a k-means algorithm according to the vector expression mode of the corresponding text from the step (2).

Images