WO2019227710A1

WO2019227710A1 - Network public opinion analysis method and apparatus, and computer-readable storage medium

Info

Publication number: WO2019227710A1
Application number: PCT/CN2018/102116
Authority: WO
Inventors: 吴壮伟
Original assignee: 平安科技（深圳）有限公司
Priority date: 2018-05-31
Filing date: 2018-08-24
Publication date: 2019-12-05
Also published as: CN108959383A

Abstract

Disclosed are a network public opinion analysis method and apparatus, and a computer-readable storage medium. The method comprises: determining a public opinion event, and collecting public opinion articles related to the public opinion event (S10); pre-processing the collected public opinion articles, and acquiring vocabulary sets in the public opinion articles to represent the public opinion articles (S20); performing clustering analysis on the vocabulary sets by means of a clustering algorithm to generate a plurality of opinions on the public opinion event, and computing word vectors of the opinions (S30); extracting core topics from the vocabulary sets contained in the opinions (S40); computing emotion scores of the opinions by means of an emotion scoring model, and computing the popularity of the opinions (S50); and computing public opinion indexes of the opinions according to the emotion scores and the popularity, determining an opinion, the absolute value of a public opinion index thereof being greater than a pre-set threshold value, to be an abnormal opinion, and generating early-warning information according to the abnormal opinion and a core topic thereof and outputting the early-warning information (S60). The method improves public opinion monitoring and early-warning performance.

Description

Network public opinion analysis method, device and computer-readable storage medium

This application is based on the Paris Convention claiming the priority of a Chinese patent application filed on May 31, 2018 with application number 201810544762.6 and entitled "Analysis Method, Apparatus and Computer-readable Storage Media for Internet Public Opinion". The entire contents are incorporated herein by reference.

Technical field

The present application relates to the field of computer technology, and in particular, to a method and device for analyzing public opinion on a network, and a computer-readable storage medium.

Background technique

Internet public opinion refers to the network public opinion on the Internet that has different views on social events, and is a manifestation of social public opinion. It is mainly based on the Internet and the event as the core. The public's expression, dissemination and interaction of the event's emotions, attitudes, opinions, opinions, and subsequent influences.

Through the Internet, the public has strong influence and tendentious opinions and views on some hotspots and focus issues in real life. There are many ways to express public opinion online, such as news comments, forum posts, Weibo blog posts, WeChat articles, and so on. In recent years, the influence of internet public opinion on political life order and social stability has been increasing day by day. Some major incidents of internet public opinion have made people begin to realize the huge role of internet in social supervision.

At the same time, if the Internet public opinion emergencies are not handled properly, it will most likely induce bad feelings of the people, cause the people to violate the rules and act excessively, and then pose a threat to social stability. Therefore, it is more important to monitor the state of Internet public opinion, and it is necessary to analyze and warn the emotional tendency and viewpoint of Internet public opinion. The current mainstream public opinion system mainly involves personnel. For example, there are public opinion analysts in the industry who screen public opinion and then track the system to make a rough judgment on the state of public opinion in terms of the impact of the incident, but this scheme exists The shortcomings of insufficient monitoring of media sources and the lack of an automated public opinion index calculation scheme have resulted in the existing public opinion system not being able to accurately obtain the event's specific public opinion index, and fail to accurately warn.

Summary of the Invention

The present application provides a network public opinion analysis method, device, and computer-readable storage medium, the main purpose of which is to improve the ability of monitoring and early warning of public opinion.

In order to achieve the above purpose, the present application also provides a method for analyzing public opinion on the network, which method includes:

Determine public opinion events, and collect public opinion articles related to the public opinion events from a preset data channel through a distributed web crawler;

Perform word segmentation processing on the public opinion article, and obtain a vocabulary set in the public opinion article to characterize the public opinion article;

Cluster analysis using a vocabulary set of a clustering algorithm to generate multiple types of opinions of public opinion events, and calculate a word vector of the opinion according to the vocabulary set included in the opinion;

Extract one or more vocabulary sets from the vocabulary set contained in the viewpoint, and use the sentiment articles represented by the extracted vocabulary set as the core topic of the viewpoint;

The word vector of the viewpoint is input into a pre-trained sentiment scoring model, and the sentiment score of the viewpoint is output. According to the popularity of the sentiment article corresponding to the vocabulary set contained in the viewpoint, the popularity of each data channel and the preset weight of each data channel are used to calculate the viewpoint heat;

Calculate a public opinion index of a viewpoint according to the sentiment score and the heat, and determine that a viewpoint whose absolute value is greater than a preset threshold is an abnormal viewpoint, and generate and output warning information according to the abnormal viewpoint and a core topic of the abnormal viewpoint.

In addition, in order to achieve the above object, the present application also provides a network public opinion analysis device. The device includes a memory and a processor. The memory stores a public opinion analysis program that can be run on the processor. The public opinion analysis When the program is executed by the processor, the following steps are implemented:

In addition, in order to achieve the above object, the present application also provides a computer-readable storage medium, where the computer-readable storage medium stores a public opinion analysis program, and the public opinion analysis program can be executed by one or more processors to achieve The steps of the analysis method of network public opinion as described above.

The public opinion analysis method, device, and computer-readable storage medium proposed in this application determine public opinion events, and collect public opinion articles related to public opinion events from preset data channels through a distributed web crawler; perform word segmentation processing on public opinion articles to obtain The vocabulary set in the public opinion article is used to characterize the public opinion article; the clustering algorithm is used to perform a cluster analysis on the vocabulary set to generate multiple views of the public opinion event, and the word vector of the viewpoint is calculated based on the word vector of the vocabulary set included in the viewpoint; One or more vocabulary sets are extracted from the vocabulary set, and the public opinion article represented by the extracted vocabulary set is taken as the core topic of the viewpoint; the word vector of the viewpoint is input into a pre-trained sentiment score model, and the sentiment score of the viewpoint is output, and Calculate the popularity of the opinion according to the popularity of the sentiment articles corresponding to the vocabulary set contained in the opinion and the preset weights of each data channel; calculate the opinion's public opinion index based on the sentiment score and the popularity, and determine that the absolute value of the public opinion index is greater than the preset The threshold view is an abnormal view. The viewpoint and the core topic of the abnormal viewpoint generate early warning information and output. This application uses cluster analysis on the collected articles to construct multiple types of viewpoints of public opinion events, achieving a high degree of generalization of events, and integrating sentiment scoring model pairs. The sentiment scores of the generalized views are calculated to realize the judgment of the impact of various views on public opinion events, and then to provide early warning, which improves the ability of monitoring and early warning of public opinion.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic flowchart of a network public opinion analysis method according to an embodiment of the present application;

FIG. 2 is a schematic diagram of an internal structure of a network public opinion analysis device according to an embodiment of the present application; FIG.

FIG. 3 is a schematic block diagram of a public opinion analysis program in a network public opinion analysis device according to an embodiment of the present application.

The implementation, functional characteristics and advantages of the purpose of this application will be further described with reference to the embodiments and the drawings.

Detailed ways

It should be understood that the specific embodiments described herein are only used to explain the application, and are not used to limit the application.

This application provides a method for analyzing public opinion on the Internet. FIG. 1 is a schematic flowchart of a method for analyzing public opinion on a network provided by an embodiment of the present application. The method may be performed by a device, which may be implemented by software and / or hardware.

In this embodiment, a method for analyzing public opinion on the network includes:

Step S10: Determine a public opinion event, and collect a public opinion article related to the public opinion event from a preset data channel through a distributed web crawler.

The public opinion event in the embodiment of the present application is generally an event that occurs at present, and the user may set one or more keywords to indicate the public opinion event. Collect public opinion articles related to the public opinion event from a preset data channel through a distributed web crawler, and store the obtained public opinion articles according to the corresponding data channel. Specifically, a URL (Uniform Resource Locator, Uniform Resource Locator) list to be crawled is set in advance, and a web crawler is used regularly to crawl related information based on the URL addresses in the list and according to a preset keyword that can reflect the public opinion event. Public opinion articles are added to the corpus. Data channels include but are not limited to Weibo, WeChat, news portals, forums, etc. The public opinion articles obtained from the above data channels mainly include news comments, forum posts, Weibo blog posts, WeChat articles, etc.

Step S20: Perform word segmentation processing on the public opinion article, and obtain a vocabulary set in the public opinion article to characterize the public opinion article.

Specifically, this step includes the following refinement steps: extract the body data of the public opinion article, remove irrelevant data such as HTML (Hyper Text Markup Language), tag data, image markup, etc., and then remove the Non-Chinese characters. Segment the retained body data by the word segmentation tool to generate Chinese-separated vocabulary sets separated by spaces. The stop word processing is performed on the vocabulary set according to a preset stop word vocabulary, and the remaining vocabulary set is used to characterize the public opinion article, that is, the space-separated vocabulary set is used to characterize the public opinion article.

In step S30, a clustering analysis is performed on a vocabulary set of a clustering algorithm to generate multiple types of opinions of public opinion events, and a word vector of the opinion is calculated according to the vocabulary set included in the opinion.

In step S40, one or more vocabulary sets are extracted from the vocabulary set contained in the viewpoint, and the public opinion article represented by the extracted vocabulary set is taken as the core topic of the viewpoint.

After obtaining the vocabulary set of each public opinion article, cluster the public opinion articles represented by the space-separated vocabulary set. Because the public's views on an event may be diverse, different people have different views, so through cluster analysis of all collected public opinion events, a high degree of generalization of public opinion articles is obtained, and various opinion categories are obtained. Specifically, this step may include the following detailed steps:

Obtain the Chinese Wikipedia corpus, based on the corpus, select multiple words in the vocabulary set as keywords based on the word frequency-inverse text frequency index TF-IDF algorithm; generate a word vector model of the Chinese corpus based on the corpus, and calculate the keywords based on the word vector model Word vector, calculate the word vector of the vocabulary set according to the word vector of the keywords; cluster all vocabulary sets of public opinion events according to the word vectors of the vocabulary set and the Kmeans algorithm (K-means algorithm) to divide the vocabulary set of public opinion events into For multiple types of viewpoints, since the vocabulary set characterizes public opinion articles, the clustering of vocabulary sets is actually the clustering of public opinion articles; the keywords of the vocabulary set included in the viewpoint are summarized and calculated based on the word vectors of the summarized keywords Perspective word vector.

Obtain the Chinese Wikipedia corpus. Based on the corpus, calculate the importance of each word in the vocabulary set of each vocabulary set according to the TF-IDF algorithm. For each vocabulary set, select the top N vocabularies with the highest significance as the keywords of the article. . The Word2vec model of Chinese corpus is generated based on the Chinese Wikipedia corpus. For each vocabulary set, the word vector of the selected N keywords is calculated by the Word2vec model, and the word vector of the vocabulary set is calculated by the word vector of the keyword. Calculate the word vectors of all vocabulary sets of public opinion events in this way. Since the vocabulary set represents public opinion articles, the keywords for extracting the vocabulary set are actually the keywords for extracting public opinion articles.

After the word vectors of each vocabulary set are calculated, all public opinion articles (represented by the vocabulary set) in the corpus are clustered according to the Kmeans algorithm and classified into multiple types of opinions. The initial value of the K value of the Kmeans algorithm is randomly set, and the K value is the number of classification groups, and the K value is adjusted according to the evaluation of the classification result until the accuracy of the classification result reaches a set threshold.

The keywords of all vocabulary sets in each viewpoint are summarized, and the word frequency of each keyword is calculated. The word frequency reflects the weight of the keyword. The Word2vec model is used to calculate the word vector of each keyword summarized in the viewpoint, and the word vector of the viewpoint is calculated according to the word vector of the keyword and the word frequency. Calculate the similarity between the vocabulary set and the viewpoint based on the calculated word vector of the viewpoint and the word vector of the vocabulary set under the viewpoint. Among them, the similarity between the vectors can be calculated by the cosine similarity, and the one with the highest similarity or Articles of public opinion characterized by multiple vocabulary sets serve as the core topic of opinion.

In step S50, the word vector of the viewpoint is input into a pre-trained sentiment score model, and the sentiment score of the viewpoint is output, and according to the popularity of the sentiment article corresponding to the vocabulary set contained in the viewpoint in each data channel and the preset weight of each data channel Calculate the heat of opinion.

The preset deep neural network model is trained according to the constructed sample library in advance, the model parameters are determined, and the deep neural network model that determines the model parameters is used as the emotion score model. Specifically, this step includes the following detailed steps: obtaining public opinion text data to which tag data is added to form a sample library, wherein the tag data is an emotional score marked on the text data according to a positive, negative, and neutral tendency of the comment, such as a positive comment Is 1, negative comments are -1, neutral comments are 0; Wikipedia corpus combined with TF-IDF algorithm to extract the keywords of public opinion text data in the sample database, and the word vector of the keywords is calculated by the trained word vector model ; Use the word vector and label data of public opinion text data in the sample library as training samples, and input them into the preset deep neural network model for training to determine the model parameters, and use the deep neural network model with the model parameters as the target. The emotion scoring model is described. In addition, in order to prevent overfitting, cross-validation is used to train the model.

Calculate the sentiment score of each viewpoint using the trained sentiment scoring model, where the value range of the sentiment score is [-1, 1]. If the sentiment score of the viewpoint is negative, it indicates that the viewpoint is biased towards negative reviews; A positive sentiment score indicates that the opinion is biased towards positive comments; if the sentiment score of the opinion is near 0, it indicates that the opinion may favor neutral comments.

Analyze the statistical data of public opinion articles represented by vocabulary collections of various viewpoints on various data channels, including Weibo, WeChat, news portals, forums, etc. For example, for WeChat articles, the statistical data mainly includes the number of reads, comments, and retweets of a single WeChat article, and for Weibo blog posts, the statistical data mainly includes the number of reposts, comments, and likes. Evaluate the popularity of opinions on various channels, and calculate the popularity of opinions based on the popularity of opinions on various data channels and preset weights of each data channel. It can be understood that the popularity of each public opinion article can also be calculated by the above method.

Step S60: Calculate a public opinion index of a viewpoint based on the sentiment score and the popularity, and determine that a viewpoint whose absolute value is greater than a preset threshold is an abnormal viewpoint, and generate early warning information based on the abnormal viewpoint and a core topic of the abnormal viewpoint. And output.

After calculating the popularity and sentiment scores of each point of view, calculate its public opinion index. Among them, public opinion index = popularity * sentiment score. The size of the public opinion index reflects the influence of the opinion on public opinion. The absolute value of the public opinion index is closer to 1. , It shows that the greater the influence of this opinion on public opinion, when the calculated public opinion index reaches a preset threshold, the opinion is judged to be an abnormal opinion. For example, if the preset threshold is 0.8 and the calculated public opinion index of a certain viewpoint is -0.9, the absolute value is 0.9, which is greater than the preset threshold, and the public opinion index is biased toward negative evaluation. At this time, early warning information can be output. And the warning message contains the core topics of this view.

The public opinion analysis method proposed in this embodiment determines public opinion events, collects public opinion articles related to public opinion events from preset data channels through a distributed web crawler, performs word segmentation processing on public opinion articles, and obtains vocabulary sets in public opinion articles to Characterize public opinion articles; use a clustering algorithm to perform a cluster analysis on the vocabulary set to generate multiple opinions of public opinion events, and calculate the word vector of the viewpoint based on the word vector of the vocabulary set contained in the viewpoint; extract one or more from the vocabulary set contained in the viewpoint Two vocabulary sets, and the public opinion articles represented by the extracted vocabulary set are taken as the core topic of the viewpoint; the word vector of the viewpoint is input into a pre-trained sentiment score model, and the sentiment score of the viewpoint is output, and the corresponding The popularity of the public opinion article in each data channel and the preset weights of each data channel calculate the popularity of the opinion; calculate the opinion public opinion index based on the sentiment score and the popularity, and determine that the opinion whose absolute value of the public opinion index is greater than a preset threshold is an abnormal opinion. According to the anomalous perspective and the core of the anomalous perspective Generate early warning information for questions and output them. This application uses cluster analysis on collected articles to construct multiple views of public opinion events, and achieves a high degree of generalization of events. It also integrates the sentiment scoring model to calculate the sentiment scores of the summarized views. In order to judge the influence of various opinions on public opinion events, and then carry out early warning, the ability of monitoring and early warning of public opinion has been improved.

The present application also provides a network public opinion analysis device. Referring to FIG. 2, a schematic diagram of an internal structure of a network public opinion analysis apparatus according to an embodiment of the present application is shown.

In this embodiment, the network public opinion analysis device 1 may be a PC (Personal Computer) or a terminal device such as a smart phone, a tablet computer, or a portable computer. The network public opinion analysis device 1 includes at least a memory 11, a processor 12, a network interface 13, and a communication bus 14.

The memory 11 includes at least one type of readable storage medium. The readable storage medium includes a flash memory, a hard disk, a multimedia card, a card-type memory (for example, SD or DX memory, etc.), a magnetic memory, a magnetic disk, an optical disk, and the like. The memory 11 may be an internal storage unit of the network public opinion analysis device 1 in some embodiments, such as a hard disk of the network public opinion analysis device 1. The memory 11 may also be an external storage device of the network public opinion analysis device 1 in other embodiments, for example, a plug-in hard disk, a smart memory card (SMC), and a secure digital device provided on the network public opinion analysis device 1. (Secure Digital, SD) card, Flash card, etc. Further, the memory 11 may include both an internal storage unit of the network public opinion analysis device 1 and an external storage device. The memory 11 can be used not only to store application software and various types of data installed on the network public opinion analysis device 1, such as the code of the public opinion analysis program 01, but also to temporarily store data that has been or will be output.

The processor 12 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments, and is configured to run program codes or processes stored in the memory 11 Data, for example, public opinion analysis program 01 and so on.

The network interface 13 may optionally include a standard wired interface, a wireless interface (such as a WI-FI interface), and is generally used to establish a communication connection between the device 1 and other electronic devices.

The communication bus 14 is used to implement connection communication between these components.

Optionally, the device 1 may further include a user interface. The user interface may include a display, an input unit such as a keyboard, and the optional user interface may further include a standard wired interface and a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-type liquid crystal display, an OLED (Organic Light-Emitting Diode, organic light emitting diode) touch device, or the like. The display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the network public opinion analysis device 1 and to display a visualized user interface.

FIG. 2 only shows a network public opinion analysis device 1 having components 11-14 and a public opinion analysis program 01. Those skilled in the art can understand that the structure shown in FIG. 1 does not constitute an analysis device for network public opinion 1 The definition may include fewer or more components than shown, or some components may be combined, or different component arrangements.

In the embodiment of the apparatus 1 shown in FIG. 2, the public opinion analysis program 01 is stored in the memory 11; when the processor 12 executes the public opinion analysis program 01 stored in the memory 11, the following steps are implemented:

Determine a public opinion event, and collect a public opinion article related to the public opinion event from a preset data channel through a distributed web crawler.

The public opinion event in the embodiment of the present application is generally an event that occurs at present, and the user may set one or more keywords to indicate the public opinion event. Collect public opinion articles related to the public opinion event from a preset data channel through a distributed web crawler, and store the obtained public opinion articles according to the corresponding data channel. Specifically, a list of URLs to be crawled is set in advance, and a web crawler is used to periodically crawl related public opinion articles according to the URL addresses in the above list according to the preset keywords that can reflect the public opinion event and add them to the corpus. Data channels Including but not limited to Weibo, WeChat, News Portal, Forum, etc. The public opinion articles obtained from the above data channels mainly include news comments, forum posts, Weibo blog posts, WeChat articles, etc.

Word segmentation is performed on the public opinion article, and a vocabulary set in the public opinion article is obtained to characterize the public opinion article.

Specifically, this step includes the following refinement steps: extract the body data of the public opinion article, remove irrelevant data such as HTML tag data, image tags, and then remove non-Chinese characters in the body data through regular expressions. Segment the retained body data by the word segmentation tool to generate Chinese-separated vocabulary sets separated by spaces. The stop word processing is performed on the vocabulary set according to a preset stop word vocabulary, and the remaining vocabulary set is used to characterize the public opinion article, that is, the space-separated vocabulary set is used to characterize the public opinion article.

Cluster analysis is performed by using a clustering algorithm vocabulary set to generate multiple types of opinions of public opinion events, and a word vector of the opinion is calculated according to the vocabulary set included in the opinion.

One or more vocabulary sets are extracted from the vocabulary set contained in the viewpoint, and the public opinion article represented by the extracted vocabulary set is taken as the core topic of the viewpoint.

The keywords of public opinion articles represented by all vocabulary sets in each viewpoint are summarized, and the word frequency of each keyword is calculated. The word frequency reflects the weight of the keyword. The Word2vec model is used to calculate the word vector of each keyword summarized in the viewpoint, and the word vector of the viewpoint is calculated according to the word vector of the keyword and the word frequency. Calculate the similarity between the vocabulary set and the viewpoint based on the calculated word vector of the viewpoint and the word vector of the vocabulary set under the viewpoint. Among them, the similarity between the vectors can be calculated by the cosine similarity, and the one with the highest similarity or Articles of public opinion characterized by multiple vocabulary sets serve as the core topic of opinion.

The word vector of the viewpoint is input into a pre-trained sentiment scoring model, and the sentiment score of the viewpoint is output. According to the popularity of the sentiment article corresponding to the vocabulary set contained in the viewpoint, the popularity of each data channel and the preset weight of each data channel are used to calculate the viewpoint heat.

The device for analyzing public opinion in this embodiment. The device first determines public opinion events, and then collects public opinion articles related to public opinion events from preset data channels through a distributed web crawler; performs word segmentation processing on the public opinion articles to obtain public opinion articles. Vocabulary set to characterize public opinion articles; cluster analysis is performed on the vocabulary set using a clustering algorithm to generate multiple views of public opinion events, and the word vector of the viewpoint is calculated based on the word vector of the vocabulary set contained in the viewpoint; Extract one or more vocabulary sets, and use the sentiment articles represented by the extracted vocabulary set as the core topic of the viewpoint; input the word vector of the viewpoint into a pre-trained sentiment scoring model, output the sentiment's sentiment score, and include The popularity of the public opinion article corresponding to the vocabulary set in each data channel and the preset weight of each data channel calculate the popularity of the opinion; calculate the opinion public opinion index based on the sentiment score and the popularity, and determine the opinion whose absolute value of the public opinion index is greater than the preset threshold Is an abnormal view, according to the abnormal view and the difference The core topics of common viewpoints generate early warning information and output. Through the cluster analysis of the collected articles, this application constructs multiple viewpoints of public opinion events, achieving a high degree of generalization of events, and integrates the sentiment scoring model to summarize the viewpoints obtained. Emotional scores are calculated to determine the impact of various opinions on public opinion events, and then to provide early warning, which improves the ability to monitor and alert public opinion.

Optionally, in other embodiments, the public opinion analysis program may also be divided into one or more modules, and the one or more modules are stored in the memory 11 and are implemented by one or more processors (this embodiment is The processor 12) executes to complete this application. The modules referred to in this application refer to a series of computer program instruction segments capable of performing specific functions, and are used to describe the execution process of a public opinion analysis program in an analysis device for network public opinion.

For example, referring to FIG. 3, it is a schematic diagram of a program module of a public opinion analysis program in an embodiment of an apparatus for analyzing public opinion on the Internet in this application. In this embodiment, the public opinion analysis program can be divided into a data collection module 10 and an article segmentation module 20. , Article clustering module 30, topic extraction module 40, score calculation module 50, and index calculation module 60, for example:

The data collection module 10 is configured to determine a public opinion event and collect a public opinion article related to the public opinion event from a preset data channel through a distributed web crawler;

The article segmentation module 20 is configured to perform word segmentation processing on the public opinion article, and obtain a vocabulary set in the public opinion article to characterize the public opinion article;

The article clustering module 30 is configured to perform cluster analysis by using a clustering algorithm vocabulary set to generate multiple types of opinions of public opinion events, and calculate a word vector of the opinion according to the vocabulary set included in the opinion;

The topic extraction module 40 is configured to extract one or more vocabulary sets from the vocabulary set contained in the viewpoint, and use the public opinion articles represented by the extracted vocabulary set as the core topic of the viewpoint;

The score calculation module 50 is used to input a word vector of a viewpoint into a pre-trained sentiment scoring model, output a sentiment score, and according to the popularity of the sentiment article corresponding to the vocabulary set contained in the viewpoint in each data channel and each data channel Calculation of the popularity of views by preset weights;

The index calculation module 60 is configured to calculate a public opinion index of a viewpoint according to the sentiment score and the heat, and determine that a viewpoint whose absolute value is greater than a preset threshold is an abnormal viewpoint, and according to the abnormal viewpoint and the core of the abnormal viewpoint The topic generates warning information and outputs it.

The functions or operation steps implemented when the program modules such as the data acquisition module 10, the article word segmentation module 20, the article clustering module 30, the topic extraction module 40, the score calculation module 50, and the index calculation module 60 are executed are substantially the same as those in the above embodiment. , Will not repeat them here.

In addition, an embodiment of the present application further proposes a computer-readable storage medium. The computer-readable storage medium stores a public opinion analysis program, and the public opinion analysis program can be executed by one or more processors to implement the following operations:

The specific implementation manner of the computer-readable storage medium of the present application is basically the same as each embodiment of the above-mentioned network public opinion analysis device and method, and is not repeated here.

It should be noted that, the serial numbers of the embodiments of the present application are only for description, and do not represent the advantages and disadvantages of the embodiments. And the terms "including," "including," or any other variation thereof, are intended to cover non-exclusive inclusion, such that a process, device, article, or method that includes a series of elements includes not only those elements, but also The other elements listed, or those that are inherent to such a process, device, article, or method. Without more restrictions, an element limited by the sentence "including a ..." does not exclude that there are other identical elements in the process, device, article, or method that includes the element.

Through the description of the above embodiments, those skilled in the art can clearly understand that the methods in the above embodiments can be implemented by means of software plus a necessary universal hardware platform, and of course, also by hardware, but in many cases the former is better. Implementation. Based on such an understanding, the technical solution of the present application, in essence, or a part that contributes to the existing technology, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM) as described above. , Magnetic disk, optical disc), including a number of instructions to enable a terminal device (which may be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in the embodiments of the present application.

The above are only preferred embodiments of the present application, and thus do not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the description and drawings of the application, or directly or indirectly used in other related technical fields Are included in the scope of patent protection of this application.

Claims

A method for analyzing public opinion on the Internet, characterized in that the method includes:

Determine public opinion events, and collect public opinion articles related to the public opinion events from a preset data channel through a distributed web crawler;

Perform word segmentation processing on the public opinion article, and obtain a vocabulary set in the public opinion article to characterize the public opinion article;

Cluster analysis using a vocabulary set of a clustering algorithm to generate multiple types of opinions of public opinion events, and calculate a word vector of the opinion according to the vocabulary set included in the opinion;

Extract one or more vocabulary sets from the vocabulary set contained in the viewpoint, and use the sentiment articles represented by the extracted vocabulary set as the core topic of the viewpoint;

The word vector of the viewpoint is input into a pre-trained sentiment scoring model, and the sentiment score of the viewpoint is output. According to the popularity of the sentiment article corresponding to the vocabulary set contained in the viewpoint, the popularity of each data channel and the preset weight of each data channel are used to calculate the viewpoint heat;

Calculate a public opinion index of a viewpoint according to the sentiment score and the heat, and determine that a viewpoint whose absolute value is greater than a preset threshold is an abnormal viewpoint, and generate and output warning information according to the abnormal viewpoint and a core topic of the abnormal viewpoint.
The network public opinion analysis method according to claim 1, wherein the step of performing word segmentation processing on the public opinion article and obtaining a vocabulary set in the public opinion article to characterize the public opinion article comprises:

Extract the body data of public opinion articles, and remove non-Chinese characters from the body data through regular expressions;

Word segmentation processing is performed on the text data after removing non-Chinese characters, and the text data is converted into a vocabulary set separated by spaces;

De-stop word processing on the vocabulary set to obtain the vocabulary set.
The network public opinion analysis method according to claim 1, wherein the cluster analysis is performed by using a clustering algorithm vocabulary set to generate multiple types of opinions of public opinion events, and the calculation is based on the vocabulary set included in the opinions The steps for this opinion's word vector include:

Obtaining a Chinese Wikipedia corpus, and based on the corpus, selecting multiple words in a vocabulary set as keywords based on a word frequency-inverse text frequency index TF-IDF algorithm;

Generating a word vector model of a Chinese corpus based on the corpus, calculating a word vector of a keyword through the word vector model, and calculating a word vector of a vocabulary set according to the word vector of the keyword;

Clustering all vocabulary sets of the public opinion event according to the word vector of the vocabulary set and the Kmeans algorithm to divide the vocabulary set of the public opinion event into multiple types of viewpoints;

The keywords of the vocabulary set included in the viewpoint are summarized, and the word vectors of the viewpoint are calculated according to the word vectors of the summarized keywords.
The network public opinion analysis method according to claim 3, wherein the extracting one or more vocabulary sets from the vocabulary set contained in the viewpoint, and using the public opinion article represented by the extracted vocabulary set as the core topic of the viewpoint The steps include:

Calculate the similarity between the vocabulary set and its corresponding viewpoint according to the word vector of the viewpoint and the word vector of the vocabulary set contained in the viewpoint;

A public opinion article characterized by one or more vocabulary sets with a similarity greater than a preset threshold is used as the core topic of the opinion.
The network public opinion analysis method according to claim 1, wherein the training step of the emotion score model comprises:

Obtain public opinion text data with labeled data to form a sample library;

Combine the TF-IDF algorithm to extract the keywords of the public opinion text data in the sample database, and calculate the word vector of the keywords through the trained word vector model;

The word vector and label data of public opinion text data in the sample library are used as training samples, and input to a preset deep neural network model for training to determine model parameters, and a deep neural network model that determines the model parameters is used as the training sample. Emotional scoring model.
The network public opinion analysis method according to claim 2, wherein the training step of the emotion score model comprises:

Obtain public opinion text data with labeled data to form a sample library;

Combine the TF-IDF algorithm to extract the keywords of the public opinion text data in the sample database, and calculate the word vector of the keywords through the trained word vector model;

The word vector and label data of public opinion text data in the sample library are used as training samples, and input to a preset deep neural network model for training to determine model parameters, and a deep neural network model that determines the model parameters is used as the training sample. Emotional scoring model.
The network public opinion analysis method according to claim 3, wherein the training step of the emotion score model comprises:

Obtain public opinion text data with labeled data to form a sample library;

Combine the TF-IDF algorithm to extract the keywords of the public opinion text data in the sample database, and calculate the word vector of the keywords through the trained word vector model;

The word vector and label data of public opinion text data in the sample library are used as training samples, and input to a preset deep neural network model for training to determine model parameters, and a deep neural network model that determines the model parameters is used as the training sample. Emotional scoring model.
A network public opinion analysis device is characterized in that the device includes a memory and a processor. The memory stores a public opinion analysis program that can be run on the processor, and the public opinion analysis program is used by the processor. Implement the following steps during execution:

Determine public opinion events, and collect public opinion articles related to the public opinion events from a preset data channel through a distributed web crawler;

Perform word segmentation processing on the public opinion article, and obtain a vocabulary set in the public opinion article to characterize the public opinion article;

Cluster analysis using a vocabulary set of a clustering algorithm to generate multiple types of opinions of public opinion events, and calculate a word vector of the opinion according to the vocabulary set included in the opinion;

Extract one or more vocabulary sets from the vocabulary set contained in the viewpoint, and use the sentiment articles represented by the extracted vocabulary set as the core topic of the viewpoint;

The word vector of the viewpoint is input into a pre-trained sentiment scoring model, and the sentiment score of the viewpoint is output. According to the popularity of the sentiment article corresponding to the vocabulary set contained in the viewpoint, the popularity of each data channel and the preset weight of each data channel are used to calculate the viewpoint heat;

Calculate a public opinion index of a viewpoint according to the sentiment score and the heat, and determine that a viewpoint whose absolute value is greater than a preset threshold is an abnormal viewpoint, and generate and output warning information according to the abnormal viewpoint and a core topic of the abnormal viewpoint.
The network public opinion analysis device according to claim 8, wherein the step of performing word segmentation processing on the public opinion article and obtaining a vocabulary set in the public opinion article to characterize the public opinion article comprises:

Extract the body data of public opinion articles, and remove non-Chinese characters from the body data through regular expressions;

Word segmentation processing is performed on the text data after removing non-Chinese characters, and the text data is converted into a vocabulary set separated by spaces;

De-stop word processing on the vocabulary set to obtain the vocabulary set.
The network public opinion analysis device according to claim 8, wherein the cluster analysis is performed by using a clustering algorithm vocabulary set to generate multiple types of opinions of public opinion events, and is calculated based on the vocabulary set included in the opinions The steps for this opinion's word vector include:

Obtaining a Chinese Wikipedia corpus, and based on the corpus, selecting multiple words in a vocabulary set as keywords based on a word frequency-inverse text frequency index TF-IDF algorithm;

Generating a word vector model of a Chinese corpus based on the corpus, calculating a word vector of a keyword through the word vector model, and calculating a word vector of a vocabulary set according to the word vector of the keyword;

Clustering all vocabulary sets of the public opinion event according to the word vector of the vocabulary set and the Kmeans algorithm to divide the vocabulary set of the public opinion event into multiple types of viewpoints;

The keywords of the vocabulary set included in the viewpoint are summarized, and the word vectors of the viewpoint are calculated according to the word vectors of the summarized keywords.
The network public opinion analysis device according to claim 10, wherein the one or more vocabulary sets are extracted from the vocabulary set contained in the viewpoint, and the public opinion article represented by the extracted vocabulary set is used as the core topic of the viewpoint The steps include:

Calculate the similarity between the vocabulary set and its corresponding viewpoint according to the word vector of the viewpoint and the word vector of the vocabulary set contained in the viewpoint;

A public opinion article characterized by one or more vocabulary sets with a similarity greater than a preset threshold is used as the core topic of the opinion.
The network public opinion analysis device according to claim 8, wherein the training step of the emotion score model comprises:

Obtain public opinion text data with labeled data to form a sample library;

Combine the TF-IDF algorithm to extract the keywords of the public opinion text data in the sample database, and calculate the word vector of the keywords through the trained word vector model;

The word vector and label data of public opinion text data in the sample library are used as training samples, and input to a preset deep neural network model for training to determine model parameters, and a deep neural network model that determines the model parameters is used as the training sample. Emotional scoring model.
The network public opinion analysis device according to claim 9, wherein the training step of the emotion score model comprises:

Obtain public opinion text data with labeled data to form a sample library;

Combine the TF-IDF algorithm to extract the keywords of the public opinion text data in the sample database, and calculate the word vector of the keywords through the trained word vector model;

The word vector and label data of public opinion text data in the sample library are used as training samples, and input to a preset deep neural network model for training to determine model parameters, and a deep neural network model that determines the model parameters is used as the training sample. Emotional scoring model.
The network public opinion analysis device according to claim 10, wherein the training step of the emotion score model comprises:

Obtain public opinion text data with labeled data to form a sample library;

Combine the TF-IDF algorithm to extract the keywords of the public opinion text data in the sample database, and calculate the word vector of the keywords through the trained word vector model;

The word vector and label data of public opinion text data in the sample library are used as training samples, and input to a preset deep neural network model for training to determine model parameters, and a deep neural network model that determines the model parameters is used as the training sample. Emotional scoring model.
A computer-readable storage medium is characterized in that the computer-readable storage medium stores a public opinion analysis program, and the public opinion analysis program can be executed by one or more processors to implement the following steps:

Determine public opinion events, and collect public opinion articles related to the public opinion events from a preset data channel through a distributed web crawler;

Perform word segmentation processing on the public opinion article, and obtain a vocabulary set in the public opinion article to characterize the public opinion article;

Cluster analysis using a vocabulary set of a clustering algorithm to generate multiple types of opinions of public opinion events, and calculate a word vector of the opinion according to the vocabulary set included in the opinion;

Extract one or more vocabulary sets from the vocabulary set contained in the viewpoint, and use the sentiment articles represented by the extracted vocabulary set as the core topic of the viewpoint;

The word vector of the viewpoint is input into a pre-trained sentiment scoring model, and the sentiment score of the viewpoint is output. According to the popularity of the sentiment article corresponding to the vocabulary set contained in the viewpoint, the popularity of each data channel and the preset weight of each data channel are used to calculate the viewpoint heat;

Calculate a public opinion index of a viewpoint according to the sentiment score and the heat, and determine that a viewpoint whose absolute value is greater than a preset threshold is an abnormal viewpoint, and generate and output warning information according to the abnormal viewpoint and a core topic of the abnormal viewpoint.
The computer-readable storage medium of claim 15, wherein the step of performing word segmentation processing on the public opinion article and obtaining a vocabulary set in the public opinion article to characterize the public opinion article comprises:

Extract the body data of public opinion articles, and remove non-Chinese characters from the body data through regular expressions;

Word segmentation processing is performed on the text data after removing non-Chinese characters, and the text data is converted into a vocabulary set separated by spaces;

De-stop word processing on the vocabulary set to obtain the vocabulary set.
The computer-readable storage medium according to claim 15, wherein the cluster analysis is performed by using a clustering algorithm vocabulary set to generate multiple types of opinions of public opinion events, and calculation is performed based on the vocabulary set included in the opinions The steps for this opinion's word vector include:

Obtaining a Chinese Wikipedia corpus, and based on the corpus, selecting multiple words in a vocabulary set as keywords based on a word frequency-inverse text frequency index TF-IDF algorithm;

Generating a word vector model of a Chinese corpus based on the corpus, calculating a word vector of a keyword through the word vector model, and calculating a word vector of a vocabulary set according to the word vector of the keyword;

Clustering all vocabulary sets of the public opinion event according to the word vector of the vocabulary set and the Kmeans algorithm to divide the vocabulary set of the public opinion event into multiple types of viewpoints;

The keywords of the vocabulary set included in the viewpoint are summarized, and the word vectors of the viewpoint are calculated according to the word vectors of the summarized keywords.
The computer-readable storage medium according to claim 17, wherein the extracting one or more vocabulary sets from the vocabulary set contained in the viewpoint, and using public opinion articles represented by the extracted vocabulary set as the core topics of the viewpoint The steps include:

Calculate the similarity between the vocabulary set and its corresponding viewpoint according to the word vector of the viewpoint and the word vector of the vocabulary set contained in the viewpoint;

A public opinion article characterized by one or more vocabulary sets with a similarity greater than a preset threshold is used as the core topic of the opinion.
The computer-readable storage medium of claim 15, wherein the step of training the emotion score model comprises:

Obtain public opinion text data with labeled data to form a sample library;

Combine the TF-IDF algorithm to extract the keywords of the public opinion text data in the sample database, and calculate the word vector of the keywords through the trained word vector model;

The word vector and label data of public opinion text data in the sample library are used as training samples, and input to a preset deep neural network model for training to determine model parameters, and a deep neural network model that determines the model parameters is used as the training sample. Emotional scoring model.
The computer-readable storage medium of claim 16, wherein the training step of the emotion score model comprises:

Obtain public opinion text data with labeled data to form a sample library;

Combine the TF-IDF algorithm to extract the keywords of the public opinion text data in the sample database, and calculate the word vector of the keywords through the trained word vector model;

The word vector and label data of public opinion text data in the sample library are used as training samples, and input to a preset deep neural network model for training to determine model parameters, and a deep neural network model that determines the model parameters is used as the training sample. Emotional scoring model.