TWI534640B - Chinese network information monitoring and analysis system and its method - Google Patents

Description

Chinese network information monitoring and analysis system and method thereof

The present invention relates to a Chinese network information monitoring and analysis system and method thereof, and more particularly to a monitoring and analysis system and method thereof, including an online community, various media, and a forum equal to information published on the Internet.

With the advancement of network technology, the population of the Internet has continued to grow, and the degree of life networking has also increased day by day. Today, the Internet is used for various online types of browsing, commenting, chatting, and feelings. A variety of online activities have become a daily must for Internet users.

In today's highly popular Internet, micro-blogs, micro-media and other such short-form social communications continue to flourish, which is different from traditional instant messaging, chat rooms, and bulletin boards. In other ways, micro-blogs and micro-media systems enable people to communicate in a "life-by-stream" manner. This type of communication has thoughts, opinions, and comments about people's real-life experiences of sharing through electronic devices connected to the Internet.

These huge amounts of information often reveal many trends in the social trends, such as the extent to which a product is discussed in the market and its evaluation, even the number of times it has been discussed in the past few days. Effective acquisition and analysis will bring useful analytical data to the target information you want to know.

One of the objectives of the present invention is to achieve various social media, forum websites, etc. on the Internet. The acquisition and analysis of network information.

Another object of the present invention is to achieve accurate monitoring and analysis results by the combination of a specific algorithm and a specific analysis method.

To achieve the above and other objects, the present invention provides a Chinese network information monitoring and analysis system for monitoring and analyzing on the Internet according to at least one Chinese target information input, including: a word storage device, and a system A plurality of Chinese word segment information, plural Chinese synonym information, and plural Chinese word information are stored; a computer computing device is connected to the thesaurus storage device, and includes: a target information processing module, a network information processing analysis module, and a network The information collation module and the data storage device are connected to the computer computing device, and the webpage data and the word segmentation information retrieved according to the serial code corresponding to the retrieved webpage are used to classify the captured data. Web page capture data. The target information processing module receives the at least one Chinese target information; the network information processing analysis module searches for and generates an analysis result on the Internet; the network information collation module is based on the Chinese target information Selecting a corresponding category in a database storage device and comparing the word segmentation information of the captured webpage, and capturing a page of the webpage when a matching webpage is used to generate a webpage retrieval data to provide the selected webpage Web page capture data.

The network information processing and analysis module comprises: a network information acquisition unit for extracting webpage data on the internet; and a word segmentation processing unit for using the webpage captured The data is processed according to the punctuation marks and the Chinese word segmentation information, and then the second-order processing is performed according to the maximum matching method to generate the result of the word segmentation of the corresponding webpage; the word frequency processing unit is based on the And the Chinese synonym information and the Chinese implied word information, the frequency of occurrence of the corresponding sentence is counted in the result of the broken sentence to generate a word frequency counting result of the corresponding webpage; and a webpage fingerprint processing unit is used for the webpage Take webpage data to classify webpage attributes, The label of the webpage source code of the webpage captured is used as a node for segment cutting, and the TF/IDF weight analysis component is used according to the word frequency counting result to generate the weight of each word segment of the corresponding webpage, and the self-weight value is large. Selecting a predetermined number of word segments from the smallest arrangement and rearranging them according to their characters to generate segmentation information of one of the corresponding web pages, and finally converting the selected characters into a string of the number of reserved bits according to the hash algorithm. The code, which in turn generates the serial code of the corresponding web page.

In an embodiment of the present invention, the network information capturing unit is further configured to perform web page data retrieval by logging in to the network platform that needs to log in according to the preset login information.

The invention further provides a Chinese network information monitoring and analysis method, which is used for monitoring and analyzing on the Internet according to the input Chinese target information, and comprises the following steps: capturing webpage data on the internet; and performing webpage data. The word segmentation is processed, and the first-order processing of the word segmentation sentence is performed by the punctuation marks and the Chinese word segment table information in the preset word storage device, and the second-order processing is performed by the maximum matching method to generate a word segmentation result. The word frequency processing of the webpage data is performed, and the Chinese synonym information and the Chinese implied word information in the vocabulary storage device are preset to count the words and the frequency of the corresponding words in the lexicon storage device in the result of the word segmentation sentence and the frequency thereof To generate a word frequency counting result; performing webpage fingerprint processing, first using the label of the webpage source code of the captured webpage data as a node of the paragraph cutting, and using the TF/IDF weight analysis component according to the word frequency counting result to generate The weight of each participle, and the pre-determined number of participles are selected from the arrangement of the weights from large to small and according to their characters The new arrangement is to generate a participle information, and finally the selected characters are converted into a series of codes of the reserved bit number according to the hash algorithm, thereby generating the serial code corresponding to the retrieved webpage data; storing the webpage data corresponding to the webpage data And the word segmentation information and the serial code; and in the stored word segmentation information and the serial code, the corresponding category is selected according to the Chinese target information and the word segmentation information of the captured webpage is compared, and the matching page is captured. The page of the web page generates a web page to retrieve the data to provide the captured web page.

Thereby, the present invention is applied to various types of social media data collected on the network, including unstructured data (such as text), semi-structured data (such as HTML files), and structured data (such as tables). And analyze, filter, convert, extract, model and semantic analysis of the data, and then conduct various monitoring and investigation of the Chinese target information, for example: understanding customer behavior, corporate brand and product reputation evaluation, specific micro The effectiveness of the media, in turn, can help quantify the number of successful marketing campaigns. In addition, it can prevent companies from inadvertently revealing the client’s personal resources on the Internet.

1‧‧‧Chinese Network Information Monitoring and Analysis System

2‧‧‧Internet

3‧‧‧User's electronic communication device

100‧‧‧ Thesaurus storage device

200‧‧‧Computer computing device

210‧‧‧Target Information Processing Module

230‧‧‧Network Information Processing Analysis Module

231‧‧‧Network Information Capture Unit

233‧‧‧Sentence word segment processing unit

235‧‧ ‧ word frequency processing unit

237‧‧‧Webpage fingerprint processing unit

250‧‧‧Network Information Organizer

300‧‧‧Database storage device

S101~S111‧‧‧Steps

1 is a system block diagram of a Chinese network information monitoring and analysis system according to an embodiment of the present invention.

2 is a flow chart of a method for operating a Chinese network information monitoring and analysis system according to an embodiment of the present invention.

FIG. 3 is a system block diagram of a Chinese network information monitoring and analysis system according to another embodiment of the present invention.

In order to fully understand the object, features and advantages of the present invention, the present invention will be described in detail by the following specific embodiments and the accompanying drawings. Departure, after the retrieval and classification of the webpage data, the characteristic words and their weights representing the webpage information are stored in the database storage device, and are stored by each webpage stored in the database storage device. The feature words and their weights are used as objects to be compared to use the feature items to evaluate the degree of relevance of the unknown text in the web page material to the topic. Among them, the selection of feature words and their weights is called feature selection of topic samples. The frequency distribution of words and phrases in different content documents is different. Therefore, feature selection and weight evaluation can be performed according to the frequency characteristics of words and phrases. The technology of the present invention enables accurate monitoring and investigation of Chinese target information, and is stored in a database storage device for users to search and obtain in the database. Web page capture data.

First, please refer to FIG. 1 , which is a system block diagram of a Chinese network information monitoring and analysis system according to an embodiment of the present invention. The present invention provides a connection between the user's electronic communication device 3 by connecting the Internet information monitoring and analysis system 1 of the Internet 2, wherein the user's electronic communication device 3 is exemplified by a user. It is not limited to a single number of users, and the number of simultaneous connections depends on the device level of the computer computing device 1. The user's electronic communication device 3 can be an electronic communication device such as a desktop computer, a smart phone, a personal digital assistant device, a tablet computer, or the like, which can be directly or indirectly connected to the Internet.

The Chinese network information monitoring and analysis system 1 of the present invention comprises: a thesaurus storage device 100, a computer computing device 200, and a database storage device 300. The computer computing device 200 connects the thesaurus storage device 100 and the database storage device 300 to search and retrieve webpages from the databases.

The thesaurus storage device 100 stores a plurality of Chinese word segment information, plural Chinese synonym information, and plural Chinese word information. The Chinese word segmentation information stored in the thesaurus storage device 100 contains a large number of common words that do not become feature items. In order to improve the operational efficiency of the Chinese network information monitoring and analysis system 1, the system is built by the thesaurus storage device 100. A large number of Chinese word segmentation tables are provided, which can significantly improve the operating efficiency of the system under the premise of ensuring the accuracy of feature selection. In addition, taking into account the diversity of natural language, the thesaurus storage library 100 is used to build auxiliary lexicons such as the Chinese synonym database and the Chinese connotation (Conditional Connective) vocabulary to improve the accuracy of information matching when performing word frequency statistics. .

The database storage device 300 is coupled to the computer computing device 200 for storing the captured webpage data and its word segmentation according to the serial code corresponding to the webpage captured by the computer computing device 200. Information, and then classify the captured web pages to retrieve data. Through continuous updating of the data in the database storage device 300, the present invention constructs a complete webpage retrieval data and its classification (by frequency characteristics and weight evaluation of words and phrases), and can be based on Chinese target information submitted by the user. For further search and retrieval on the Internet.

The computer computing device 200 is connected to the thesaurus storage device 100 and the database storage device 300, and includes a target information processing module 210, a network information processing and analysis module 230, and a network information organization module 250. The target information processing module 210 is configured to receive Chinese target information submitted by the user. The network information processing analysis module 230 is used to search and generate analysis results on the Internet 2. The network information collating module 250 selects the corresponding category in the database storage device 300 according to the Chinese target information and compares the word segmentation information of the captured webpage, and captures the page of the webpage when the matching webpage is generated to generate A web page captures information to provide the captured web page data to the user's electronic communication device 3.

The network information processing and analysis module 230 further includes: a network information capturing unit 231, a word segmentation processing unit 233, a word frequency processing unit 235, and a webpage fingerprint processing unit 237.

The network information capturing unit 231 is configured to perform webpage data retrieval on the Internet 2, and the webpage data is captured through the Internet 2 for public information such as a website page, a search engine, and a micro media. For subsequent analysis and classification.

The word segmentation processing unit 233 is configured to perform the first-order processing of the word segmentation sentence according to the punctuation marks and the Chinese word segment table information, and then perform the second-order processing according to the maximum matching method to generate Corresponding to the result of the word break of the web page. The so-called first-order processing means that the punctuation marks are used to break the sentence first, and then the content of the Chinese word segment table information is used to make a keyword break in the corresponding text of the web page to obtain the feature words initially; and then the maximum matching method is used. Detailed segmentation To achieve higher accuracy.

The maximum matching method is a conventional mechanical word segmentation method, which is a term in the Chinese character string to be analyzed according to a certain strategy and a "sufficiently large" thesaurus (for example, the thesaurus storage device 100 of the present invention). Matching, if a string is found in the thesaurus, the match is successful (a word is recognized). According to different scanning directions, the string matching word segmentation method can be divided into forward matching and reverse matching; according to the case of preferential matching with different lengths, it can be divided into the largest (longest) matching and the smallest (shortest) matching. One of the features of the present invention is the selection of a specific matching method to achieve the best combination effect with subsequent analysis. Therefore, the present invention uses the "maximum" matching method, and further, in one embodiment, the positive direction is used separately. And the inverse maximum matching method to competing to produce the best word segmentation result.

The word frequency processing unit 235 counts the frequency of occurrence of the corresponding word in the word segmentation result according to the Chinese synonym information and the Chinese word information, to generate a word frequency counting result of the corresponding web page.

The webpage fingerprint processing unit 237 is configured to perform webpage attribute classification on the retrieved webpage data, and the label of the webpage source code (HTML document) of the retrieved webpage is used as a node for segment cutting, and the result is counted according to the word frequency. The TF/IDF weight analysis component is used together to generate the weight of each word segment of the corresponding webpage, and the predetermined predetermined number of word segments are selected from the arrangement of the weighted value to be small and rearranged according to the character to generate one of the corresponding webpages. The word segmentation information is finally converted into a series of codes of the reserved bit number according to the hash algorithm, thereby generating the serial code of the corresponding webpage. Wherein, the TF/IDF weight analysis component uses the conventional TF/IDF formula to calculate the weight value of each word after the broken word segmentation in the captured webpage data, and the TF/IDF formula is as follows (1) Show:

In the formula (1), i= represents a specific word, j= represents the file in which the word is located, tf _i,j represents the frequency at which the word of i appears in the file j, and N represents the file in the set. The number, df _i represents the number of files containing the word i, the log item is the IDF value, and w _i,j is the weight of the calculated i in the file j. The weight of all words appearing in the file can be calculated by the calculation of the TF/IDF weight analysis component.

In order to improve the operation efficiency, the system performs dimension reduction processing on the feature vector, and only retains the words with higher weights as the feature items of the document, thereby forming the target feature vector with lower dimension, and the webpage fingerprint processing unit 237 is in an embodiment. The middle system selects the first 10 features as the feature strings of the retrieved webpage data, and further, in the step of performing webpage fingerprint processing, the number of reserved bits is set to 128 bits.

Next, please refer to FIG. 2, which is a flowchart of a method for running a Chinese network information monitoring and analysis system according to an embodiment of the present invention.

First, in step S101, the webpage data is retrieved on the Internet; then, in step S103, the word segmentation processing of the webpage data is performed, and the punctuation marks and the Chinese word segmentation information in the preset word storage device are used. The first-order processing of the broken words is performed, and the second-order processing is performed by the maximum matching method to generate a word-breaking sentence result; then, in step S105, the word frequency processing of the web page data is performed, and the word library storage device is preset Chinese synonym information and other Chinese implied word information counts the word segmentation sentence and the frequency of the corresponding word in the word bank storage device to generate a word frequency counting result; then, step S107, web page fingerprint processing, first The label of the webpage source code of the captured webpage data is used as a node for segment cutting, and the TF/IDF weight analysis component is used according to the word frequency counting result to generate the weight of each word segment, and the self-weight value is large to small. Select a predetermined number of participles in the arrangement and rearrange them according to their characters to generate a participle information. The algorithm converts the selected characters into a string of the number of reserved bits, and then generates a serial code corresponding to the retrieved webpage data; then, in step S109, the word segmentation information and the serial code corresponding to the webpage data are stored; In step S111, in the stored word segmentation information and the serial code, the corresponding category is selected according to the Chinese target information and the word segmentation information of the captured webpage is compared to generate a matching result and corresponding information. The page of the webpage is retrieved when there is a matching webpage to generate a webpage retrieval data to provide the captured webpage retrieval data.

Next, please refer to FIG. 3, which is a system block diagram of a Chinese network information monitoring and analysis system according to another embodiment of the present invention. The network information capturing unit 231 is further configured to log in to the web platform that needs to log in according to the preset login information to perform webpage data retrieval. As shown in FIG. 3, the webpage data can be retrieved and analyzed and summarized in the social media, and then stored in the database storage device 300.

In summary, the present invention can select the feature information of the target according to the Chinese target information submitted by the user, automatically collect the data on the Internet according to the feature capital, and classify and import the collected webpage data into the database. Provide personalized search service for Chinese online information through automatic operation and update of the system.

The invention has been described above in terms of the preferred embodiments, and it should be understood by those skilled in the art that the present invention is not intended to limit the scope of the invention. It should be noted that variations and permutations equivalent to those of the embodiments are intended to be included within the scope of the present invention. Therefore, the scope of protection of the present invention is defined by the scope of the patent application.

1‧‧‧Chinese Network Information Monitoring and Analysis System

2‧‧‧Internet

3‧‧‧User's electronic communication device

100‧‧‧ Thesaurus storage device

200‧‧‧Computer computing device

210‧‧‧Target Information Processing Module

230‧‧‧Network Information Processing Analysis Module

231‧‧‧Network Information Capture Unit

233‧‧‧Sentence word segment processing unit

235‧‧ ‧ word frequency processing unit

237‧‧‧Webpage fingerprint processing unit

250‧‧‧Network Information Organizer

300‧‧‧Database storage device

Claims (7)

A Chinese network information monitoring and analysis system is used for monitoring and analyzing on the Internet according to at least one Chinese target information submitted by the user's electronic communication device, including: a word storage device, and a plurality of Chinese stored in the system The word segment information, the plural Chinese synonym information, and the plural Chinese word information; a computer computing device is connected to the thesaurus storage device, comprising: a target information processing module, receiving the at least one Chinese target information; Processing the analysis module, searching and generating the analysis result on the internet, comprising: a network information acquisition unit, which is used for extracting webpage data on the internet; a word segmentation processing unit is used Performing the first-order processing of the word segmentation sentence according to the punctuation marks and the Chinese word segment table information, and then performing the second-order processing according to the maximum matching method to generate the word segmentation result of the corresponding webpage; a word frequency processing unit is based on the Chinese synonym information and the Chinese implied word information Counting the frequency of occurrence of the corresponding word to generate a word frequency counting result of the corresponding webpage; and a webpage fingerprint processing unit for classifying the webpage attribute of the captured webpage data, which is the webpage of the captured webpage The label of the source code is used as a node for segment cutting, and the TF/IDF weight analysis component is used according to the word frequency counting result to generate the weight of each word segment of the corresponding webpage, and the pre-scheduled selection is selected from the arrangement of the weights to the smallest. The number of word segments are rearranged according to their characters to generate segmentation information of one of the corresponding web pages, and finally the selected characters are converted into the number of reserved bits according to the hash algorithm. a string of codes, which in turn generates a serial number of the corresponding webpage; and a network information collation module, which selects a corresponding category in a database storage device according to the Chinese target information and compares the word segmentation information of the captured webpage The page of the web page is retrieved to generate a web page capture data for providing the retrieved web page capture data; wherein the database storage device is coupled to the computer computing device, The webpage corresponding to the webpage is used to store the retrieved data and the word segmentation information of the webpage to classify the captured webpage data. For example, the Chinese network information monitoring and analysis system described in the first application of the patent scope, wherein the network information retrieval unit is further configured to log in to the web platform that needs to log in information according to the preset login information. . For example, the Chinese network information monitoring and analysis system described in claim 1 wherein the maximum matching method used by the word segmentation processing unit includes a forward matching method and a maximum matching method. A Chinese network information monitoring and analysis method is used for monitoring and analyzing on the Internet according to the input Chinese target information, and includes the following steps: capturing webpage data on the Internet; performing word segmentation of the webpage data Processing, using the Chinese word segmentation information in the punctuation and the preset vocabulary storage device to perform the first-order processing of the word segmentation sentence, and performing the second-order processing by the maximum matching method to generate a word segmentation result; performing the webpage The word frequency processing of the data, the Chinese synonym information and the Chinese implied word information in the vocabulary storage device are preset to count the words and the frequency of the corresponding words in the lexicon storage device in the result of the word segmentation to generate a Word frequency count result; Performing webpage fingerprint processing, firstly using the label of the webpage source code of the retrieved webpage data as a node of the paragraph cutting, and using the TF/IDF weight analysis component according to the word frequency counting result, to generate the weight of each word segment, and The pre-determined number of word segments are selected from the arrangement of the largest and smallest weights, and are rearranged according to the characters to generate a piece of word information, and finally the selected characters are converted into the number of reserved bits according to the hash algorithm. Serial code, which further generates a serial code corresponding to the captured webpage data; stores the word segmentation information and the serial code corresponding to the webpage data; and in the stored segmentation information and the serial code, selects a corresponding category according to the Chinese target information and compares The segmentation information of the captured webpage is retrieved from the webpage of the webpage to obtain a webpage retrieval data to provide the captured webpage retrieval data. For example, the Chinese network information monitoring and analysis method described in claim 4, wherein the step of extracting webpage data on the Internet further includes the step of extracting webpage data in a web platform that needs to log in information. For example, in the Chinese network information monitoring and analysis method described in claim 4, in the step of performing webpage fingerprint processing, the predetermined predetermined number of word segmentation is selected as the first ten. For example, in the Chinese network information monitoring and analysis method described in claim 6, wherein in the step of performing webpage fingerprint processing, the number of reserved bits is 128 bits.