CN102789479A - Vocabulary relevance calculating method on basis of semantic analysis of search result - Google Patents

Abstract

The invention belongs to the technical field of computational linguistics, and specifically relates to a calculation method for vocabulary relevancy based on semantic analysis of search results. The method of the invention automatically submits a search command to an Internet search engine according to a certain search strategy to obtain a search result, and uses Web page information extraction technology and text semantic analysis technology to calculate the co-occurrence degree of words, thereby obtaining the correlation degree of words. The invention avoids building and maintaining the knowledge base system locally, and the calculation result of the correlation degree can reflect the impact on the time dimension, and can also perform the calculation of the correlation degree of the mixed vocabulary containing numbers well. This method is suitable for various application occasions that require semantic relevance of words.

Description

A Lexical Correlation Calculation Method Based on Semantic Analysis of Search Results

technical field

The invention belongs to the technical field of computational linguistics, and in particular relates to a quantitative calculation method for vocabulary correlation. the

Background technique

With the development of Web2.0 application technology, various blogs, online forums, and socialized interactive media continue to appear, and a large amount of text information is generated every day, such as various news reports, product introductions, product reviews, and so on. Among them, whether it is finding hot spots from news reports or automatically analyzing opinions from product reviews, a more critical technical issue needs to be involved, that is, how to effectively calculate the correlation between two words. Therefore, calculating lexical correlation has become a key basic problem in network text information processing. There are currently several methods for computing lexical relevance. One is a statistical method based on a large-scale corpus knowledge base. First, a set of feature words is determined, and the vocabulary is represented as a point in the vector space defined by this set of feature words, and then the similarity of the cosine of the included angle is used to Calculate [6, 7]. The second method is to use a semantic dictionary to express the vocabulary as a set of semantic vectors according to the organizational structure of the semantic dictionary to calculate the relevance of a given vocabulary. Commonly used dictionaries include WordNet, HowNet, etc. [3, 5]. Another method is to map vocabulary to a semantic space with a smaller dimension based on LSA (latent semantic analysis), and calculate the correlation in the semantic space. This method is based on matrix SVD (singular value decomposition) decomposition. Recently, with the continuous improvement of various encyclopedias (such as Wikipedia) on the Internet, the use of such knowledge bases for lexical correlation computation has also received attention, which explicitly represent text or vocabulary as bands in the concept space of Wikipedia. Weight vector [1, 4]. the

Although these methods can calculate lexical correlation better in some occasions, in specific applications, a large number of feature space construction calculations are required, and knowledge bases need to be maintained and updated. The dependence of the system is relatively large, which leads to unsatisfactory applications in reality. Specifically, the existing problems are listed as follows:

1. The problem of constructing lexical semantic vectors. Elements in a semantic vector are words selected from a semantic dictionary or corpus, representing the set of words relevant to the lexical comparison to be computed. It is necessary to perform feature analysis and calculation in larger text information, especially for Chinese applications, and word segmentation processing is also required. For mixed words such as "113 meter column", it is easy to cause deviations in vector calculation.

2. Need to build a huge knowledge base system. The method of statistical calculation of correlation needs to build and maintain a huge knowledge base system, and requires more extra processing in data storage and retrieval. the

3. Adaptability to different knowledge base systems. The knowledge base system is a basis for correlation calculation, but the current method mainly relies on the English Wikipedia system, and its concept feature extraction method is not suitable for Chinese vocabulary. Therefore, after switching to other systems, it is necessary to redefine the calculation method of semantic vectors, which also limits the practical value of this method to a certain extent. the

It can be seen that when performing semantic correlation of vocabulary, it is very necessary for the application of correlation calculation methods to consider the practical problems of knowledge base construction and maintenance, and to improve the calculation ability of different types of vocabulary correlation. the

Contents of the invention

The purpose of the present invention is mainly aimed at the deficiencies in existing various lexical correlation calculation methods, and proposes a lexical correlation calculation method based on the combination of search engine technology and text semantic analysis technology. the

The method for calculating the correlation degree of words proposed by the present invention automatically submits search commands to Internet search engines according to a certain search strategy, obtains search results, and uses Web page information extraction technology and text semantic analysis technology to calculate the degree of word co-occurrence, thereby Get word relevance. the

The method for calculating the degree of vocabulary relevance proposed by the present invention, the concrete steps are as follows:

(1) Set the two words w1, w2 that need to be calculated for correlation, and the threshold ξ of the number of records;

(2) According to whether the vocabulary is Chinese or English, generate a search command that matches www.bing.com, and specify it as a search that limits the scope of the website, and the scope is set to en.wikipedia.org or baike.baidu.com;

(3) Automatically establish an HTTP (Hypertext Transfer Protocol) network connection, and send retrieval commands to the bing search system through this connection;

(4) Receive and process the returned search results, that is, HTML (Hypertext Markup Language) text information. When the records on one page are processed, the search records on the next page are automatically processed until all search records are processed or Until a certain number of records is reached; Web information extraction technology [2] is used to automatically obtain the search records on the page, and the vocabulary frequency information is counted based on the abstract text in each search record;

(5) Based on the word frequency information obtained by statistics, calculate the correlation between two words, and prompt relevant information.

The flow chart of the present invention is shown in Figure 1. the

In the present invention, in step (2), the Bing search engine is used as the vocabulary context information acquisition means, and en.wikipedia.org or baike.baidu.com is used as the Chinese-English knowledge base. the

In the present invention, step (3) establishes an HTTP network connection, and establishes a search command that meets the requirements, and sends the search command to the bing search system through the connection. the

In the present invention, step (4) extracts each record in the search result page, extracts the summary text information therein, and performs text segmentation according to the separator "..." to obtain several segments. Perform vocabulary frequency information statistics for each segment. the

In the present invention, step (4) determines whether to obtain more records according to the condition endRec<TotalRec and whether the condition Trec is less than the set record number threshold ξ holds true. Among them, TotalRec indicates the total number of records in the search results, endRec indicates the number of records on the current page, and the number of records that Trec has processed. the

In the present invention, step (5) calculates the correlation degree of two words w1, w2 by the following formula:

R(w1, w2) = TC*2 / (T1+T2)

Among them, T1 is the number of occurrences of word w1, T2 is the number of occurrences of word w2, and TC is the number of occurrences of both.

In the present invention, by constructing the training set, the similarity calculation result and the labeling result are calculated by the Pearson correlation coefficient, so as to determine the threshold value of the number of records required in the calculation process. the

The present invention has substantive features and significant progress: (1) Using a search engine system avoids building a large knowledge base system locally. The existing Wikipedia-based correlation calculation method needs to download all the content of the website, and new problems will arise in terms of storage space and system maintenance and updates. Based on the processing ability of the search engine technology in vocabulary matching, the present invention can not only obtain relevant information for correlation degree calculation, but also does not need to establish and maintain similar local information databases. At the same time, by changing the retrieval scope, the correlation calculation based on different knowledge base systems can be easily realized, which can be adapted to the calculation of English vocabulary and Chinese vocabulary; (2) No need for complex semantic analysis, based on relative Simple vocabulary co-occurrence calculations are used to obtain vocabulary correlations and construct semantic vectors, which can be applied to the correlation calculations of mixed vocabulary such as numbers. And for the calculation of Chinese vocabulary, word segmentation and other processing are not required. (3) Since the search engine system will automatically update the recently modified website content, the correlation calculation method based on search engine results can more effectively reflect the correlation of two words in the time dimension, and has the ability of time perception. the

The present invention uses search engine technology and simple semantic analysis technology to establish a vocabulary correlation calculation method, which avoids building and maintaining a knowledge base system locally, and the correlation calculation result can reflect the influence of the time dimension. For mixed vocabulary correlation calculations containing numbers Also works well. This method is suitable for various application occasions that require semantic relevance of words. the

Description of drawings

Fig. 1 is a flow chart of the present invention. the

Detailed ways

(1) Set the two words w1, w2 that need to be calculated for correlation, and the correlation threshold ξ. the

(2) If the vocabulary is Chinese, UTF8 encoding is performed on the vocabulary first. And specify the search scope: en.wikipedia.org, baike.baidu.com. Based on this information, a search command for the bing search engine (www.bing.com) is generated. the

(3) Establish an HTTP network connection, and send retrieval commands to the bing search system through this connection. the

(4) Initialize variable values: T1=0, T2=0, TC=0, Trec=0. the

(5) Receive and process the results returned by the search engine. Content extraction is performed on the HTML text information of the page content. Use the regular expression "[0-9,]+ - [0-9,]+ results\\(total [0-9,]+ results\\)" to extract the total records of the search results prompted on the page and the current The record number range of the page, and recorded into the variables TotalRec and beginRec, endRec three variables. the

(6) Locate and extract each record of the current page according to the delimiter "<li class=\"sa_wr\"><div class=\"sa_cc\">" between the retrieved records, and extract the summary text information in it . Segment the text according to the text delimiter "..." to get several segments. Perform vocabulary frequency information statistics for each segment. the

If w1 appears in the segment, the number T1 of w1 increases by 1; if w2 appears in the segment, the number T2 of w2 increases by 1; if w1 and w2 appear in the segment at the same time, the co-occurrence frequency TC increases by 1 . the

Increment the number of records Trec already processed by 1. the

(7) Determine whether the conditional condition Trec<ξ holds true. If not established, go to step (9) to execute. the

(8) Determine whether the condition endRec<TotalRec is true, that is, whether the last page is reached. If not, generate a retrieval command to obtain the content of the next page, send it to the bing search system, and repeat steps (5), (6), (7), and (8). Otherwise, it will prompt "Insufficient information, correlation calculation cannot be performed.", and end the processing flow. the

(9) Calculate the correlation between the two words according to the following formula:

R(w1, w2) = TC*2 / (T1+T2)

Method of setting threshold ξ: It is necessary to determine a vocabulary set, including several vocabulary pairs and correlation labeling results X. Based on these word pairs, select different retrieval ranges, adjust the value of the threshold ξ, obtain the correlation calculation result Y , and calculate the Pearson correlation coefficient r of the two sets of X and Y :

where n is the number of elements in the set. The value range of r is [-1, +1]. When the correlation coefficient r reaches a reasonable range (generally when r is greater than 0.4), it means that the selected calculation parameter ξ is acceptable.

It can be seen from the above implementation process that the present invention introduces simple semantic processing based on search engine retrieval results, which can obtain relevant information for correlation calculation and does not need to establish and maintain similar local information databases. The calculation method based on search engine results can more effectively reflect the correlation of two words in the time dimension, and can be applied to the calculation of mixed words containing numbers. Using the method based on Pearson correlation calculation to determine the maximum number of records that need to be retrieved can more reasonably calculate the vocabulary correlation according to the set threshold. the

Concrete example:

Assuming that the two words to be calculated are "doctor" and "nurse", and the English Wiki en.wikipedia.org is used as the search scope, the following initial search commands are automatically generated and searched through the HTTP connection.

http://cn.bing.com/search?q=site%3aen.wikipedia.org+doctor+%26+nurse&qs=n&pq=site%3aen.wikipedia.org+doctor+%26+nurse&sc=0-0&sp=-1&sk =&first=1

Then by automatically increasing the last first=1 in the command, the correlation degree is 0.6613 when there are 60 records.

As another example, if the two words to be calculated are "computer" and "computer", if Baidu Encyclopedia baike.baidu.com is used as the search range, the following initial search command will be automatically generated and searched through the HTTP connection. the

http://cn.bing.com/search?q=site%3abaike.baidu.com+%22%e8%ae%a1%e7%ae%97%e6%9c%ba%22+%26+%22% e7%94%b5%e8%84%91%22&qs=n&pq=site%3abaike.baidu.com+%22%e8%ae%a1%e7%ae%97%e6%9c%ba%22+%26+% 22%e7%94%b5%e8%84%91%22&sc=0-0&sp=-1&sk=&first=1

Then by automatically adding first=1 at the end of the command, after retrieving 140 records, the obtained vocabulary correlation is 0.9111.

references:

[1] E. Gabrilovich, S. Markovitch. Computing Semantic Relatedness Using Wikipedia-based Explicit Semantic Analysis. In Proceedings of the 20th International Joint Conference on Artificial Intelligence, 2007.

[2] X. W. Ji, J. P. Zeng, S. Y. Zhang, C. R. Wu. Tag Tree Template for Web Information and Schema Extraction. Expert Systems With Applications, 2010,37(12): 8492 -8498.

[3] A. Budanitsky, G. Hirst. Evaluating WordNet-based Measures of Lexical Semantic Relatedness. Computational Linguistics, 2006, 32(1):13-47.

[4] M. Strube and S. P. Ponzetto. WikiRelate! Computing Semantic Relatedness Using Wikipedia. In AAAI’06, 2006.

[5] Jiang Min, Xiao Shibin, Wang Hongwei, Shi Shuicai. An improved word semantic similarity calculation based on "HowNet". Chinese Journal of Information, 2008, No.5.

[6] Lu Song. Unguided Acquisition of Word Relevance Knowledge in Natural Language and Construction of Balanced Classifier, Doctoral Thesis of Institute of Computing Technology, Chinese Academy of Sciences, 2001.

[7] Liu Qun, Li Sujian. Lexical Semantic Similarity Calculation Based on "HowNet" [J]. Chinese Computational Linguistics, 2002,7(2):59-76.

Claims (5)

1. vocabulary relatedness computation method based on the Search Results semantic analysis is characterized in that concrete steps are following:

(1) setting need be carried out two vocabulary w1 of relatedness computation, w2, and record number threshold xi;

(2) be Chinese or English according to vocabulary, generation meets the retrieval command of www.bing.com, and is appointed as the retrieval that limits the website scope, and scope is set at en.wikipedia.org or baike.baidu.com;

(3) set up the HTTP network automatically and connect, connect through this and send retrieval command to the bing search system;

(4) receive and handle the result for retrieval that is returned; It is the HTML text message; After the recording processing on the page finished, the search records that automatically performs down one page was handled, till all search records dispose or reach certain record number; Adopt the Web information extraction technology to obtain the search records on the page automatically, based on the statistics of the summary texts in each search records vocabulary frequency information;

(5) the vocabulary frequency information that obtains based on statistics calculates the degree of correlation of two vocabulary, and points out relevant information.

2. the vocabulary relatedness computation method based on the Search Results semantic analysis as claimed in claim 1; It is characterized in that: the method for the said statistics vocabulary of step (4) frequency information is: extract each record in the result for retrieval page; Extract summary texts information wherein; And according to decollator " ... " Carry out text dividing, obtain several segmentations; Carry out the Information Statistics of the vocabulary frequency for each segmentation.

3. the vocabulary relatedness computation method based on the Search Results semantic analysis as claimed in claim 1; It is characterized in that: in the step (4); According to condition endRec < TotalRec, and whether condition Trec count threshold xi less than the record of setting and set up, and determines whether obtaining more record; Wherein, TotalRec representes the total number of records of result for retrieval, and endRec representes the record number of current page, the record number that Trec had handled.

4. the vocabulary relatedness computation method based on the Search Results semantic analysis described in claim 1 is characterized in that: calculate two vocabulary w1 through following formula in the step (5), the degree of correlation of w2:

R（w1,?w2）=?TC*2?/?(T1+T2)

Wherein, T1 is the number of times that w1 occurs, and T2 is the number of times that w2 occurs, and TC is the number of times that both occur simultaneously.

5. the vocabulary relatedness computation method described in claim 1 based on the Search Results semantic analysis; It is characterized in that: through the structure training set; Similarity result of calculation and annotation results are carried out the calculating of Pearson correlation coefficient, thereby threshold xi counted in the record that needs in definite computation process.

Images