CN102799586A - Transferred meaning degree determining method and device for sequencing searching result - Google Patents

Abstract

The present invention provides a method and device for determining the escape degree used for sorting search results, wherein the method includes: A. analyzing the closeness of the search request input by the user, and determining the closeness of each word pair in the search request ; B. According to the result of structural information processing of each webpage in the search results corresponding to the search request, count the physical distance distribution of each word in the search request in each of the webpages; C. use the search The closeness corresponding to each word pair in the request and the physical distance distribution in each web page determine the escape degree of each web page in the search result for the search request, and the escape degree is used for the search The pages in the results are sorted. Using the escape degree determined by the present invention to sort the search results can improve the sorting effect of the search results, thereby saving network resources.

Description

Method and device for determining escape degree for sorting search results

【Technical field】

The invention relates to the field of computer technology, in particular to a method and device for determining the escape degree used for sorting search results.

【Background technique】

With the continuous development of computer technology, search engines have become the main means for people to obtain information. When the user inputs a search request query, the search engine will include pages matching the query in the search results and return it to the user.

The sorting of each page in the search results is based on the matching degree between the query input by the user and the page. In the current search technology, the matching degree usually only depends on the physical distance of each word in the query on the page. But in fact, there is a difference in the degree of closeness between the words contained in the query. In the same query, if the physical distance between the words with a high degree of density in page 1 and the words with a small degree of density in page 2 The same physical distance, then obviously the ranking of page 1 should be higher than the ranking of page 2, but this feature cannot be reflected in the ranking of search results in the current search technology, because the ranking of search results is not good, which leads to long-term search for users. Time takes up network resources, resulting in a waste of network resources.

【Content of invention】

The invention provides a method and device for determining the degree of escape for sorting search results, so as to improve the sorting effect of search results and save network resources.

The specific technical scheme is as follows:

A method for determining the degree of escape for sorting search results, the method comprising:

A. Analyze the closeness of the search request input by the user, and determine the closeness of each word pair in the search request;

B. According to the result of structural information processing of each web page in the search result corresponding to the search request, count the physical distance distribution of each word pair in the search request in each web page;

C. Using the closeness corresponding to each word pair in the search request and the physical distance distribution in each web page, determine the escape degree of each web page in the search result for the search request, and the escape degree Used to sort the web pages in the search results.

Wherein, the step A specifically includes:

A1. Perform word segmentation processing on the search request;

A2. Using the words obtained after the word segmentation process, determine each word pair in the search request;

A3. Query the pre-mined proper noun dictionary and/or co-occurrence dictionary to determine the closeness of each word pair, wherein the proper noun dictionary contains pre-mined proper nouns, and the co-occurrence dictionary contains pre-determined The co-occurrence status of each word pair in the existing data sources.

Preferably, the step A1 further includes: filtering the words obtained after the word segmentation process based on a stop vocabulary list.

Specifically, the step A2 includes:

Two adjacent words in the words obtained after the word segmentation process form word pairs; or,

Words with strong expressive ability among the words obtained after the word segmentation process are used to form word pairs, wherein the words with strong expressive ability are determined according to part of speech or sentence components in the search request.

In the step A3, querying the thesaurus excavated in advance to determine the compactness of each word pair specifically includes:

If the proper nouns in the dictionary of proper names include a word pair i, then the tightness of the word pair i is determined as a preset tightness value, and the word pair i is any word pair in the search request. one.

In the step A3, querying the pre-mined co-occurrence dictionary to determine the closeness of each word pair specifically includes:

Query the co-occurrence dictionary to determine the co-occurrence status of the word pair i in the existing data source, the co-occurrence status includes the number of occurrences of the word pair i in each distance range level, and the word pair i is in the search request any of the pairs of words;

Determine the distance range grade with the largest relative occurrence probability value of the word pair i in each distance range grade;

The determined closeness corresponding to the distance range level is taken as the closeness of the word pair i, where different distance range levels are preset to correspond to different closeness levels.

In addition, the mining of the co-occurrence dictionary specifically includes:

D1. After performing word segmentation processing on the data source and filtering based on the stop vocabulary list, the obtained words are combined in pairs to form a word pair;

D2. Count the co-occurrence status of the word pairs obtained in the step D1 in the data source, and store the co-occurrence status in the co-occurrence dictionary.

If in said step A3, both the thesaurus and the co-occurrence dictionary are used, and the closeness of the word pair i can be determined through the query of the thesaurus, then the closeness of the word pair i determined by querying the thesaurus The closeness is the closeness of the word pair i, and the word pair i is any one of the word pairs in the search request.

Specifically, the structural information processing of the webpage includes:

Divide web pages into web blocks, paragraphs and sentences;

The position information of each word in the web page is recorded and stored in the database, where the position information includes: the web page block, segment, sentence and sentence offset.

Based on this, the step B specifically includes:

B1. Determine the co-occurrence status of the word pair i in the web page d according to the position information of the two words of the word pair i in the search request recorded in the database, and the word pair i It is any one of the word pairs in the search request, and the web page d is any web page in the search results;

B2. According to the co-occurrence status determined in the step B1, the physical distance distribution of the word pair i in the webpage d is counted.

Described step C specifically comprises:

C1. Using the closeness of the word pair i in the search request to determine the weighted value weight(i) of the word pair i;

C2. Using the physical distance distribution of the word pair i in the web page d in the search results to determine the degree of satisfaction fit(i, d) of the web page d to the word pair i;

确定所述网页d针对所述搜索请求q的转义度offset_ratio(d，q)，其中，φ为所述搜索请求q中的词语对构成的集合。C3, according to the formula

Determine the offset_ratio(d, q) of the web page d for the search request q, where φ is a set of word pairs in the search request q.

The weight(i) is:

weight(i)=f1(tight(i), imp(i)); wherein tight(i) is the tightness of the word pair i, and imp(i) is the word pair i in the search request q The degree of importance, f1(tight(i), imp(i)) is a function that takes tight(i) as the main factor and imp(i) as the adjustment factor, in the case of the same imp(i) tight(i) The larger the value, the larger the weight(i) value; or,

weight(i)=f2(tight(i)), where f2(tight(i)) is a function for normalizing tight(i).

The imp(i) is determined by at least one of the following factors:

The part of speech of the word pair i in the search request, the sentence component of the word pair i in the search request, and the inverted document rate of the word pair i.

The fit(i, d) is:

fit(i, d)=f3(HIT(i, d), tight(i)); wherein HIT(i, d) identifies the physical distance distribution of the word i in the web page d that is counted, tight(i ) is the tightness of the word pair i, f3(HIT(i, d), tight(i)) is the distance range of the word pair i determined by HIT(i, d) as the main factor and tight( i) As a function of the adjustment factor, the smaller the distance range of the word pair i determined by HIT(i, d) under the same tight(i) is, the larger the value of fit(i, d) is; or,

fit(i,d)=f4(HIT(i,d)), f4(HIT(i,d)) is to map the distance range of the word pair i determined by HIT(i,d) into a specific fit(i , d) function of value.

Determining the distance range of the words to i by HIT (i, d) specifically includes:

Adopt the minimum distance range of the word pair i in the HIT (i, d) as the distance range of the word pair i; or,

According to the HIT(i, d), the distance range level with the largest relative occurrence probability value is used as the distance range level of the word pair i.

A device for determining the degree of escape for sorting search results, the device comprising: a closeness analysis unit, a distance distribution determination unit, and a degree of escape determination;

The closeness analysis unit is used to analyze the closeness of the search request input by the user, and determine the closeness of each word pair in the search request;

The distance distribution determining unit is configured to calculate the physical distance distribution of each word pair in the search request in each web page according to the result of structural information processing of each web page in the search result corresponding to the search request;

The escaping degree determination unit is configured to use the closeness corresponding to each word pair in the search request and the physical distance distribution in each webpage to determine the translation of each webpage in the search result for the search request. degree of meaning, and the degree of escape is used to sort the webpages in the search results.

Wherein, the closeness analysis unit specifically includes: a word segmentation processing subunit, a word pair determination subunit and a closeness determination subunit;

The word segmentation processing subunit is configured to perform word segmentation processing on the search request;

The word pair determination subunit is used to determine each word pair in the search request by using the words obtained after the word segmentation process;

The compactness determination subunit is used to query a pre-mined dictionary of proper names and/or a co-occurrence dictionary, and determine the compactness of each word pair, wherein the dictionary of proper names contains proper nouns mined in advance, The co-occurrence dictionary includes the pre-determined co-occurrence status of each word pair in the existing data source.

Preferably, the compactness analysis unit further includes: a filtering processing subunit, which is used to filter the words obtained after the word segmentation processing by the word segmentation processing subunit based on the stop vocabulary list, and send the filtered words to Subunits are determined for the word pairs.

Specifically, the word pair determination subunit forms word pairs in pairs of adjacent words among the words obtained after the word segmentation process; or,

Words with strong expressive ability among the words obtained after the word segmentation process are used to form word pairs, wherein the words with strong expressive ability are determined according to part of speech or sentence components in the search request.

If the proper noun in the dictionary of proper names contains a word pair i, the closeness determination subunit determines the closeness of the word pair i as a preset closeness value, and the word pair i is the search Any of the pairs of words in the request.

The compactness determination subunit specifically includes: a dictionary query module, a distance level determination module and a compactness determination module;

The dictionary query module is used to query the co-occurrence dictionary to determine the co-occurrence status of the word pair i in the existing data source, the co-occurrence status includes the number of occurrences of the word pair i in each distance range level, the word Pair i is any one of the word pairs in the search request;

The distance level determination module is used to determine the distance range level with the largest relative occurrence probability value of the word pair i in each distance range level according to the query result of the dictionary query module;

The closeness determination module is used to use the closeness corresponding to the distance range level determined by the distance level determination module as the closeness of the word pair i, wherein different distance range levels are preset to correspond to different closeness .

Furthermore, the closeness analysis unit also includes: a co-occurrence dictionary mining subunit, which is used to perform word segmentation processing on the data source and filter based on the stop vocabulary list, and combine the obtained words in pairs to form word pairs , counting the co-occurrence status of the obtained word pairs in the data source, and storing the co-occurrence status in the co-occurrence dictionary.

If the closeness determination subunit adopts both the thesaurus and the co-occurrence dictionary, and the closeness of the word pair i can be determined through the query of the thesaurus, then the word pair i determined by querying the thesaurus The closeness of is taken as the closeness of the word pair i, and the word pair i is any one of the word pairs in the search request.

Furthermore, the device also includes: a structural information processing unit, configured to divide the webpage into webpage blocks, paragraphs and sentences, record the position information of each word in the webpage and store it in the database, wherein the position information includes: Web page blocks, paragraphs, sentences and intra-sentence offsets.

The distance distribution determination unit specifically includes: a co-occurrence status determination subunit and a distance distribution statistics subunit;

The co-occurrence determination subunit is configured to determine the position of the word pair i in the webpage d according to the position information of the two words of the word pair i in the webpage d in the search request recorded in the database Co-occurrence situation, the word pair i is any one of the word pairs in the search request, and the web page d is any web page in the search results;

The distance distribution statistical subunit is configured to count the physical distance distribution of the word pair i in the web page d according to the co-occurrence status determined by the co-occurrence status determination sub-unit.

The escape degree determination unit specifically includes: a weighted value determination subunit, a satisfaction degree determination subunit and an escape degree determination subunit;

The weighted value determination subunit is used to determine the weighted value weight(i) of the word pair i by using the closeness of the word pair i in the search request;

The satisfaction determination subunit is used to determine the satisfaction degree fit(i, d) of the webpage d to the word pair i by using the physical distance distribution of the word pair i in the web page d in the search results ;

The escaping degree determines subunits for use according to the formula Determine the offset_ratio(d, q) of the web page d for the search request q, where φ is a set of word pairs in the search request q.

The weighted value determination subunit determines the weighted value weight(i) of the word pair i according to weight(i)=f1(tight(i), imp(i)) or weight(i)=f2(tight(i) );

Where tight(i) is the tightness of the word pair i, imp(i) is the importance degree of the word pair i in the search request q, and f1(tight(i), imp(i)) is the tight(i) as the main factor and imp(i) as the function of the adjustment factor, in the case of the same imp(i), the larger the value of tight(i) is, the larger the value of weight(i) is, f2(tight(i)) is a function to normalize tight(i).

In this case, the escape degree determination unit further includes: an importance determination subunit, configured to determine the imp(i) according to at least one of the following factors:

The part of speech of the word pair i in the search request, the sentence component of the word pair i in the search request, and the inverted document rate of the word pair i.

The satisfaction determination subunit determines the webpage according to fit(i, d)=f3(HIT(i,d), tight(i)) or fit(i,d)=f4(HIT(i,d)) The degree of satisfaction fit(i, d) of d to the word i;

Wherein HIT (i, d) marks the physical distance distribution of the said words to i in web page d, and tight (i) is the tightness of said words to i, f3 (HIT (i, d), tight ( i)) is a function of the distance range of the word pair i determined by HIT (i, d) as the main factor and tight (i) as the adjustment factor, under the same tight (i) situation by HIT (i, d) The smaller the distance range of the determined word pair i is, the larger the value of fit(i, d) is, and f4(HIT(i, d)) is the distance range mapping of the word pair i determined by HIT(i, d) is a function of specific fit(i, d) values.

In this case, the escape degree determination unit also includes: a distance range determination subunit, configured to determine the distance range of the word pair i according to the HIT(i, d), specifically including:

Adopt the minimum distance range of the word pair i in the HIT (i, d) as the distance range of the word pair i; or,

According to the HIT(i, d), the distance range level with the largest relative occurrence probability value is used as the distance range level of the word pair i.

It can be seen from the above technical solutions that the degree of escape determined by the method and device provided by the present invention is based on the closeness of each word pair in the query and the physical distance distribution in the webpage. High, indicating that the higher the degree of matching between the web page and the word pairs with high density in the query, the better the sorting result based on this, and the user can obtain the desired information more quickly by sorting the search results in this way, thus saving network resources .

【Description of drawings】

Fig. 1 is the main method flowchart provided by the embodiment of the present invention;

FIG. 2 is a flowchart of a method for performing compactness analysis on a query provided in Embodiment 1 of the present invention;

Fig. 3 is the flow chart of the method for counting the physical distance distribution of each word pair in the webpage in the statistics query provided by the second embodiment of the present invention;

Fig. 4 is the flow chart of the method for determining the escape degree of a webpage for query provided by the third embodiment of the present invention;

FIG. 5 is a structural diagram of an apparatus for determining the degree of escape provided by Embodiment 4 of the present invention.

【Detailed ways】

In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be described in detail below in conjunction with the accompanying drawings and specific embodiments.

Fig. 1 is the flow chart of the main method provided by the embodiment of the present invention, as shown in Fig. 1, the method may include the following steps:

Step 101: Analyze the closeness of the query input by the user, and determine the closeness of each word pair in the query.

Step 102: According to the result of structural information processing of each web page in the search result corresponding to the query, make statistics on the physical distance distribution of each word pair in the query in the web page.

Step 103: Utilize the corresponding closeness of each word pair in the query and the physical distance distribution in each web page of the search result to determine the degree of escape of each web page of the search result for the query, and the degree of escape is used for each page of the search result Pages are sorted.

Each step in the above method is described in detail below. Firstly, the above-mentioned step 101, that is, the process of analyzing the closeness of the query will be described in detail with reference to the embodiment.

Embodiment one,

FIG. 2 is a flow chart of a method for analyzing the compactness of a query provided by Embodiment 1 of the present invention. As shown in FIG. 2 , the method may include the following steps:

Step 201: Perform word segmentation processing on query.

The word segmentation processing method in this step can be adopted but not limited to: a method based on a dictionary and the longest match, or a method based on a statistical model, etc. Since the word segmentation processing is an existing mature technology, it will not be described in detail here.

Preferably, each word obtained after the word segmentation process can be further filtered based on the stop vocabulary list, and words with poor expressive ability, such as adverbs, function words, auxiliary words, etc., can be filtered out.

Take the query "who sang the family who loves each other" as an example, and the words obtained after word segmentation are: "love each other", "de", "a family", "yes", "who", "sing" and "of".

When filtering based on the stop word list, the "of" is filtered out, and the remaining words are: "love each other", "family", "is", "who" and "sing".

Step 202: Use the result of word segmentation to determine each word pair in the query.

When determining the word pairs in the query, at least one of the following strategies can be adopted:

Strategy 1: Among the words obtained after word segmentation, adjacent words form word pairs in pairs.

Strategy 2: Among the words obtained after word segmentation, the words with strong expressive ability form word pairs in pairs.

Among them, words with strong expressive ability can be determined according to part of speech or sentence components, for example, at least one of nouns, verbs, adjectives, and pronouns is determined as words with strong expressive ability, or at least one of subject, predicate, and object A word determined to be expressive.

Still taking the query "who sang the family who love each other" as an example, corresponding to strategy 1, the adjacent words in "love each other", "family", "yes", "who" and "sing" After two pairs of words are formed, the word pairs are obtained: "love each other"-"family", "family"-"yes", "yes"-"who", "who"-"sing".

Corresponding to strategy 2, among the words obtained after the query is segmented, the words with strong expressive ability are "love each other", "family" and "sing", and pair them into pairs: "love each other"-" "One family", "love each other" - "sing" and "one family" - "sing".

Step 203: Query the pre-mined proper name dictionary and/or co-occurrence dictionary, and determine the closeness of each word pair, wherein the proper name dictionary contains pre-mined proper nouns, and the co-occurrence dictionary contains predetermined words For co-occurrences in existing data sources.

Two kinds of dictionaries are involved in this step: proper name dictionaries and/or co-occurrence dictionaries. Among them, the excavation process of the proper name dictionary can adopt the existing technology. Currently, proper nouns can be divided into 18 types: personal names, place names, film and television names, country names, unit names, organization names, etc. Each type can be mined using its own corpus. For example, for the proper nouns of the film and television title type, the title of the video website can be used as the corpus for mining. Various types of excavation methods are not specifically limited here.

When using the thesaurus to determine the closeness of each word pair, if the proper noun in the thesaurus contains a certain word pair, then the closeness of the word pair can be determined as a preset closeness value. For example: the word pair composed of "love each other" and "a family" hits the proper noun "a family that loves each other" in the dictionary of proper names, that is, the proper nouns in the dictionary of proper names contain the pair of words, so you can set The closeness of the word pair formed by "loving each other" and "family" is the highest closeness.

The following describes the mining of co-occurrence dictionaries. The data source for mining co-occurrence dictionaries may be at least one of the following: web page content, web page titles, and queries in search logs. Carry out word segmentation processing on each data source respectively, preferably, after filtering out the words with poor expressive ability in the words obtained after the word segmentation processing based on the stop word table, combine them in pairs to form word pairs, and count the word pairs in the data source Co-occurrence status, and stored in the co-occurrence dictionary.

The co-occurrence status of each word pair in the co-occurrence dictionary can be stored as: the word pair, the co-occurrence distance range of the word pair, and the number of co-occurrence times within the distance range. Among them, the distance range can be preset to several levels, for example, divided into five levels: web page block, segment, sentence, within N words (N is an integer greater than 2, for example, within 3 words), adjacent.

When using the co-occurrence dictionary to analyze the closeness of the query input by the user, query the co-occurrence dictionary to determine the number of occurrences of each word pair in the query at each distance range level, and determine the distance with the largest relative occurrence probability value in each distance range level range level, the determined closeness corresponding to the distance range level is used as the closeness of the word pair. Wherein different distance range levels can be preset to correspond to different tightness.

For example, for the word pair "who"-"sing", in the co-occurrence dictionary, the number of co-occurrences of the adjacent level is 2, the number of co-occurrences of the level within 3 words is 10, and the number of co-occurrences of the sentence level is 18 , the number of co-occurrences at the segment level is 40, and the number of co-occurrences at the page block level is 60. Then determine that the distance range grade of the maximum relative occurrence probability value is: within 3 words, therefore, determine the co-occurrence distance range grade of the word pair of "who"-"singing" to be: within 3 words, this word pair The tightness level is the second tightness level.

Among them, the relative occurrence probability P _j of the jth level can be:

其中，x_j为词语对在第j个等级的共现次数，x_j+1为词语在第j+1个等级的共现次数，各等级按照紧密度从高到低排序。相对出现概率值也可以采用其它的定义，在此不做限制。

Among them, x _j is the number of co-occurrences of word pairs at level j, x _j+1 is the number of co-occurrences of words at level j+1, and each level is sorted from high to low in terms of closeness. Other definitions may also be used for the relative occurrence probability value, which is not limited here.

If the proper name dictionary and the co-occurrence dictionary are used at the same time, and a word pair hits the proper name dictionary and the co-occurrence dictionary at the same time, the proper name dictionary can be given a higher priority, that is, the word pair determined by querying the proper name dictionary The closeness of is used as the final closeness of the word pair.

In this embodiment, the closeness of each word pair can be expressed in the form of the above-mentioned different levels of closeness, or can be expressed in the form of a specific closeness value.

So far, the process shown in the first embodiment is completed, and the step 102, that is, the process of how to process the structure information of each web page in the search results will be described in detail below in conjunction with the second embodiment.

Embodiment two,

Fig. 3 is the flow chart of the method for statistically distributing the physical distances of the word pairs in the webpage in the second embodiment of the present invention. As shown in Fig. 3, the method may include the following steps:

Step 301: Perform structural information processing on each webpage in the search results corresponding to the query, the structural information processing includes: dividing webpage blocks, paragraphs, and sentences.

Wherein, the divided webpage blocks may include but not limited to: title (title) block, anchor (anchor) block, navigation (mypos) block, and content block. Among them, the anchor block and the content block can have a finer-grained division.

The divided web page blocks can be further segmented, and each segment can be further processed into sentences.

Step 301 can be carried out offline. After the above-mentioned structural information processing process of the webpage, each word has an absolute position in the webpage, and the position information of each word in each webpage can be stored in the database for online execution Query usage during step 302. Wherein, the location information may be the webpage block, segment, sentence and intra-sentence offset where each word is specifically located.

Step 302: According to the position information of each word in the web page, calculate the physical distance distribution of each word pair in the query in the web page.

According to the position information of the two words in the web page in each word pair in the query, the co-occurrence status of the word pair in the web page can be determined, that is, the number of co-occurrence times in the web page block, paragraph or sentence, because a word pair in the web page It may appear multiple times, and the physical distance distribution of word pairs in web pages can be counted based on the co-occurrence status of words in web pages, thereby forming an array HIT (i, d), where i identifies word pairs, d identifies web pages, and HIT (i, d) Identify the statistical distribution of the physical distance of the word pair i in the webpage d.

So far, the process shown in the second embodiment is completed, and the method for determining the escape degree of each web page for the query will be described in detail below in conjunction with the third embodiment.

Embodiment three,

Fig. 4 is the flow chart of the method for determining the escaping degree of the web page for the query provided by Example 3 of the present invention. As shown in Fig. 4, the method may include the following steps:

Step 401: Using the closeness of each word pair in the query and the importance of the word pair in the query, determine the weighted value weight(i) of the word pair.

Among them, weight(i)=f1(tight(i), imp(i)), tight(i) is the closeness of word pair i, and imp(i) is the importance degree of word pair i in the query. f1(tight(i), imp(i)) can be a function that uses tight(i) as the main factor and imp(i) as the adjustment factor. In the case of the same imp(i), the value of tight(i) is larger The larger the weight(i) value is. For example, the value obtained by normalizing imp(i) may be multiplied by tight(i).

Here is a specific implementation of one of f1(tight(i), imp(i)):

Firstly, the level corresponding to the tight(i) value is mapped to the corresponding weight value g_tight_map[tight(i)], where different levels corresponding to the tight(i) value can be mapped to different weight values, for example, assuming that tight(i) corresponds to The five levels correspond to integer values between [0, 4] and are mapped to weight values to form an array, assuming g_tight_map[5]={16, 8, 4, 2, 1}.

Then take weight(i)=f1(tight(i), imp(i))=g_tight_map[tight(i)]+imp(i), wherein, the value range of g_tight_map[tight(i)] can be greater than imp (i) value range, so as to realize tight(i) as the main factor and imp(i) as the adjustment factor.

imp(i) may determine the importance of word pair i in the query based on at least one of the following factors: part of speech in the query, or sentence components in the query, or inverted document rate (IDF).

Freq_i为所述词语对i在大规模语料库中的绝对词频，M为所有词语对在大规模语料库中的绝对词频的最大值。Among them, the inverted document rate IDF _i of word pair i is:

Freq _i is the absolute word frequency of the word pair i in the large-scale corpus, and M is the maximum value of the absolute word frequencies of all word pairs in the large-scale corpus.

In addition, it is also possible to determine the weighted value of the word pair only by using the closeness of each word pair in the query, that is, weight(i)=f2(tight(i)), at this time, f2(tight(i)) can be the pair of tight( i) A function for normalization processing.

Step 402: Using the physical distance distribution of each word pair in the query in the webpage and the closeness of each word pair, respectively determine the satisfaction degree fit(i, d) of each word pair on the webpage.

Wherein fit(i, d)=f3(HIT(i, d), tight(i)), HIT(i, d) marks the physical distance distribution of the word i in the web page d that has been counted, and tight(i) is The closeness of words to i. Specifically, f3(HIT(i, d), tight(i)) can use the distance range of the word pair i determined by HIT(i, d) as the main factor, and tight(i) as a function of the adjustment factor, in In the case of the same tight(i), the smaller the distance range of the word pair i determined by HIT(i, d), the larger the value of fit(i, d).

The following is a specific implementation of one of f3 (HIT(i, d), tight(i)):

HIT(i, d) reflects the physical distance distribution of word pair i in webpage d, which can be understood as the number of co-occurrences in each physical distance range. Suppose HIT(i, d) is an array hit[5], hit [0] represents the number of adjacent co-occurrences of word pair i in web page d; hit[1] represents the co-occurrence frequency of 3 words in web page d; hit[2] represents the co-occurrence frequency of three words in web page d times; hit[3] represents the number of co-occurrences in the middle segment of web page d; hit[4] represents the number of co-occurrences in the block of web page d. tight(i) is the closeness of word pair i, assuming that tight(i) is an integer value in the range of [0, 4].

First, HIT(i, d) can be quantized into a distance range value, then the calculated value hit_value=16*hit[0]+8*hit[1]+4*hit[2]+2*hit[1]+hit [0]. The value range of hit_value can be defined as [0, 16]. If the calculated hit_value is greater than 16, the value is 16 directly.

Predefine the combination of each tight(i) value and hit_value value to map to different fit(i, d), the mapping relationship can be embodied as a two-dimensional array g_hit_map_fit[tight(i)][hit_value], such as g_hit_map_fit[5] [17], the values in the two-dimensional array can be floating point numbers in the range of [0, 1]. That is, fit(i, d)=f3(HIT(i, d), tight(i))=g_hit_map_fit[tight(i)][hit_value].

When the distance range of word pair i is determined by HIT (i, d), the minimum distance range of word pair i in HIT (i, d) can be directly used as the distance range of word pair i, or it can be based on HIT (i, d ) takes the distance range level with the largest relative occurrence probability value as the distance range level of the word pair i.

In addition, it is also possible to determine fit(i, d) only by the distance range of words determined by HIT(i, d), that is, fit(i, d)=f4(HIT(i, d)), at this time , f4(HIT(i, d)) may be a function that maps the distance range of the word pair i determined by HIT(i, d) into a specific fit(i, d) value. For example, different distance range levels are corresponding to different fit(i, d) values in advance, after the distance range level of the word pair i determined by HIT(i, d), determine the fit(i , d) value.

Step 403: Using the weighted value of each word pair in the query and the degree of satisfaction of each word pair in the web page, determine the offset_ratio(d, q) of the web page for the query.

offset_ratio(d，q)为网页d针对query q的转义度，φ为q中的词语对构成的集合。in,

offset_ratio(d, q) is the escape degree of web page d for query q, and φ is a set of word pairs in q.

After determining the escaping degree of each web page for the query in the search results of the query, the search results may be sorted in descending order of the escaping degree. The higher the escaping degree of the web page for the query, the higher the degree of matching between the web page and the word pairs with high density in the query, and the better the sorting result based on this.

The above is the description of the method provided by the present invention, and the device provided by the present invention will be described in detail below.

Embodiment four,

FIG. 5 is a structural diagram of a device for determining the degree of escape provided by Embodiment 4 of the present invention. The device can be set on the server where the search engine is located, or can be set on other servers that can interact with the search engine. As shown in FIG. 5 , the device may include: a closeness analysis unit 500 , a distance distribution determination unit 510 and a meaning degree determination unit 520 .

The closeness analysis unit 500 analyzes the closeness of the query input by the user, and determines the closeness of each word pair in the query.

The distance distribution determining unit 510 calculates the physical distance distribution of each word pair in the query in each webpage of the search result according to the result of structural information processing performed on each webpage in the search result corresponding to the query.

The distance distribution determining unit 510 may acquire the search result corresponding to the query from the search engine.

The degree of escape determination unit 520 uses the closeness of each word pair in the query and the physical distance distribution in each webpage to determine the degree of escape of each webpage in the search results for the query, and the degree of escape is used to determine the degree of escape for each webpage in the search results. Sort.

Wherein, the closeness analysis unit 500 may specifically include: a word segmentation processing subunit 501 , a word pair determination subunit 502 and a closeness determination subunit 503 .

The word segmentation processing subunit 501 performs word segmentation processing on the query. The word segmentation processing method adopted may include but not limited to: a method based on a dictionary and the longest match, or a method based on a statistical model.

The word pair determination subunit 502 determines each word pair in the query by using the words obtained after word segmentation.

The compactness determination subunit 503 queries the pre-mined proper name dictionary and/or co-occurrence dictionary, and determines the closeness of each word pair, wherein the proper name dictionary contains pre-mined proper nouns, and the co-occurrence dictionary contains pre-determined each The co-occurrence of word pairs in existing data sources.

Preferably, the closeness analysis unit 500 may further include: a filter processing subunit 504 arranged between the word segmentation processing subunit 501 and the word pair determination subunit 502 . The filtering processing subunit 504 filters the words obtained after the word segmentation processing by the word segmentation processing subunit 501 based on the stop vocabulary, and sends the filtered words to the word pair determination subunit 502 . The word pair determining subunit 502 determines each word pair in the query using the words filtered by the filtering processing subunit 504 .

When the word pair determination subunit 502 determines each word pair in the query, it can form a word pair in pairs of adjacent words in the words obtained after the word segmentation process; Two pairs of words form a word pair, and the words with strong expressive ability are determined according to the part of speech or the sentence composition in the query.

When the compactness determining subunit 503 uses the thesaurus to determine the compactness of each word pair, if the proper noun in the thesaurus contains the word pair i, the compactness determining subunit 503 can calculate the compactness of the word pair i Determined as the preset closeness value, the word pair i is any one of the word pairs in the query, and the use of the thesaurus is not shown in FIG. 5 .

The excavation process of the proper name dictionary can adopt the method of the existing technology. Currently, proper nouns can be divided into 18 types: personal names, place names, film and television names, country names, unit names, organization names, etc.

When the closeness determination subunit 503 uses the co-occurrence dictionary to determine the closeness of each word pair, the closeness determination subunit 503 may specifically include: a dictionary query module 5031 , a distance level determination module 5032 and a closeness determination module 5033 .

The dictionary query module 5031 queries the co-occurrence dictionary to determine the co-occurrence status of the word pair i in the existing data source, and the co-occurrence status includes the occurrence times of the word pair i in each distance range level.

The distance level determination module 5032 determines the distance range level with the largest relative occurrence probability value of the word pair i in each distance range level according to the query result of the dictionary query module 5031 .

The closeness determination module 5033 uses the closeness corresponding to the distance range level determined by the distance level determination module 5032 as the closeness of the word pair i, where different distance range levels are preset to correspond to different closeness levels.

In order to realize the offline mining of co-occurrence dictionaries, the closeness analysis unit 500 may also include: a co-occurrence dictionary mining subunit 505, which performs word segmentation processing on the data source and filtering based on the stop vocabulary, and then combines the obtained words in pairs Form word pairs, count the co-occurrence status of the word pairs in the data source, and store the co-occurrence status in the co-occurrence dictionary.

The data sources used therein may include but not limited to: web page content, web page title, and query in search logs.

The co-occurrence status of each word pair in the co-occurrence dictionary can be stored as: the word pair, the co-occurrence distance range of the word pair, and the number of co-occurrence times within the distance range. Among them, the distance range can be preset to several levels, for example, divided into five levels: web page block, paragraph, sentence, within N words and adjacent, where N is an integer greater than 2.

If the closeness determining subunit 503 adopts the proper name dictionary and the co-occurrence dictionary at the same time, the closeness of the word pair i can be determined through the query of the proper name dictionary, then the closeness of the word pair i determined by querying the proper name dictionary is used as The closeness of words to i.

In order to realize that the distance distribution determination unit 510 counts the physical distance distribution of each word pair in the query in each webpage of the search result, the device may also include: a structural information processing unit 530, which is used to divide the webpage into webpage blocks, paragraphs and sentences, The position information of each word in the web page is recorded and stored in the database, where the position information includes: the web page block, segment, sentence and sentence offset.

The division of web page blocks involved in this embodiment includes but not limited to: title block, anchor block, mypos block or content block. Among them, the anchor block and the content block can have a finer-grained division.

Based on this, the distance distribution determining unit 510 may specifically include: a co-occurrence status determining subunit 511 and a distance distribution statistics subunit 512 .

The co-occurrence status determining subunit 511 determines the co-occurrence status of the word pair i in the web page d according to the position information of the two words of the word pair i in the query recorded in the database, and the web page d is the web page d in the search results. any web page.

The distance distribution statistics subunit 512 calculates the physical distance distribution of the word pair i in the web page d according to the co-occurrence status determined by the co-occurrence status determination sub-unit 511 .

The structure of the escape degree determination unit 520 is described in detail below. The escape degree determination unit 520 may specifically include: a weight value determination subunit 521 , a satisfaction degree determination subunit 522 and an escape degree determination subunit 523 .

The weighted value determining subunit 521 determines the weighted value weight(i) of the word pair i by using the closeness of the word pair i in the query.

The satisfaction degree determining subunit 522 determines the degree of satisfaction fit(i, d) of the word pair i on the web page d by using the physical distance distribution of the word pair i in the webpage d in the search results.

确定网页d针对query q的转义度offset_ratio(d，q)，其中，φ为query q中的词语对构成的集合。Escape degree determines subunit 523 according to the formula

Determine the offset ratio offset_ratio(d, q) of web page d for query q, where φ is a set of word pairs in query q.

Wherein, the weighted value determination subunit 521 can determine the weighted value weight(i ).

tight(i) is the tightness of word pair i, imp(i) is the importance of word pair i in query q, f1(tight(i), imp(i)) is tight(i) as the main factor and Using imp(i) as a function of the adjustment factor, in the case of the same imp(i), the larger the value of tight(i) is, the larger the value of weight(i) is, and f2(tight(i)) is to normalize tight(i) A function for unification.

At this point, the escape degree determination unit 520 may also include: an importance determination subunit 524, which is used to determine imp(i) according to at least one of the following factors: the part of speech of the word pair i in the query, the word pair i in the query The sentence components in and the inverted document rate of word pair i.

The degree of satisfaction determination subunit 522 can determine the pair of web pages d according to fit (i, d) = f3 (HIT (i, d), tight (i)) or fit (i, d) = f4 (HIT (i, d)). The degree of satisfaction fit(i,d) of words to i.

Among them, HIT(i, d) identifies the physical distance distribution of the word pair i in webpage d, tight(i) is the tightness of word pair i, and f3(HIT(i, d), tight(i)) is The distance range of the word pair i determined by HIT(i,d) is used as the main factor and tight(i) is used as a function of the adjustment factor, and the words determined by HIT(i,d) under the same tight(i) The smaller the distance range to i, the larger the fit(i, d) value, f4(HIT(i, d)) is to map the distance range of the word pair i determined by HIT(i, d) to a specific fit(i , d) function of value.

At this point, the escape degree determination unit 520 can also include: a distance range determination subunit 525, which is used to determine the distance range of the word pair i according to HIT (i, d), specifically can include:

The minimum distance range of word pair i in HIT(i, d) is used as the distance range of word pair i; or, according to HIT(i, d), the distance range grade with the largest relative occurrence probability value is used as the distance range grade of word pair i .

After the device shown in Figure 5 determines the escape degree of each webpage in the search results for query, the escape degree can be provided to the search engine for sorting each webpage in the search results, the higher the query escape degree The higher the ranking of the web page.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included in the present invention. within the scope of protection.

Claims (30)

1. A method for determining the escape degree for sorting search results, characterized in that the method comprises: A. Analyze the closeness of the search request input by the user, and determine the closeness of each word pair in the search request; B. According to the result of structural information processing of each web page in the search result corresponding to the search request, count the physical distance distribution of each word pair in the search request in each web page; C. Using the closeness corresponding to each word pair in the search request and the physical distance distribution in each web page, determine the escape degree of each web page in the search result for the search request, and the escape degree Used to sort the web pages in the search results. 2. The method according to claim 1, wherein said step A specifically comprises: A1. Perform word segmentation processing on the search request; A2. Using the words obtained after the word segmentation process, determine each word pair in the search request; A3. Query the pre-mined proper noun dictionary and/or co-occurrence dictionary to determine the closeness of each word pair, wherein the proper noun dictionary contains pre-mined proper nouns, and the co-occurrence dictionary contains pre-determined The co-occurrence status of each word pair in the existing data sources. 3. The method according to claim 2, characterized in that, in the step A1, further comprising: filtering the words obtained after the word segmentation process based on a stop vocabulary list. 4. The method according to claim 2, wherein said step A2 specifically comprises: Two adjacent words in the words obtained after the word segmentation process form word pairs; or, Words with strong expressive ability among the words obtained after the word segmentation process are used to form word pairs, wherein the words with strong expressive ability are determined according to part of speech or sentence components in the search request. 5. method according to claim 2, is characterized in that, in the described step A3, inquire about the thesaurus that excavates in advance and determine that the tightness of each word pair specifically includes: If the proper nouns in the dictionary of proper names include a word pair i, then the tightness of the word pair i is determined as a preset tightness value, and the word pair i is any word pair in the search request. one. 6. The method according to claim 2, characterized in that, in the step A3, querying the co-occurrence dictionary excavated in advance to determine the closeness of each word pair specifically includes: Query the co-occurrence dictionary to determine the co-occurrence status of the word pair i in the existing data source, the co-occurrence status includes the number of occurrences of the word pair i in each distance range level, and the word pair i is in the search request any of the pairs of words; Determine the distance range grade with the largest relative occurrence probability value of the word pair i in each distance range grade; The determined closeness corresponding to the distance range level is taken as the closeness of the word pair i, where different distance range levels are preset to correspond to different closeness levels. 7. The method according to claim 2 or 6, wherein the mining of the co-occurrence dictionary specifically comprises: D1. After performing word segmentation processing on the data source and filtering based on the stop vocabulary list, the obtained words are combined in pairs to form a word pair; D2. Count the co-occurrence status of the word pairs obtained in the step D1 in the data source, and store the co-occurrence status in the co-occurrence dictionary. 8. The method according to claim 2, characterized in that, if in said step A3, a thesaurus and a co-occurrence dictionary are adopted simultaneously, and the tightness of the word pair i can be determined through the query of the thesaurus, The closeness of the word pair i determined by querying the thesaurus is taken as the closeness of the word pair i, and the word pair i is any one of the word pairs in the search request. 9. The method according to claim 1, wherein the structural information processing of the webpage comprises: Divide web pages into web blocks, paragraphs and sentences; The position information of each word in the web page is recorded and stored in the database, where the position information includes: the web page block, segment, sentence and sentence offset. 10. The method according to claim 9, wherein said step B specifically comprises: B1. Determine the co-occurrence status of the word pair i in the web page d according to the position information of the two words of the word pair i in the search request recorded in the database, and the word pair i It is any one of the word pairs in the search request, and the web page d is any web page in the search results; B2. According to the co-occurrence status determined in the step B1, the physical distance distribution of the word pair i in the webpage d is counted. 11. The method according to claim 1, wherein said step C specifically comprises: C1. Using the closeness of the word pair i in the search request to determine the weighted value weight(i) of the word pair i; C2. Using the physical distance distribution of the word pair i in the web page d in the search results to determine the degree of satisfaction fit(i, d) of the web page d to the word pair i; C3, according to the formula

Determine the offset_ratio(d, q) of the web page d for the search request q, where φ is a set of word pairs in the search request q. 12. The method according to claim 11, characterized in that, the weight(i) is: weight(i)=f1(tight(i), imp(i)); wherein tight(i) is the tightness of the word pair i, and imp(i) is the word pair i in the search request q The degree of importance, f1(tight(i), imp(i)) is a function that takes tight(i) as the main factor and imp(i) as the adjustment factor, in the case of the same imp(i) tight(i) The larger the value, the larger the weight(i) value; or, weight(i)=f2(tight(i)), where f2(tight(i)) is a function for normalizing tight(i). 13. The method according to claim 12, wherein said imp(i) is determined by at least one of the following factors: The part of speech of the word pair i in the search request, the sentence component of the word pair i in the search request, and the inverted document rate of the word pair i. 14. The method according to claim 11, wherein the fit(i, d) is: fit(i, d)=f3(HIT(i, d), tight(i)); wherein HIT(i, d) identifies the physical distance distribution of the word i in the web page d that is counted, tight(i ) is the tightness of the word pair i, f3(HIT(i, d), tight(i)) is the distance range of the word pair i determined by HIT(i, d) as the main factor and tight( i) As a function of the adjustment factor, the smaller the distance range of the word pair i determined by HIT(i, d) under the same tight(i) is, the larger the value of fit(i, d) is; or, fit(i,d)=f4(HIT(i,d)), f4(HIT(i,d)) is to map the distance range of the word pair i determined by HIT(i,d) into a specific fit(i , d) function of value. 15. method according to claim 14, is characterized in that, by HIT (i, d) determines the distance scope of described word to i specifically comprises: Adopt the minimum distance range of the word pair i in the HIT (i, d) as the distance range of the word pair i; or, According to the HIT(i, d), the distance range level with the largest relative occurrence probability value is used as the distance range level of the word pair i. 16. A device for determining the degree of escape for the sorting of search results, characterized in that the device comprises: a compactness analysis unit, a distance distribution determination unit and a degree of escape determination; The closeness analysis unit is used to analyze the closeness of the search request input by the user, and determine the closeness of each word pair in the search request; The distance distribution determining unit is configured to calculate the physical distance distribution of each word pair in the search request in each web page according to the result of structural information processing of each web page in the search result corresponding to the search request; The escaping degree determination unit is configured to use the closeness corresponding to each word pair in the search request and the physical distance distribution in each webpage to determine the translation of each webpage in the search result for the search request. degree of meaning, and the degree of escape is used to sort the webpages in the search results. 17. The device according to claim 16, wherein the closeness analysis unit specifically comprises: a word segmentation processing subunit, a word pair determination subunit and a closeness determination subunit; The word segmentation processing subunit is configured to perform word segmentation processing on the search request; The word pair determination subunit is used to determine each word pair in the search request by using the words obtained after the word segmentation process; The compactness determination subunit is used to query a pre-mined dictionary of proper names and/or a co-occurrence dictionary, and determine the compactness of each word pair, wherein the dictionary of proper names contains proper nouns mined in advance, The co-occurrence dictionary includes the pre-determined co-occurrence status of each word pair in the existing data source. 18. The device according to claim 17, wherein the compactness analysis unit further comprises: a filter processing subunit, which is used to perform word segmentation based on stop words for the words obtained after the word segmentation processing subunit performs word segmentation processing. For filtering the table, the words obtained after filtering are sent to the word pair determination subunit. 19. The device according to claim 17, wherein the word pair determining subunit forms a word pair by two adjacent words in the words obtained after the word segmentation process; or, Words with strong expressive ability among the words obtained after the word segmentation process are used to form word pairs, wherein the words with strong expressive ability are determined according to part of speech or sentence components in the search request. 20. The device according to claim 17, wherein if the proper noun in the dictionary of proper names contains a word pair i, the closeness determining subunit determines the closeness of the word pair i as A preset closeness value, the word pair i is any one of the word pairs in the search request. 21. The device according to claim 17, wherein the compactness determination subunit specifically comprises: a dictionary query module, a distance level determination module and a compactness determination module; The dictionary query module is used to query the co-occurrence dictionary to determine the co-occurrence status of the word pair i in the existing data source, the co-occurrence status includes the number of occurrences of the word pair i in each distance range level, the word Pair i is any one of the word pairs in the search request; The distance level determination module is used to determine the distance range level with the largest relative occurrence probability value of the word pair i in each distance range level according to the query result of the dictionary query module; The closeness determination module is used to use the closeness corresponding to the distance range level determined by the distance level determination module as the closeness of the word pair i, wherein different distance range levels are preset to correspond to different closeness . 22. The device according to claim 17 or 21, wherein the compactness analysis unit further comprises: a co-occurrence dictionary mining subunit, which is used to perform word segmentation processing and stop word list based on the data source. After filtering, the obtained words are combined in pairs to form word pairs, the co-occurrence status of the obtained word pairs in the data source is counted, and the statistical co-occurrence status is stored in the co-occurrence dictionary. 23. The device according to claim 17, wherein if the closeness determination subunit adopts both a thesaurus and a co-occurrence dictionary, and the closeness of the word pair i can be determined through the query of the thesaurus , then the closeness of the word pair i determined by querying the thesaurus is taken as the closeness of the word pair i, and the word pair i is any one of the word pairs in the search request. 24. The device according to claim 16, characterized in that, the device also comprises: a structural information processing unit, which is used to divide the webpage into webpage blocks, paragraphs and sentences, record the position information of each word in the webpage and store it in the database , wherein the location information includes: the web page block, segment, sentence and intra-sentence offset. 25. The device according to claim 24, wherein the distance distribution determination unit specifically comprises: a co-occurrence status determination subunit and a distance distribution statistics subunit; The co-occurrence determination subunit is configured to determine the position of the word pair i in the webpage d according to the position information of the two words of the word pair i in the webpage d in the search request recorded in the database Co-occurrence situation, the word pair i is any one of the word pairs in the search request, and the web page d is any web page in the search results; The distance distribution statistical subunit is configured to count the physical distance distribution of the word pair i in the web page d according to the co-occurrence status determined by the co-occurrence status determination sub-unit. 26. The device according to claim 16, wherein the escape degree determination unit specifically comprises: a weighted value determination subunit, a satisfaction degree determination subunit and an escape degree determination subunit; The weighted value determination subunit is used to determine the weighted value weight(i) of the word pair i by using the closeness of the word pair i in the search request; The satisfaction determination subunit is used to determine the satisfaction degree fit(i, d) of the webpage d to the word pair i by using the physical distance distribution of the word pair i in the web page d in the search results ; The escaping degree determines subunits for use according to the formula Determine the offset_ratio(d, q) of the web page d for the search request q, where φ is a set of word pairs in the search request q. 27. The device according to claim 26, characterized in that, the weighted value determination subunit is according to weight(i)=f1(tight(i), imp(i)) or weight(i)=f2(tight( i)) determine the weighted value weight(i) of the word i; Where tight(i) is the tightness of the word pair i, imp(i) is the importance degree of the word pair i in the search request q, and f1(tight(i), imp(i)) is the tight(i) as the main factor and imp(i) as the function of the adjustment factor, in the case of the same imp(i), the larger the value of tight(i) is, the larger the value of weight(i) is, f2(tight(i)) is a function to normalize tight(i). 28. The device according to claim 27, wherein the escape degree determination unit further comprises: an importance determination subunit, configured to determine the imp(i) according to at least one of the following factors: The part of speech of the word pair i in the search request, the sentence component of the word pair i in the search request, and the inverted document rate of the word pair i. 29. The device according to claim 26, characterized in that, the satisfaction determination subunit follows fit(i, d)=f3(HIT(i, d), tight(i)) or fit(i, d )=f4(HIT(i, d)) determines the degree of satisfaction fit(i, d) of said webpage d to said words to i; Wherein HIT (i, d) marks the physical distance distribution of the said words to i in web page d, and tight (i) is the tightness of said words to i, f3 (HIT (i, d), tight ( i)) is a function of the distance range of the word pair i determined by HIT (i, d) as the main factor and tight (i) as the adjustment factor, under the same tight (i) situation by HIT (i, d) The smaller the distance range of the determined word pair i is, the larger the value of fit(i, d) is, and f4(HIT(i, d)) is the distance range mapping of the word pair i determined by HIT(i, d) is a function of specific fit(i, d) values. 30. The device according to claim 29, wherein said escape degree determining unit further comprises: a distance range determining subunit for determining the distance of said word pair i according to said HIT(i, d) scope, including: Adopt the minimum distance range of the word pair i in the HIT (i, d) as the distance range of the word pair i; or, According to the HIT(i, d), the distance range level with the largest relative occurrence probability value is used as the distance range level of the word pair i.

Images