CN108491462A - A kind of semantic query expansion method and device based on word2vec - Google Patents

Abstract

The invention discloses a kind of semantic query expansion method and device based on word2vec, belongs to technical field of information retrieval.The method of the present invention includes：The pre-treatment step of the given inquiry of user：Word segmentation processing is carried out to inquiry, remove stop words and carries out stem reduction；Expansion word Candidate Set selecting step：Initial extension word is chosen using word2vec tools；Establish extension vocabulary step：Expansion word Candidate Set is filtered, true extension vocabulary is established；Query expansion step：User's inquiry and its expansion word are matched with indexed set, return to relevant documentation and sorted.The present invention proposes a kind of query vector generation method towards expansion word to filter candidate expansion word and build extension vocabulary, to the correlation for preferably embodying expansion word and entirely inquiring, and then the effect of raising query expansion.

Description

A semantic query extension method and device based on word2vec

technical field

The invention relates to a word2vec-based semantic query extension method and device, belonging to the technical field of information retrieval.

Background technique

Query expansion technology is an important issue in the field of information retrieval. In the current information retrieval model and system, information is stored in the form of words, words or phrases. When the user gives a query, only when the query words in the query set appear in the document, it is possible to Related documents were retrieved. However, in human natural language, the same concept often has many different expressions. For example, when searching for automobile, if it is not expanded, then those that include car, sedan, Ford, etc. are highly relevant to the user's original query but due to Words are different and cannot be retrieved, so that users cannot get satisfactory results. It is precisely because of the mismatch of query words that users sometimes have to change the query words to find the information they need. Therefore, in order to reduce the burden on users, it is necessary for the information retrieval system to automatically select some other words related to the query. To assist the query, that is, to solve this word mismatch problem through query expansion technology.

When a user submits a query, search engines usually use query expansion as an indispensable module in order to improve the user's search satisfaction. At present, the commonly used query expansion methods mainly include the following:

1. Query expansion method based on semantic knowledge dictionary:

The method based on the semantic knowledge dictionary mainly uses WordNet, HowNet or other semantic knowledge dictionaries such as Cilin to select words that have a certain semantic correlation with the query word for expansion. This method is generally based on the context of the query word. Synonyms, synonyms, etc. This method relies too much on a complete semantic system and is independent of the corpus to be retrieved. Therefore, the selected extended words usually cannot reflect the characteristics of the corpus, and it is difficult to achieve good query results.

2. Query expansion based on global analysis:

The global analysis is to first conduct a correlation analysis on the words or phrases in all documents, calculate the correlation degree of each pair of words, and then add the words with the highest correlation with the query words to the initial query to generate a new query. The advantage of this method is that the relationship between words can be explored to the maximum extent, especially after the dictionary is established, the query can be expanded with high efficiency; the disadvantage is that when the document set is large, the establishment of all word relationship dictionaries regardless of It is often not feasible in terms of time or space, and the cost of updating if the document set changes is even greater.

3. Query expansion based on local analysis:

The local analysis method mainly uses the secondary retrieval method to solve the expansion problem, uses the query given for the first time to directly retrieve, and obtains the n documents most relevant to the original query as the source of the expansion word, and finds the original query in these n documents. The most relevant words are added to the initial query to build a new query. The current popular query expansion method based on partial analysis is pseudo-relevance feedback, which is developed on the basis of relevant feedback. The related documents are used as the source of extended words, and the pseudo-related feedback does not need to interact with the user, and directly considers the returned first n documents as related articles. Although the local analysis method is currently the most widely used query expansion method, its shortcoming is that when the documents retrieved for the first time are not highly relevant to the original query, it is easy to add a large number of irrelevant words to the query, resulting in " Query Drift" problem.

With the introduction of semantic models such as Word2Vec and Glove, word embedding technology has attracted the attention of many researchers in many fields of natural language processing in recent years. The word vectors trained by the training model provided by word2vec and Glove reflect the semantic and grammatical relationship in natural language, and the similarity between terms can be judged by calculating the cosine value between word vectors, so it can be used very well for query expansion.

At present, the research work on query expansion based on Word2Vec, but most of the work has the following two main deficiencies:

(1) When constructing the expanded vocabulary, only the words related to the query words are selected as the expanded words, without considering the relevance to the entire query.

(2) Even if the correlation with the entire query is considered, most of the work considers that the query vector is fixed for all replacement words, so the query vector is mostly the simple sum or mean value of each query word vector.

But usually, for a certain extension word of query word q, the influence of other query words on this extension word should not be equal to the influence of q on this extension word. The idea of generating different query vectors with different words in the query as the center word is widely used in semantic disambiguation and other word embedding-based information retrieval fields and has achieved better results, but it has not been effectively applied in the field of query expansion.

Contents of the invention

The technical problem to be solved by the present invention is to provide a semantic query expansion method and device based on word2vec, with the purpose of constructing an extended vocabulary with higher relevance to the query, so as to more comprehensively return documents related to the user query.

The technical scheme of the present invention is: a kind of semantic query extension method based on word2vec, comprising:

Query and document preprocessing steps: for the query submitted by the user, remove the stop words, extract the keywords of the user query and restore the word stem to form the query Q; perform the same preprocessing on the document set to obtain the document set D;

The selection step of the expanded word candidate set: for the query Q after preprocessing, use the word vector trained based on the word2vec model to calculate and obtain the n most similar terms of each query keyword to form the expanded word candidate set C

Establishing an extended vocabulary step: For each term in C, calculate its similarity with the entire query, and select k extended words with the highest similarity to construct an extended vocabulary T;

The step of establishing an inverted index of a document set: establishing an inverted index for the preprocessed document set D;

Extended retrieval step: calculate the correlation between the expanded query and the documents in the corresponding inverted index, and sort the documents according to the correlation.

The query and document preprocessing steps specifically include the following steps:

(1) Carry out word segmentation processing on the query submitted by the user through spaces and punctuation marks;

(2) After word segmentation, stop words are removed, and words that do not represent concepts are filtered out;

(3) Stem reduction is performed after removing stop words to generate query Q;

(4) Do the same preprocessing on the document set to generate a new document set D.

The selection step of the expanded word candidate set specifically includes the following steps:

(1) Given a corpus, train word vectors through the training model provided by word2vec. Word vectors are a set of multidimensional real-valued vectors, which reflect the semantic and grammatical relationships in natural language, so the similarity between terms can be judged by calculating the cosine value between word vectors;

(2) After the word vector is obtained, for each keyword q _i in Q, calculate the cosine similarity of the word vector and obtain the n words most similar to q _i to form an expanded word candidate set for the query.

The step of establishing the expanded vocabulary specifically includes the following steps:

(1) For the query Q formed by the above processing, for each keyword q _i in Q, generate a query vector of Q relative to q _i according to the following formula

In the formula, vec(q _i ) represents the vector of query word q _i , and sim(q _i , q _j ) represents the similarity between q _i and q _j .

(2) For each candidate expansion word t of q _i , calculate the similarity between t and query Q according to the following formula:

For the candidate expansion words of different query words, different query vectors are used Calculate the similarity between the expanded word and the query Q, so the present invention will generate the query vector The method is called the query vector generation method for expansion words, correspondingly, Also known as query vectors for expanded words;

(3) The extended words of each query word calculate the similarity with respect to the entire query Q according to the above model, and then reorder the extended words according to the similarity, and return the k extended words with the highest similarity as the final extended word set T ;

(4) Generate an extended query Q _exp =Q∪T.

The step of establishing an inverted index of a document set specifically includes the following steps:

(1) For the preprocessed document set D, count all the words of D and remove the duplicates to generate a document word set V;

(2) For each term v in V, construct an inverted list consisting of the IDs (d _id ) of all documents d (where d∈D) containing v and the number of occurrences of v in d tf _v,d , each item in the list is expressed as a binary group <d _id ,tf _v,d >, and the set of all inverted lists constitutes the inverted index set I;

(3) For each term v, count the number m of documents where it appears, and calculate the idf score of v according to the following formula:

where |D| denotes the total number of documents in D.

The step of extending the retrieved document specifically includes the following steps:

(1)(1) For each keyword in Q _exp , query the inverted index set I to obtain the inverted list corresponding to the keyword, and record the set of these inverted lists as

(2) Contrast present For each document d in The tf-idf score of each list in the list is obtained to obtain the correlation R(Q _exp ,d) between Q _exp and document d. The formula for calculating R(Q _exp ,d) is as follows:

In the formula, λ represents an adjustment parameter, which is used to control the weight of query words and expansion words when calculating the correlation.

(3) The documents are sorted according to the degree of relevance, so as to return the N documents most relevant to the original query.

A semantic query expansion device based on word2vec, comprising:

The query and document set preprocessing module is used to perform word segmentation, stop word removal and word stem restoration on the document set and user-submitted queries to form query Q and document set D;

The extended word candidate set selection module is used to query each keyword in Q, use the word vector based on word2vec model training to calculate and obtain the n most similar terms of each query keyword, and form the expanded word candidate set C ;

The extended vocabulary construction module is used to calculate the similarity between it and the entire query for each term in the extended word candidate set, and select some extended words with higher similarity to construct the extended vocabulary T;

The document set inverted index module is used to establish an inverted index for the preprocessed document set D;

The extended retrieval module is used to calculate the correlation between the extended query and the documents in the corresponding inverted index, and obtain relevant documents.

The beneficial effects of the present invention are: a word2vec-based semantic query expansion method is proposed, the similarity of the replacement word to the entire query is considered, and a query vector generation method oriented to the expansion word is introduced to generate different queries for the expansion words corresponding to different query words Vector, to obtain an expanded word set that is more relevant to the query, and then to obtain a better query expansion effect.

Description of drawings

Fig. 1 is the functional block diagram of the semantic query expansion based on word2vec of the present invention;

Fig. 2 is the expansion word candidate set diagram of each keyword in the query set of the present invention;

Fig. 3 is a diagram of an inverted index set in the present invention.

Detailed ways

The present invention will be further described below in combination with the accompanying drawings and specific embodiments.

Embodiment 1: As shown in Figure 1-3, a semantic query expansion method based on word2vec, including:

Query and document preprocessing steps:

(1) Carry out word segmentation processing on the query submitted by the user through spaces and punctuation marks;

(2) After word segmentation, stop words are removed, and words that do not represent concepts are filtered out;

(3) Stem reduction is performed after removing stop words to generate query Q.

(4) Do the same preprocessing on the document set to generate a new document set D.

Example 1: Query preprocessing: Suppose the query submitted by the user is "problems associated with highspeed aircraft"

(1) First, word segmentation is performed on the query submitted by the user, and the query performance after word segmentation is: {problems, associated, with, high, speed, aircraft};

(2) Remove the stop words, and then select the nouns in the query to form the final query. The query performance is: {problems, speed, aircraft};

(3) Perform word stemming on keywords in the query, problems are plural nouns, and the restored query keyword set Q={problem, speed, aircraft}.

Example 2: Document set preprocessing: Suppose there is a document set consisting of the following four documents:

D ₀ ="The main problem limiting the high velocity performance of helicopter is resistance"

D ₁ ="high altitude and high speed flying aircraft are often moreslender shape"

D ₂ ="There are many airplanes in the sky that make up a row"

D ₃ ="whether to fly today is a problem"

Find all words in the string by spaces and delimiters, remove stop words and perform stemming, and the new document set formed is:

D ₀ ="problem,limit,velocity,performance,helicopter,resistance"

D ₁ = "altitude,speed,fly,aircraft,slender,shape"

D ₂ = "airplane, sky, row"

D ₃ = "fly, problem"

Steps for selecting the extended word candidate set:

(1) Select the Wikipedia corpus, and train a 200-dimensional word vector file through the CBOW model provided by word2vec;

(2) After the word vector is obtained, for each keyword in Q, the n most similar words are obtained by calculating the cosine similarity of the word vector, as the extended word candidate set of the query.

For each keyword in the query Q={problem, speed, aircraft}, select the top 10 most semantically relevant extended words through the trained word vector, and the situation of the extended word candidate set is shown in Figure 3.

Construct the extended vocabulary T step:

(1) For each keyword q _i in Q, generate a query vector of Q relative to q _i according to the following formula

In the formula, vec(q _i ) represents the vector of query word q _i , and sim(q _i , q _j ) represents the similarity between q _i and q _j .

(2) For each candidate expansion word t of q _i , calculate the similarity between t and query Q according to the following formula:

(3) The extended words of each query word calculate the similarity relative to the entire query Q according to the above model, then reorder the similarity, and return the k extended words with the highest similarity as the final extended word set T;

(4) Generate an extended query Q _exp =Q∪T.

Example:

(1) First, the 200-dimensional word vector of each keyword in the query Q can be obtained according to the trained word vector:

vec(problem)＝[0.29686138, 1.71120727,...,-0.6585713,-1.86508703]

vec(speed)＝[-2.00363445,1.05960512,...,-0.475373,-4.39991331]

vec(aircraft)＝[-3.54158616,3.28720021,...,-2.34602952,-3.29022384]

Then calculate the query vector for each keyword in Q to expand words, the calculation process is as follows:

2) Take the keyword aircraft in the query Q as an example, that is, q ₃ =aircraft, calculate the similarity between each expanded word t of q ₃ and the query Q:

 …

(3) By analogy, calculate the similarity between each extended word in Figure 2 and the original query Q, and then sort the extended words in the candidate set according to the similarity, and obtain k extended words that are most similar to the query Q, with k =4 as an example, the final expanded vocabulary T is as follows:

T = {helicopter, airplane, velocity, altitude}

(4) Merge query words and expansion words to obtain extended query Q _exp :

Q _exp = Q∪T

＝{problem,speed,aircraft}∪{helicopter,airplane,velocity,altitude}

＝{problem,speed,aircraft,helicopter,airplane,velocity,altitude}

Building an inverted index of a document set includes the following steps:

(1) For the preprocessed document set D, count the independent terms in D to generate a vocabulary V;

(2) For each term v in V, construct an inverted list consisting of the IDs (d _id ) of all documents d (where d∈D) containing v and the number of occurrences of v in d tf _v,d , each item in the list is expressed as a binary group <d _id ,tf _v,d >, and the set of all inverted lists constitutes the inverted index set I;

(3) For each term v, count the number m of documents where it appears, and calculate the idf score of v according to the following formula:

where |D| denotes the total number of documents in D.

Example:

(1) After the document set is preprocessed by word segmentation and stop words removal, the following document set D is obtained:

D ₀ ="problem,limit,velocity,performance,helicopter,resistance"

D ₁ = "altitude,speed,fly,aircraft,slender,shape"

D ₂ = "airplane, sky, row"

D ₃ = "fly, problem"

Count the independent terms in D to generate vocabulary V:

V＝{altitude, speed, fly, aircraft, slender, shape, problem, limit, velocity, performance,

helicopter, resistance, airplane, sky, row}

(2) Taking the word velocity in the vocabulary V as an example, traverse the document set D to find the document containing velocity D ₁ , record its ID=D ₁ , and count the number of times it appears in document D ₁ as 1, then the inverse of velocity The expression form of the sorting table is <D ₁ ,1>; by analogy, calculate and establish a set of inverted lists of all terms in V to form an inverted index set I;

(3) For each word v in V, count the number of documents m (that is, the length of the inverted list of v), and calculate the idf score:

For example, v=velocity, the length of the inverted list is 1, that is, there is only one document containing problem in the document set, and m=1, so the idf score of the word velocity is calculated as:

Calculate the idf scores of all words accordingly, and record the idf in the index, and the final inverted index set I is shown in Figure 3.

Extended search steps:

(1) For each keyword in Q _exp , query the inverted index set I to obtain the inverted list corresponding to the keyword, and record the set of these inverted lists as

(2) Contrast present For each document d in The tf-idf score of each list in the list is obtained to obtain the correlation R(Q _exp ,d) between Q _exp and document d. The formula for calculating R(Q _exp ,d) is as follows:

In the formula, λ represents an adjustment parameter, which is used to control the weight of query words and expansion words when calculating the correlation.

(3) The documents are sorted according to the degree of relevance, so as to return the N documents most relevant to the original query.

Example:

(1) For the Q _exp generated above, query the inverted index set in Figure 3, obtain the inverted list corresponding to all keywords in Q _exp , and find the union I _Qexp :

I _Qexp ＝I(problem)∪I(speed)∪......∪I(airplane)∪I(altitude)

＝{D ₁ ,D ₃ }∪{D ₀ }∪...∪{D ₂ }∪{D ₀ }

={D ₀ ,D ₁ ,D ₂ ,D ₃ }

(2) For documents D ₀ , D ₁ , D ₂ and D ₃ , calculate Q _exp and its correlation degree R(Q _exp ,d), where the adjustment parameter λ=0.6, and the calculation process is as follows:

(3) Sort these documents according to the degree of relevance, such as D ₁ >D ₀ >D ₂ >D ₃ ; if N=3, then return the documents D ₁ , D ₀ , and D ₂ .

Embodiment 2: a kind of semantic query extension device based on word2vec, comprising:

The query and document set preprocessing module is used to perform word segmentation, stop word removal and word stem restoration on the document set and user-submitted queries to form query Q and document set D;

The extended word candidate set selection module is used to query each keyword in Q, use the word vector based on word2vec model training to calculate and obtain the n most similar terms of each query keyword, and form the expanded word candidate set C ;

The extended vocabulary construction module is used to calculate the similarity between it and the entire query for each term in the extended word candidate set, and select some extended words with higher similarity to construct the extended vocabulary T;

The document set inverted index module is used to establish an inverted index for the preprocessed document set D;

The extended retrieval module is used to calculate the correlation between the extended query and the documents in the corresponding inverted index, and obtain relevant documents.

The specific embodiments of the present invention have been described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above embodiments. Variations.

Claims (7)

1. a kind of semantic query expansion method based on word2vec, it is characterised in that：It the described method comprises the following steps：

(1) inquiry and document pretreatment：Inquiry participle, the removal stop words submitted for user, extract the pass of user's inquiry Keyword simultaneously carries out stem reduction, composition inquiry Q；Same pretreatment is done to document sets and obtains document sets D；

(2) selection of expansion word Candidate Set：For the inquiry Q after pretreatment, the word based on word2vec model trainings is utilized Vector calculates and obtains n most like lexical items of each searching keyword, constitutes expansion word Candidate Set C；

(3) extension vocabulary is established：To each lexical item in C, it is calculated and the similarity entirely inquired, it is highest to choose similarity K expansion word extends vocabulary T to construct；

(4) document sets inverted index is established：Inverted index is established to the document sets D after pretreatment；

(5) query expansion：The inquiry after extension and the degree of correlation of the document in corresponding inverted index are calculated, according to the degree of correlation to text Shelves are ranked up.

2. the semantic query expansion method according to claim 1 based on word2vec, it is characterised in that：Inquiry and document Pre-treatment step specifically includes following steps：

(1) word segmentation processing is carried out by space character and punctuation mark to the inquiry that user submits；

(2) stop words is removed after participle, the word that those are not represented to concept filters out；

(3) stem reduction is carried out after removing stop words, generates inquiry Q；

(4) same pretreatment is done to document sets and generates new document sets D.

3. the semantic query expansion method according to claim 1 based on word2vec, it is characterised in that：Expansion word is candidate The selecting step of collection, specifically includes following steps：

(1) corpus is given, term vector is trained by the training pattern that word2vec is provided, term vector is one group of multidimensional Real number value vector, vector reflect semanteme and grammatical relation in natural language, therefore can be by between calculating term vector Cosine value judges the similitude between lexical item；

(2) after obtaining term vector, to each keyword q in Q_i, calculated and obtained and q by the cosine similarity of term vector_iMost Similar n word constitutes the expansion word Candidate Set of inquiry.

4. the semantic query expansion method according to claim 1 based on word2vec, it is characterised in that：Extend vocabulary Establishment step specifically includes following steps：

(1) the inquiry Q that above-mentioned processing is formed, to each keyword q in Q_i, a Q is generated as follows relative to q_i's Query vector vec (Q_qi)：

In formula, vec (q_i) indicate query word q_iVector, sim (q_i,q_j) indicate q_iAnd q_jSimilarity.

(2) to q_iEach of candidate expansion word t, calculate t as follows and inquire the similarity of Q：

Sim (t, Q)=cos (vec (t), vec (Q_qi))

For the candidate expansion word of different query words, using different query vector vec (Q_qi) calculate expansion word and inquire Q's Similarity will generate query vector vec (Q_qi) method be referred to as the query vector generation method towards expansion word, correspondingly, vec (Q_qi) it is also referred to as the query vector towards expansion word；

(3) expansion word of each query word calculates the similarity relative to entire inquiry Q according to model above, then to expansion word It is resequenced according to similarity, the highest k expansion word of similarity is returned to, as final expansion-word set T；

(4) expanding query Q is generated_exp=Q ∪ T.

5. the semantic query expansion method according to claim 1 based on word2vec, it is characterised in that：Establish document sets Inverted index specifically includes following steps：

(1) to pretreated document sets D, all words and duplicate removal of D is counted, document word set V is generated；

(2) to each lexical item v in V, construction one is by all document d comprising v, the wherein ID (d of d ∈ D_id) and v in d Occurrence number tf_v,dThe Inverted List of composition, each item is expressed as two tuple ＜ d in list_id,tf_v,dThe form of ＞, all rows of falling The set of list constitutes inverted index collection I；

(3) to each lexical item v, the number of documents m of its appearance is counted, and calculates the idf scores of v according to following formula：

Wherein, | D | indicate the total quantity of document in D.

6. the semantic query expansion method according to claim 1 based on word2vec, it is characterised in that：Query expansion has Body includes the following steps：

(1) to Q_expIn each keyword, inquire inverted index collection I, obtain the corresponding Inverted List of the keyword, remember these The collection of Inverted List is combined into I_Qexp；

(2) to appearing in I_QexpIn each document d, add up its in I_QexpIn each list tf-idf scores, obtain Q_expWith text Degree of correlation R (the Q of shelves d_exp, d), calculate R (Q_exp, d) formula it is as follows：

In formula, λ indicates adjustment parameter, for controlling the weight of query word and expansion word when calculating the degree of correlation.

(3) these documents are ranked up according to the size of the degree of correlation, maximally related N number of document is inquired with former to return.

7. a kind of semantic query expanding unit based on word2vec, it is characterised in that including：

Inquiry and document sets preprocessing module, for being segmented to the inquiry of document sets and user's submission, removing stop words and word The processing such as dry reduction form inquiry Q and document sets D；

Expansion word Candidate Set chooses module, for that will inquire each keyword in Q, using based on word2vec model trainings Term vector calculates and obtains n most like lexical items of each searching keyword, constitutes expansion word Candidate Set C；

Vocabulary constructing module is extended, the similarity for each lexical item in expansion word Candidate Set, calculating it with entirely inquiring, Some higher expansion words of similarity are chosen to construct extension vocabulary T；

Document sets inverted index module, for establishing inverted index to the document sets D after pretreatment；

Query expansion module obtains related for calculating inquiry and the degree of correlation of the document in corresponding inverted index after extending Document.