CN105335510A - Text data efficient searching method - Google Patents

Abstract

The invention provides a text data efficient searching method. The method comprises the steps of performing concept description on an entity, and building an entity knowledge base; carrying out a semantic analysis on a text document according to the knowledge base; calculating a similarity value of a user search term with an entity concept, sorting search results on the basis of the calculated similarity value and returning the same to a user. According to the text data efficient searching method provided by the invention, the defects of traditional data search can be overcome, and the data search efficiency is improved in the aspects of information recall ratio and precision ratio.

Description

Text data high-efficiency search method

Technical field

The present invention relates to natural language processing, particularly a kind of text data high-efficiency search method.

Background technology

Along with the fast development of Internet technology, society enters the information age, and especially the quantity of the large data background in current financial field lower network text message document is more and more huger.Network information text document presents more complicated characteristic thereupon, exposes some problem demanding prompt solutions.But, traditional financial domain search engine is in the face of the syntactic level of text character String matching, the semantic class analysis that shortage represents for information and processes and understand, namely information is abundant, and knowledge is poor, so rely on the demand that traditional data retrieval mode has been difficult to meet more and more higher financial class user.

Summary of the invention

For solving the problem existing for above-mentioned prior art, the present invention proposes a kind of text data high-efficiency search method, comprising:

Conceptual description is carried out to entity, builds entity knowledge base;

Based on described knowledge base, semantic analysis is carried out to text document;

Calculate the similar value of user search word and entitative concept, and

Based on calculated similar value result for retrieval sorted and return to user.

Preferably, described semantic analysis is carried out to text document, comprise and the semanteme of document is marked, and extract file characteristics and carry out text mapping, from entity vocabulary, obtain entitative concept, set up the semantic feature territory of document, complete the automatic mark of document library document, and the non-semantic feature of mark and index file, generate document index storehouse and metadatabase thus, wherein based on document markup information architecture index database, be according to retrieving the document information of meeting consumers' demand with index database;

Describedly based on similar value, result for retrieval to be sorted, comprise with the entity dictionary of solid generation for foundation, perform the participle process of user search input characters, user search is divided into entitative concept set and the set of non-physical concept; Then press similar value to these two set respectively to expand, obtain two retrieval candidate collection, obtain the retrieval set after sequence, after finally result for retrieval being pressed the similar value sequence of retrieval request, result is pushed to user.

Preferably, described text mapping comprises the following steps:

First entitative concept is described as F=(U, T, J, Y), wherein U={u ₁, u ₂..., u _{| U|}represent the user using word management text document, and each user has No. ID unique mark; T={t ₁, t ₂..., t _{| T|}representing the used word of user in set, this word is arbitrary character string, J={i ₁, i ₂..., i _{| J|}representing all spectra related text document, its content depends on the type of user's tag set, and user's tag set is made up of user, word, document three key elements, namely uses (U, T, J) to be described; represent ternary relation, wherein (u, t, i) element describes the text document i that user uses the collection of word t mark; F (u, i)={ t ∈ T| (u, t, i) ∈ Y} describes user and uses one group of word to define a text document, wherein u ∈ U, i ∈ J; Principal BO=(C, R) is built, wherein C={c by two tuples ₁, c ₂..., C _{| c|}representing concept set, described representation of concept is c=(id, syn, phase, kind), id is the unique identification of concept, and syn is TongYiCi CiLin, and phase is the phrase describing concept, and kind is the part of speech of concept being classified; R={r ₁, r ₂..., r _{| R|}represent relation between concept set; Now be defined as one group of TongYiCi CiLin S, each text document w ∈ S two tuples are expressed as: (w, fq _c)), wherein, fq _cw () is the frequency of occurrences of text document w;

In the text mapping stage, utilize one of following text mapping method:

Direct mapping, is mapped to the concept in entity, is expressed as TC by each word: for all t ∈ T, have T → 2 ^c; Wherein, in concept set, each t of C is a resource in TongYiCi CiLin syn, and words of description is to the direct mapping of concept;

Part mapping, namely when word can not directly map, the time from start to end, progressively shortens into a word by phrase; Attempt which stage can mapping shortening phrase in from the left side of phrase based on grammer, then carry out from the right improving amendment;

First Document mapping, namely arrange the matrix D C:[t being used for mapping intensity between words of description and concept _i, t _j] _{m × n}, wherein, m=|T| and word quantity, n=|C| and concept quantity; In mapping process, produce initial matrix, the mapping intensity of initial matrix is the syn document word frequency be associated:

$\mathrm{DC}:[t_j,c_j]=\left\{\begin{array}{cc}f{q}_{c_j}\left(t_j\right)& c_j\∈\mathrm{TC}\left(t_i\right)\\ 0& \end{array}\right.$

After mapping terminates, the value of initial matrix DC represents t in dictionary _iand c _jmapping intensity.

The present invention compared to existing technology, has the following advantages:

The present invention proposes a kind of text data high-efficiency search method, make up the deficiency of traditional data retrieval, improve the efficiency of data retrieval from information recall ratio and precision ratio aspect.

Accompanying drawing explanation

Fig. 1 is the process flow diagram of the text data high-efficiency search method according to the embodiment of the present invention.

Embodiment

Detailed description to one or more embodiment of the present invention is hereafter provided together with the accompanying drawing of the diagram principle of the invention.Describe the present invention in conjunction with such embodiment, but the invention is not restricted to any embodiment.Scope of the present invention is only defined by the claims, and the present invention contain many substitute, amendment and equivalent.Set forth many details in the following description to provide thorough understanding of the present invention.These details are provided for exemplary purposes, and also can realize the present invention according to claims without some in these details or all details.

An aspect of of the present present invention provides a kind of text data high-efficiency search method.Fig. 1 is the text data high-efficiency search method process flow diagram according to the embodiment of the present invention.The present invention utilizes entity to set up semantic relation between word, realizes semantic retrieval, by descriptive power abundant in semanteme and powerful logical reasoning ability accurate description information text document, builds a kind of search method that can realize semantic class and analyze.Semantic retrieval is different based on the retrieval mode of keyword match from tradition, because semantic retrieval is analyzed the retrieval request that information text document and user are submitted to based on the semantic class of comprehension of information, semantic retrieval mode all imparts semantic component to search condition, Information Organization and result for retrieval, can improve retrieval precision.

Semantic data based on entity is retrieved, be to make document carry out semantic description, the semantic marker to document object is completed by entity knowledge base, and then analyze the semantic understanding of document object, and the semantic information of user search word, the semanteme that simultaneously can realize entity retrieves word is expanded, and finally complete the acquisition of desirable result for retrieval, concrete retrieving is:

Step 1: build and describe entitative concept.Describe entitative concept, build entity knowledge base.

Step 2: extract the feature of document and carry out text mapping.Describe and construction of knowledge base and management by entitative concept, semantic marker and text mapping are carried out to the document obtained, document semantic implication is analyzed.

Step 3: formulate entitative concept and expand and retrieve semantic expansion strategy.On the basis that entitative concept describes, analyze the semantic information of user search request, and carry out semanteme expansion to user search word, searching system completes the retrieving to the alternative retrieval word set generated.

Step 4: carry out the calculating of entitative concept similar value.Rely on entitative concept structural drawing, perform the calculating of entitative concept semantic distance, entitative concept node depth calculation and semantic overall similar value and calculate, serve result for retrieval sequencer procedure with this.

Step 5: the result for retrieval of searching system is sorted.Based on association similar value computation rule, the similar value that the original term of completing user and searching system return result for retrieval compares, and sorts based on the result for retrieval of similar value to searching system and feeds back to user.

Wherein, the core that entity and taxonomic hierarchies thereof characterize as data text document semantic, can describe document and instruct.Domain knowledge is also as the basic foundation that retrieval is expanded and result for retrieval sorts.Therefore, the structure of domain knowledge and maintenance, as structure and the maintenance of inference rule, be unable to do without domain knowledge management.The semanteme of document carries out semantic analysis by mark, relend and help file characteristics extractive technique, entitative concept is obtained from entity vocabulary, set up the semantic feature territory of document, complete the automatic mark of document library document, and complete the non-semantic feature of mark and index file, generate document index storehouse and metadatabase thus.Based on document markup information architecture index database, on this basis, the document information that can meet consumers' demand is retrieved.Realizing retrieval to expand and sort result, is exactly with the entity dictionary of solid generation for foundation, and the participle work of completing user retrieval input characters, is divided into entitative concept set and the set of non-physical concept by user search.Then, respectively these two set are expanded by similar value, obtain two retrieval candidate collection, the retrieval set after sequence is obtained by association similar value sort algorithm, finally, complete the submission link of retrieval request to index database and search library, by result for retrieval by after the similar value sequence of retrieval request, result is pushed to user.

Entitative concept is described as by the present invention: F=(U, T, J, Y), wherein, and U={u ₁, u ₂..., u _{| U|}represent user, describe the user using word management text document, and each user there is No. ID unique mark.T={t ₁, t ₂..., t _{| T|}representing word, the used word of user in description collections, word can be arbitrary character string (word or expression), now word is expressed as one group of sequence of terms, t={term ₁, term ₂..., term _m, t ∈ T, above-mentioned formula words of description is also mapped to one group of term, and term can be any word.J={i ₁, i ₂..., i _{| J|}represent text document, describe all spectra relevant documentation, its content depends on the type of user's tag set, and user's tag set is made up of user, word, document three key elements, namely uses (U, T, J) to be described. represent ternary relation, wherein (u, t, i) element describes the text document i that user uses the collection of word t mark.F (u, i)={ t ∈ T| (u, t, i) ∈ Y} describes user and uses one group of word to define a text document, wherein u ∈ U, i ∈ J.

In order to understand the relation between user words implication and word, building principal, being described by two tuples: BO=(C, R), wherein C={c ₁, c ₂..., C _{| c|}representing concept set, representation of concept is c=(id, syn, phase, kind), id is the unique identification of concept, and syn is TongYiCi CiLin, contain the synonymous term set of concept, phase is the phrase describing concept, and kind is the part of speech of concept being classified; R={r ₁, r ₂..., r _{| R|}represent relation between concept set.Now be defined as one group of TongYiCi CiLin S, text document w ∈ S, each text document w two tuples are expressed as: (w, fq _c)), wherein, fq _cw () is the frequency of occurrences of text document w.

A word can be mapped to one or more concept, also likely only has partial words can be mapped to one or more concept, and the present invention utilizes following several text mapping method.

Direct mapping: words of description, to the mapping of concept, is mapped to the concept in entity by each word, can be expressed as: TC: for all t ∈ T, have T → 2 ^c.Wherein, each t in concept set is a text document in syn, and words of description is to the direct mapping of concept.

Part mapping: when word can not directly map, the time from start to end, can complete part mapping as follows.Step 1: phrase is progressively shortened into a word.Step 2: based on grammer, from the left side of phrase, attempts which stage can mapping shortening phrase in, then carries out from the right improving amendment.

Document mapping: first the matrix being used for mapping intensity between words of description and concept is set: DC:[t _i, t _j] _{m × n}, wherein, m=|T| and word quantity, n=|C| and concept quantity.To produce initial matrix in mapping process, the mapping intensity of initial matrix is the syn text document word frequencies be associated:

$DC:\[t_i,c_j\]=\left\{\begin{array}{cc}{\mathrm{fq}}_{c_j}\left(t_i\right)& c_j\∈TC\left(t_i\right)\\ 0& \end{array}\right.$

After mapping terminates, the value of initial matrix DC represents t in dictionary _iand c _jmapping intensity.

Further, entitative concept expansion step of the present invention is as follows.

Step 1: each entitative concept is expanded.

Make e (c)={ x|sim (x, c) >p ∩ y| (y ≠ x) and sim (y, c) <sim (x, c) } be entitative concept set, sim () is the similar value function of two entitative concepts, wherein p preset similarity value threshold value.If carry out semanteme expansion to entitative concept can form e (C ₁)={ C ₁₁, C ₁₂..., C _1i, wherein each element or be sky in set, or sim (C _1k, C ₁) >p, and sim (C _1k, C ₁) <sim (C _1k, C _m).

That is, single entity concept can be expanded based on the calculating of association similar value, similar value is selected to be greater than the entitative concept of given threshold value p, and the entitative concept be selected is with the similar value between other user subject concepts, than this, to be selected entitative concept little with the similar value between current single entity concept.

Step 2: build entitative concept retrieval set.The element of user subject concept set is expanded, each concept element wherein expands set e widenable to one, choosing of entitative concept can be carried out respectively from each e, build entitative concept retrieval set, the member of retrieval entitative concept set expands from each entitative concept the concept chosen set, and retrieval entitative concept set description is:

f _c＝{f ₁,f ₂,...f _n}

Wherein, f ₁at e (C ₁) in choose, f _nat e (C _n) in choose.All entitative concept retrieval set can be described as, FC (C) { (F ₁, F ₂... F _n) | F ₁∈ e (C ₁) ... F _n∈ e (C _n)

Each entitative concept set f _cthe similar value of the entitative concept set C do not expanded inputted with user can be calculated by following formula:

${\mathrm{sim}}_{sem}\left(f_c,C\right)=\frac{\underset{i=1}{\overset{Max\left(\left|f_c\right|,\left|C\right|\right)}{\mathrm{\&Sigma;}}}sim\left(f_i,c_i\right)+\mathrm{\&theta;}}{Max\left(\left|f_c\right|,\left|C\right|\right)+\mathrm{\&theta;}}$

Wherein, θ is regulating parameter.

N is made to be element number in the entitative concept set C do not expanded that inputs of user, so sim _sem(f _c, C) can be described as:

${\mathrm{sim}}_{sem}\left(f_c,C\right)=\frac{\underset{i=1}{\overset{n}{\mathrm{\&Sigma;}}}sim\left(f_i,c_i\right)+\mathrm{\&theta;}}{n+\mathrm{\&theta;}}$

Wherein, allow multiple θ to be present in each entitative concept to expand in set.

For the expansion of key word, because key word is not entitative concept, the set that the set of all expansions forms is the power set of original keyword collection, be designated as P (K), element wherein itself is also set, if p is the element in power set P (K), then the similar value computing method between p and K are:

${\mathrm{sim}}_{key}(p,K)=\frac{\left|p\right|+\mathrm{\&theta;}}{\left|K\right|+\mathrm{\&theta;}}.$

Entity can describe concept in specific area and relation thereof, and entitative concept structural drawing can be formed, this sterogram can computational entity concept similar value, entitative concept structural drawing can see the directed acyclic graph with root node as, the node of entitative concept in figure represents, relation between concept is represented by directed edge, and this entitative concept structural drawing has the hierarchical structure characteristic of tree, and the characteristic of directed edge and multiple inheritance.Concrete steps are as follows:

Step 1: the calculating of entitative concept semantic distance.

Based on entitative concept figure, the semantic distance of entitative concept shows as the quantity of the directed edge of connection two concept nodes, is designated as d (C _a, C _b).Pass between entitative concept semantic distance and entitative concept semantic similitude value is: the distance between two entitative concept semantemes is larger, then the similar value between these two entitative concepts is less.

Step 2: the calculating of the entitative concept father node degree of depth.

Based on the hierarchical structure feature that entitative concept embodies, top-down tissue is carried out to entitative concept node, and descending classification, according to entitative concept common father node place level recently, its level known is darker, and entitative concept classification is thinner, and the semantic information inherited from its father node is more, its common semantic information is more, and the similar value namely between these two entitative concepts is larger.Now use depth (parent (C _a, C _b)) the recently common father node degree of depth of two concepts, parent (C are described _a, C _b) represent two concepts common father node recently.

Step 3: computing semantic degree of overlapping.Semantic Overlapping Calculation between entitative concept can have been come according to the identical father node number comprised between two entitative concepts, if but entitative concept semantic distance and the semantic degree of overlapping of entitative concept are taken in simultaneously, then there is the possibility of double counting, because imply the semantic degree of overlapping information of entitative concept in entitative concept semantic distance, so can based on entitative concept semantic distance and common father node the degree of depth in entitative concept structural drawing, calculate entitative concept similar value, make two entitative concept a, b, then a, the semantic similitude value of b represents the weighting normalization that similar value affects by both semantic distances and the degree of depth of common father node, be calculated as follows:

$sim(C_a,C_b)=\mathrm{\&alpha;}\frac{k}{d(c_a,c_b)+k}+\mathrm{\&beta;}\frac{depth\left(parent\right(c_a,c_b))}{\max depth}$

Wherein, α is semantic distance weighted value, and β represents the weighted value of common father node, and meets alpha+beta=1, and the similar value that semantic distance determines regulates by regulating parameter k, the depth capacity of presentation-entity conceptional tree.

Step 4: the overall similar value of entitative concept calculates.

If carry out semanteme to the entitative concept in user search word set to expand, the semantic expansion of retrieval can be generated gather, be designated as FC (C), if expanded the set of keywords of the non-physical concept in user search word set, the power set of set of keywords can be generated, be designated as P (K), now from FC (C), get an element, be designated as f _c, this element is one and expands concept set, then gets an element from P (K), is designated as p, and this element is one and expands set of keywords, just can form the retrieval request that is submitted to searching system, be expressed as (f _c, p), make user search word set for (C, K), then by (C, K) and (f _c, the p) calculating of similar value, can obtain the similar value of user search word set and result for retrieval.Based on expansion set of keywords similar value, expand entitative concept set similar value, classification concept set similar value, can calculate overall similar value, mathematical description is as follows:

SIM(f _c,p,C,K)＝λ ₁×sim _sem(f _c，C)+λ ₂×sim _key(p，K)

Wherein, λ ₁, λ ₂for regulating parameter, λ ₁represent the ratio of entitative concept set similar value and overall similar value, λ ₂represent the ratio of set of keywords similar value and overall similar value, and λ ₁+ λ ₂=1.

In sum, the present invention proposes a kind of text data high-efficiency search method, make up the deficiency of traditional data retrieval, improve the efficiency of data retrieval from information recall ratio and precision ratio aspect.

Obviously, it should be appreciated by those skilled in the art, above-mentioned of the present invention each module or each step can realize with general computing system, they can concentrate on single computing system, or be distributed on network that multiple computing system forms, alternatively, they can realize with the executable program code of computing system, thus, they can be stored and be performed by computing system within the storage system.Like this, the present invention is not restricted to any specific hardware and software combination.

Should be understood that, above-mentioned embodiment of the present invention only for exemplary illustration or explain principle of the present invention, and is not construed as limiting the invention.Therefore, any amendment made when without departing from the spirit and scope of the present invention, equivalent replacement, improvement etc., all should be included within protection scope of the present invention.In addition, claims of the present invention be intended to contain fall into claims scope and border or this scope and border equivalents in whole change and modification.

Claims (3)

1. a text data high-efficiency search method, is characterized in that, comprising:

Conceptual description is carried out to entity, builds entity knowledge base;

Based on described knowledge base, semantic analysis is carried out to text document;

Calculate the similar value of user search word and entitative concept, and

Based on calculated similar value result for retrieval sorted and return to user.

2. method according to claim 1, it is characterized in that, described semantic analysis is carried out to text document, comprise and the semanteme of document is marked, and extract file characteristics and carry out text mapping, entitative concept is obtained from entity vocabulary, set up the semantic feature territory of document, complete the automatic mark of document library document, and the non-semantic feature of mark and index file, generating document index storehouse and metadatabase thus, wherein based on document markup information architecture index database, is according to retrieving the document information of meeting consumers' demand with index database;

Describedly based on similar value, result for retrieval to be sorted, comprise with the entity dictionary of solid generation for foundation, perform the participle process of user search input characters, user search is divided into entitative concept set and the set of non-physical concept; Then press similar value to these two set respectively to expand, obtain two retrieval candidate collection, obtain the retrieval set after sequence, after finally result for retrieval being pressed the similar value sequence of retrieval request, result is pushed to user.

3. method according to claim 2, is characterized in that, described text mapping comprises the following steps:

First entitative concept is described as F=(U, T, J, Y), wherein U={u ₁, u ₂..., u _{| U|}represent the user using word management text document, and each user has No. ID unique mark; T={t ₁, t ₂..., t _{| T|}representing the used word of user in set, this word is arbitrary character string, J={i ₁, i ₂..., i _{| J|}representing all spectra related text document, its content depends on the type of user's tag set, and user's tag set is made up of user, word, document three key elements, namely uses (U, T, J) to be described; represent ternary relation, wherein (u, t, i) element describes the text document i that user uses the collection of word t mark; F (u, i)={ t ∈ T| (u, t, i) ∈ Y} describes user and uses one group of word to define a text document, wherein u ∈ U, i ∈ J; Principal BO=(C, R) is built, wherein C={c by two tuples ₁, c ₂..., C _{| c|}representing concept set, described representation of concept is c=(id, syn, phase, kind), id is the unique identification of concept, and syn is TongYiCi CiLin, and phase is the phrase describing concept, and kind is the part of speech of concept being classified; R={r ₁, r ₂..., r _{| R|}represent relation between concept set; Now be defined as one group of TongYiCi CiLin S, each text document w ∈ S two tuples are expressed as: (w, fq _c)), wherein, fq _cw () is the frequency of occurrences of text document w;

In the text mapping stage, utilize one of following text mapping method:

Direct mapping, is mapped to the concept in entity, is expressed as TC by each word: for all t ∈ T, have T → 2 ^c; Wherein, in concept set, each t of C is a resource in TongYiCi CiLin syn, and words of description is to the direct mapping of concept;

Part mapping, namely when word can not directly map, the time from start to end, progressively shortens into a word by phrase; Attempt which stage can mapping shortening phrase in from the left side of phrase based on grammer, then carry out from the right improving amendment;

First Document mapping, namely arrange the matrix D C:[t being used for mapping intensity between words of description and concept _i, t _j] _{m × n}, wherein, m=|T| and word quantity, n=|C| and concept quantity; In mapping process, produce initial matrix, the mapping intensity of initial matrix is the syn document word frequency be associated:

$DC:\&lsqb;t_i,c_j\&rsqb;=\left\{\begin{array}{cc}{\mathrm{fq}}_{c_j}\left(t_i\right)& c_j\&Element;TC\left(t_i\right)\\ 0& \end{array}\right.$

After mapping terminates, the value of initial matrix DC represents t in dictionary _iand c _jmapping intensity.