JP2007172616A - Document search method and device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a document search method and device that can increase search precision by effectively using classified digital information. <P>SOLUTION: Searching through a digital data set having a hierarchical classification structure for a target document according to hierarchical class information includes: (a) extracting a keyword string including at least one keyword from a search request sentence; (b) acquiring class information about a current hierarchy from the digital data set; (c) calculating a difference of each keyword from each class belonging to the current hierarchy; (d) calculating the probability that the target document is present in each class belonging to the current hierarchy according to the difference; (e1) repeating the steps (c) to (e1) as designating the next hierarchy as the current hierarchy until the number of processed hierarchies reaches a predetermined number; (e2) combining the calculated difference from each class belonging to each hierarchy and the probability that the target document is present in each class belonging to each hierarchy to compute a combined difference of each keyword across the predetermined number of hierarchies; and (f) searching for the target document according to the combined difference. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a method and apparatus for searching a target document based on classification information of each hierarchy in a hierarchically classified digital data set.

  In recent years, more and more document files have appeared on the Internet, digital libraries, news, and company LANs (Local Area Networks). In order to manage such digitized data, search for document digital information is becoming more important. Today's search for digital information is becoming more intelligent than the closed, fixed form of the past. Also, current digital information is open and updated quickly, and these digital information is usually distributed. On the other hand, the number of users of digital information systems has been expanded from conventional specialized searchers to general users such as businessmen, managers, and students. This brings various special demands on digital information systems, and personalization and intelligence are new demands on digital information retrieval systems.

  The current digital information has the following important features. That is, a lot of digital information is already classified in advance. For example, there are digital library classifications (for example, ACM, IEEE, etc.) and Web classifications (Yahoo, Google, Sina, etc.). However, the conventional digital information search system cannot improve the search accuracy by using such classified digital information.

  The present invention has been made in view of the above problems, and an object of the present invention is to provide a document search method and apparatus that can effectively use classified digital information and improve search accuracy.

  According to a first aspect of the present invention, a document search method of the present invention is a predetermined number of digital data sets having a hierarchical classification structure formed by hierarchically classifying a set of digital data of a plurality of documents. (A) extracting a keyword string including at least one keyword from a search request sentence input by a user, and (b) the digital document Classification information of the current hierarchy is obtained from the data set, (c) the degree of difference of each keyword in the keyword string is calculated for each classification belonging to the current hierarchy, and (d) based on the degree of difference, belonging to the current hierarchy The probability that the target document exists in each category is calculated. (E1) When the number of processed layers is smaller than the predetermined number, the next layer is set as the current layer, and the above steps (c), ( ), (E1) are executed again, and (e2) when the number of processed hierarchies is equal to or greater than the predetermined number, the calculated difference between each class belonging to each hierarchy and each class belonging to each hierarchy The probability that the target document exists is synthesized, a synthesis difference degree in the hierarchy of the predetermined value of each keyword is obtained, and (f) the target document is searched according to the synthesis difference degree.

According to the second aspect of the present invention, the document search apparatus of the present invention is configured to select a predetermined number of digital data sets having a hierarchical classification structure obtained by hierarchically classifying a set of digital data of a plurality of documents. An apparatus for searching a predetermined target document based on classification information of a hierarchy of the word, a word extracting means for extracting a keyword string including at least one keyword from a search request sentence input by a user, and the digital data set Classification selection subdivision means for obtaining classification information of the current hierarchy from, classification calculation means for calculating the degree of difference of each keyword in the keyword string for each classification belonging to the current hierarchy, and based on the degree of difference, the current degree Target document estimation means for calculating the probability that the target document exists in each category belonging to a hierarchy, the degree of difference with respect to each category, and belonging to each hierarchy A hierarchy that includes a probability combining unit that obtains the probability that the target document exists in each classification and obtains a combined difference between the keywords, and a search engine that searches the target document according to the combined difference. If the number is less than the predetermined number, the difference calculation means and the target document estimation means calculate the difference and the probability sequentially for each layer,
When the hierarchy number of the current hierarchy is greater than or equal to the predetermined number, the dissimilarity synthesizing unit obtains the composite dissimilarity, and the search engine performs a search.

  The document search method and apparatus of the present invention can use auxiliary information such as classification in a digital data set to estimate accurate keyword weights and effectively improve search accuracy.

  Next, embodiments of the present invention will be described with reference to the accompanying drawings.

  FIG. 1 is a block diagram showing a configuration of a document search apparatus 100 according to an embodiment of the present invention.

  The document search apparatus 100 shown in FIG. 1 searches for a target document from a digital data set 110 that is a set of digital data of a plurality of documents, for example. The digital data set 110 is divided into a plurality of classifications, and each classification is further subdivided into a plurality of sub-classifications, and the sub-classification is further subdivided into a plurality of classifications. As described above, the digital data set 110 is classified in a hierarchical manner, and has, for example, a hierarchical classification structure having N classes (first to Nth hierarchies). In each hierarchy, classification information of a plurality of classifications belonging to the hierarchy is given.

  For example, the document search apparatus 100 searches for a target document according to the classification information of the first to Mth layers in the digital data set 110. Here, M is the number of hierarchies to be searched set as required by the user. That is, although the digital data set 110 has N layers, the user may search only the M layers. M and N are integers, and N ≧ 1, N ≧ M ≧ 1.

  The document retrieval apparatus 100 includes a word extraction unit (TE) 101, a keyword selection module (TSM) 102, a classification selection subdivision module (CSM) 103, a difference calculator (DPC) 104, a target document estimation unit (PRE) 105, It has a dissimilarity synthesis module (DIM) 106, a classification determination means (CL) 107, a weight merge module (TWC) 108, and a search engine 109.

  The word extraction unit 101 extracts a keyword string including at least one keyword from the search request sentence input by the user and sets it as the current keyword string.

  The keyword selection module 102 removes the noise keyword in the current keyword string based on the keyword string of the current hierarchy, the corresponding difference degree, and the appearance frequency of the keyword, and ranks one rank higher than the current hierarchy (ie, one rank from the current hierarchy). (Lower level) Keyword column.

  The classification selection subdivision module 103 obtains classification information of the current hierarchy from the digital data set 110.

  The dissimilarity calculator 104 calculates the dissimilarity of each keyword in the current keyword string for each classification belonging to the current hierarchy.

  The target document estimation means 105 calculates the probability that the target document exists in each classification belonging to the current hierarchy, based on the above-described keyword dissimilarity.

  The dissimilarity synthesis module 106 synthesizes the dissimilarity of each keyword obtained by the dissimilarity calculator 104 and the probability that the target document obtained by the target document estimation means 105 exists in each classification belonging to each hierarchy, For each keyword in the current keyword string, the composite dissimilarity with respect to the target document in the first to Mth layers is obtained.

  The classification determination unit 107 removes the noise classification based on the probability that the target document exists in each classification belonging to each hierarchy, and uses the classification information of the next hierarchy.

  The weight merge module 108 synthesizes the global dissimilarity.

  The search engine 109 searches for the target document based on the composite difference.

  When the document retrieval apparatus 100 retrieves the target document according to the classification information of the first hierarchy to the Mth hierarchy, if the hierarchy number of the current hierarchy is smaller than M, the difference calculator 104 and the target document estimation means 105 The dissimilarity and the probability are sequentially calculated for the hierarchy. When the hierarchy number of the current hierarchy is greater than or equal to M, the dissimilarity composition module 106 obtains a composite dissimilarity and the search engine 109 performs a search.

  The document search apparatus 100 having the above configuration uses the auxiliary information such as classification in the digital data set 110 and adopts the keyword weight calculation method of classification to estimate accurate keyword weight and effectively The search accuracy can be improved.

  The configuration shown in FIG. 1 is a specific example for explaining the present invention, and does not limit the present invention. For example, the keyword selection module 102 can improve accuracy and shorten the response time by removing keyword noise, but the keyword selection module 102 may be omitted. That is, the difference calculator 104 may receive the keyword string directly from the word extraction unit 101. Further, the classification determining means 107 and the weight merging module 108 can improve the search accuracy and shorten the response time by removing the classification noise. Furthermore, when calculating the keyword weight, Although the weight calculation method can be used in combination to improve the search accuracy and the versatility of the system, the classification determination unit 107 and the weight merge module 108 may be omitted. That is, the target document estimation unit 105 does not need to feed back the noise removal information to the classification selection subdivision module 103 via the classification determination unit 107, and the search engine 109 directly receives the keyword from the dissimilarity synthesis module 106. A composite dissimilarity of the columns may be received. In this case, for example, the keyword is one letter or one phrase.

  The document search apparatus 100 described above can also be applied to unclassified digital data sets (that is, the number of classes and the number of classifications is 1). Further, in order to improve the versatility of the system, the dissimilarity composition module 106 performs composition for all the dissimilarities calculated by the dissimilarity calculator 104. Here, preferably, the global keyword is calculated by a statistical method.

In addition, the distinctiveness of keywords with respect to classification is obtained according to the following criteria.
(1) Estimate based on keyword distinctiveness with respect to classification.
(2) Estimate based on differences in the expressiveness of keywords for different classifications.
(3) It is determined in consideration of the appearance frequency of keywords in the classification and the attributes of the classification itself.

  FIG. 2 is a flowchart illustrating a document search method according to an embodiment of the present invention.

  For example, the target document is searched based on the classification information of the first to Mth layers in the digital data set 110 described above.

  As shown in FIG. 2, in step S <b> 201, a keyword string including at least one keyword is extracted from the search request sentence input by the user, and is set as the current keyword string.

  In step S202, classification information of the current hierarchy is obtained from the digital data set 110.

  In step S203, the degree of difference of each keyword in the current keyword string is calculated for each classification belonging to the current hierarchy.

  In step S204, the probability that the target document exists in each category belonging to the current hierarchy is calculated based on the difference between the keywords.

  In step S205, if the hierarchy number of the current hierarchy is smaller than M, the process proceeds to step S206. If the current hierarchy number is greater than or equal to M, the process proceeds to step S209.

  In step S206, the next hierarchy is set as the current hierarchy.

  In step S207, noise in the current keyword string is removed based on the keyword string of the current hierarchy, the degree of difference corresponding thereto, and the appearance frequency of the keyword, and the keyword string is set to the next hierarchy.

  In step S208, noise is removed based on the probability that the target document exists in each classification belonging to the current hierarchy, and classification information for the next hierarchy is obtained.

  In step S209, the degree of difference between the keywords obtained in step 203 and the probability that the target document obtained in step 204 exists in each category belonging to each hierarchy are synthesized, and the number of each keyword in the current keyword string is determined. The degree of composition difference with respect to the target document is obtained from the first layer to the Mth layer.

  In step S210, global dissimilarities are synthesized.

  In step S211, the target document is searched according to the compositional difference.

  In the above document search method, it is possible to estimate accurate keyword weights by using auxiliary information such as classification in the digital data set 110 and effectively improve the search accuracy.

  The configuration shown in FIG. 2 is a specific example for explaining the present invention, and does not limit the present invention. For example, in step 207, the keyword noise is removed in the search process to improve accuracy and shorten the response time. However, step 207 may be omitted. That is, the process may proceed directly from step 206 to step 208. Further, in step 208 and step 210, it is possible to improve the search accuracy and shorten the response time by removing the classification noise. Further, when calculating the keyword weight, a global keyword weight calculation method is used. Can be used together to improve search accuracy and system versatility, but step 208 and step 210 may be omitted. That is, the process may return directly from step 206 to step 203, or may proceed directly from step 209 to step 211. In this case, for example, the keyword is one character or one phrase.

  The document retrieval method described above can also be applied to unclassified digital data sets. In addition, in order to improve the versatility of the system, all differences calculated in step 203 are combined. Here, preferably, the global keyword is calculated by a statistical method.

In addition, the distinctiveness of keywords with respect to classification is obtained according to the following criteria.
(1) Estimate based on keyword distinctiveness with respect to classification.
(2) Estimate based on differences in the expressiveness of keywords for different classifications.
(3) It is determined in consideration of the appearance frequency of keywords in the classification and the attributes of the classification itself.

  FIG. 3 is a flowchart showing an operation of searching for a target document according to the embodiment of the present invention.

  First, the user inputs a search request sentence. The search request text indicates the intention of the search by the user. For example, the user inputs one sentence, one paragraph, and one sentence as a search request sentence.

Word extracting means 101 first performs a keyword extracting the search request statement of the user, the keyword _{t 1,} t 2, ..., obtaining a keyword string T consisting of _{t m.}

T = (t ₁ , t ₂ ,..., T _m )
Here, it is assumed that the data set to be searched is the digital data set 110 described above. That is, the digital data set 110 has a hierarchical classification structure.

First, the classification column C in the first hierarchy is selected by the classification selection subdivision module (CSM) 103. The classification column C includes sub classifications c ₁ , c ₂ ,..., C _n .

C = (c ₁ , c ₂ ,..., C _n )
Each keyword in the keyword string T has different distinctiveness for different documents. Here, the distinctiveness of a word with respect to a document is called a word “weight”. Estimating keyword weights is important for search systems. The present invention has realized a system for estimating keyword weights in a hierarchical classification structure. The system gradually approaches the final weight of the keyword by repeatedly subdividing the classified data.

Next, the keyword string T = (t ₁ , t ₂ ,..., T _m ) and the classification string C = (c ₁ , c ₂ ,..., C _n ) are input to the dissimilarity calculator (DPC) 104 and the difference The degree calculator (DPC) 104 calculates the discriminating power of each keyword in the keyword string T with respect to the classification string C, and obtains the difference DP.

_{DP = (dp 1, dp 2} , ..., dp m).

  Next, the dissimilarity DP is input to the keyword selection module (TSM) 102, and keywords that cause noise are filtered. As a result, the difference DP of each keyword with respect to the new keyword string T and the classification string C is obtained.

  Next, the new keyword string T and the dissimilarity DP are input to the target document estimation means (PRE) 105, and the target document estimation means (PRE) 105 has a target document to be searched for in each classification of the classification string C. Calculate the probability PC.

PC = (pc ₁ , pc ₂ ,..., Pc _n )
Actually, although the target document to be searched belongs to the sub-category _ck , another sub-category is often searched by mistake due to the keyword in the search request sentence input by the user. Here, such classification is called noise classification.

  In order to suppress the influence of such noise classification, these noises are removed using the classification determination means (CL) 107. As a result, the following new classification sequence C is obtained.

C = (c ₁ , c ₂ ,..., C _q )
The classification selection subdivision module (CSM) 103 further subdivides each sub-classification c _k in the classification column C.

C _k = (c _k1 , c _k2 ,..., C _kn ).

As with treatment for the upper classification column C as described above, the keyword string _{T = (t 1, t 2} , ..., t m) and classification column _{C k = (c k1, c} k2, ..., c kn) the dissimilarity enter the calculator (DPC) 104, difference calculating unit (DPC) 104 calculates the distinguishing power for the classification column _{C k} for each keyword in the keyword column T, obtaining a dissimilarity DP. Next, the target document estimation means (PRE) 105 calculates the probability PC existing in each classification in C _k of the target document that the user wants to search, and the classification confirmation means (CL) 107 determines the target classification. select.

If necessary, each classification in the classification C _k is further subdivided, and the above operation is repeated in the next hierarchy. For example, the number of hierarchies is determined by the structure and accuracy requirements of the data set.

  As described above, the degree of difference is calculated, the keyword string to be searched by the user is obtained, the degree of difference DP of each keyword is obtained for the classification belonging to each hierarchy, and the probability PC that the target document exists in each classification is calculated. To do.

  The result obtained above is input to the dissimilarity composition module (DIM) 106, and the dissimilarity composition module 106 calculates the final dissimilarity of keywords based on the above results.

  The degree of difference has a strong distinctive power with respect to different classifications, and if the degree of difference is used, the classification to which the target document belongs can be easily specified. However, if the keyword used is an idiom that is frequently used in documents in the target category (that is, the frequency of occurrence is high), it is difficult to select the desired target document from the category for that keyword. It is. In this case, the final keyword weight is calculated using a keyword calculation method based on another statistical method.

  For example, a keyword weight is calculated by the following formula using a TF * IDF weight calculator.

ここで、Ｎは文書の総数であり、ｎｔはキーワードｔを含む文書の数である。なお、上記の式は有名なRobertson/Spark-Jones式の変形である。

Here, N is the total number of documents, and nt is the number of documents including the keyword t. The above formula is a variation of the famous Robertson / Spark-Jones formula.

  As described above, the weight obtained by the TF * IDF method and the overall difference obtained as described above are input to the weight merging module (TWC) 108, and the final keyword weight is calculated.

  Next, each configuration of the search device according to the present embodiment will be described.

  First, the word extraction means (TE) 101 will be described.

  The word extraction means (TE) 101 extracts a keyword string from the search request sentence input by the user.

Specifically, the word extraction means (TE) 101
(1) Describe user search request text
(2) Perform initial filtering according to the nature of the word to remove unused words. For example, a measure and a number.
(3) Remove some noise words with reference to the stop word table. For example, there are “use” and “effect”.
(4) Reduce noise. Here, words that may cause erroneous search results are removed. Specifically, a threshold value ts is provided, and words whose appearance frequency is lower than the threshold value ts are removed.

  Specific examples will be described below.

  The user enters the next search request text.

“The present invention can automatically track and always collect sunlight at the maximum area of the light collector, and also always propagate the light to a certain position at a certain angle to enter and diverge the room. I'm happy with the entry device. The present invention provides good applicability. "
There are two typographical errors in the above document: “pleasant” (disclosure) and “provide” (provide). The former is a less frequently used word, and the latter is a relatively frequently used word.

  First, the following word string is obtained. That is, “the present invention can be automatically, tracked, always collect the light collector's maximum, area, sunlight, further propagate, to light, constant, angle, position, Indoor, approach, divergence, entrance, room, equipment, pleasant, good, applicability, offer ".

  Next, by selecting only nouns, verbs, and adjectives, the following word sequence is obtained. That is, "book, invention, automatic, tracking, can, light collector, maximum, area, sunlight, collection, constant, angle, position, propagation, indoor, approach, divergence, entrance, room, equipment, pleasure Good, applicability, offering ".

  Next, the words existing in the stop word table are removed. For example, there are “Yes” and “Done”.

  Next, since the appearance frequency of “joy” is very low, this word is removed. In this way, the following word string is obtained. That is, "book, invention, automatic, tracking, light collector, maximum, area, sunlight, collection, constant, angle, position, propagation, indoor, approach, divergence, entrance, room, equipment, good, applicability, "I will offer."

  Next, the classification selection subdivision module (CSM) 103 will be described.

  A classification selection subdivision module (CSM) 103 obtains classification information from the digital data set 110. The input to the classification selection subdivision module (CSM) 103 may be null or classification. A classification selection subdivision module (CSM) 103 determines whether or not to further subdivide the input classification. Therefore, if the digital data sets are different, the classification selection subdivision module (CSM) 103 is also different.

  Specifically, if the input to the classification selection subdivision module (CSM) 103 is null, the classification selection subdivision module (CSM) 103 determines whether a classification exists in the digital data set 110. .

  If there is a classification, the classification selection subdivision module (CSM) 103 reads the arrangement information and determines whether or not the difference information is necessary.

  When the dissimilarity information is necessary, the classification selection subdivision module (CSM) 103 outputs the classification information of the first hierarchy (the highest rank hierarchy).

  If it is determined that no classification exists in the digital data set 110, the classification selection subdivision module (CSM) 103 notifies the search system that the calculation of the degree of difference has been completed.

  On the other hand, if the input to the classification selection subdivision module (CSM) 103 is not null but classification information, the classification selection subdivision module (CSM) 103 determines whether or not a subclass exists in the classification. To do.

  If there is a sub-classification, the classification selection subdivision module (CSM) 103 reads the arrangement information and determines whether or not to further subdivide.

  When further subdividing, the classification selection subdivision module (CSM) 103 outputs the classification information of the next layer.

  If it is determined that no classification exists in the digital data set 110, the classification selection subdivision module (CSM) 103 notifies the search system that the calculation of the degree of difference has been completed.

  For example, the digital data set 110 is assumed to be a patent database compliant with IPC. In addition, the search system assumes that the difference is calculated up to the third layer.

For example, when the input to the classification selection subdivision module (CSM) 103 is N / A, the classification selection subdivision module (CSM) 103 outputs the IPC of the first layer. That is,
Section A-Daily Essentials Section B-Processing Operation; Transportation Section C-Chemical; Metallurgy Section D-Textile; Paper Section E-Fixed Structure Section F-Mechanical Engineering; Lighting; Heating; Weapons; Section-Physics H Section-Electricity 1) If the input to the Classification Selection Subdivision Module (CSM) 103 is "H Section Electric", the output is as follows.
H01 Basic electrical element H02 Electric power generation, conversion, and distribution H03 Basic electronic circuit H04 Electrical communication technology H05 Electrical technology not classified elsewhere 2) The input to the classification selection subdivision module (CSM) 103 is “H01C resistor” In some cases, this classification is a classification belonging to the third hierarchy, so the corresponding output is N / A. This indicates that there is no further classification. When such an output is received, the calculation of the difference is terminated.

  Next, the dissimilarity calculator (DPC) 104 will be described.

  The input to the dissimilarity calculator (DPC) 104 is a keyword string and a classification string, and its output is a dissimilarity value.

  Every document has its own characteristics, and most characteristics are expressed in words, which are the most basic semantic expression units. In other words, each word represents a document characteristic. All such information can be used to represent the attributes of a document, such as whether a word appears in the document, how often, where it appears, and so on.

Each keyword t in the keyword string T has a different ability to represent a classification. Here, such an expression ability is denoted as Pt. For all t _i belonging to the keyword column T and c _j belonging to the classification column C (ie, t _i εT, c _j εC), the following matrix of Pt is obtained.

ここで、

here,

ここで、ｎ_ｔｉは、キーワードｔが分類ｃ_ｉにおいて出現する回数である。
ｌ_ｊは分類ｃ_ｊの長さである（即ち、分類ｃ_ｊにおける単語の数である）。

_{Here, n ti} is the number of times a keyword t appear in the classification _{c i.}
l _j is the length of class c _j (ie, the number of words in class c _j ).

  Also,

ｌ_aveは分類の平均の長さである。

l _ave is the average length of the classification.

F _ij is the appearance frequency of a word in a classification, and represents the number of times a word appears in a certain classification.

If a word is concentrated in one classification, the value of _fij is relatively high. However, the relationship between the appearance frequency and the classification length must be considered. In the above calculation, the classification length and the average classification length were considered simultaneously.

  If a word has the same frequency of appearance in each category, the shorter the category, the higher the expressiveness of this word for that category. Parameters k and b are factors that adjust the appearance frequency and the length of the classification.

  As described above, for each keyword, a PT representing the classification expression power of the keyword is obtained as follows.

ここで、

here,

ＰＴにより、単語の現在階層に属する分類に対する相違度を推定できる。以下の式を用いて相違度を計算する。

By PT, it is possible to estimate the degree of difference of a word belonging to the current hierarchy. The dissimilarity is calculated using the following formula.

ここで、ｎは分類の数である。

Here, n is the number of classifications.

  In the above formula, it is assumed that the keywords are uniformly distributed in each category, and the target document is uniformly distributed in each category.

  The value of dp reflects the distribution difference in each classification of words. If a word appears in one or a few categories, the dp value is relatively high (if the word appears in only one category, the dp value is 1). That is, the word tends to lead the search to these classifications as a whole. On the other hand, if a word appears in all categories, the value of dp is relatively small (if the word has the same distribution in each category, the value of dp is 0, which is the minimum).

  FIG. 4 is a flowchart showing the operation of the dissimilarity calculator (DPC) 104 according to the embodiment of the present invention.

For example, it is assumed that the input keyword string and classification string are T (t ₁ , t ₂ , t ₃ ) and C (c ₁ , c ₂ , c ₃ ).

According to the data set 110, the number of words of each of the classifications c ₁ , c ₂ , c ₃ ) is 2000, 3000, and 5000, and the initialization parameters are defined as k = 2, b = 0.7. It was.

The data set 110 was searched and the following results were obtained.
t1 appears 100 times in c1, appears 300 times in c2, appears 200 times in c3,
t2 appears 70 times in c1, 500 times in c2, 1000 times in c3,
t3 appears 200 times in c1, appears 300 times in c2, and appears 500 times in c3.

  Therefore, the following matrix Pt is obtained.

従って、以下のような相違度の列が得られる。

Therefore, the following dissimilarity columns are obtained.

  DP = (0.430, 0.518, 0.0528).

  Next, the keyword selection module (TSM) 102 will be described.

  The keyword selection module (TSM) 102 mainly removes noise in the keyword string. The input to the keyword selection module (TSM) 102 is the keyword string, the corresponding dissimilarity, and the keyword frequency. The output from the keyword selection module (TSM) 102 is the selected keyword string and the corresponding dissimilarity.

  The keyword selection module (TSM) 102 removes any words whose dissimilarity is below a threshold. Here, the threshold value is (Max (dp) * parameter). Parameter is a parameter set in advance.

  Through such a filtering step, the keyword selection module (TSM) 102 outputs a new keyword string and a corresponding dissimilarity.

次に、目標文書推定手段（ＰＲＥ）１０５を説明する。

Next, the target document estimation means (PRE) 105 will be described.

  The target document estimation means (PRE) 105 calculates the probability existing in each classification of the target document. The input to the target document estimation means (PRE) 105 is a keyword string, the corresponding dissimilarity, and classification information, and the output is the probability that the target document exists in each classification.

  FIG. 5 is a flowchart showing the operation of the target document estimation means (PRE) 105 according to the embodiment of the present invention.

  A matrix Pt is obtained in the same manner as the dissimilarity calculator (DPC) 104 according to the input keyword string and classification information.

各分類は、マトリックスＰｔの一行に対応する。さらに、以下のように定義する。

Each classification corresponds to one row of the matrix Pt. Furthermore, it defines as follows.

ここで、

here,

目標文書推定手段１０５は、相違度に基づいて目標文書の各分類に属する確率を計算する。以下に定義する。ｃ_ｊ∈Ｃの場合、

The target document estimation unit 105 calculates a probability belonging to each classification of the target document based on the degree of difference. It is defined below. If c _j ∈C,

ここで、ＰＣ_ｊ ^＋はＰＣ_ｊの反転である。

Here, PC _j ⁺ is the inversion of PC _j .

_{Also, DP = (dp 1, dp} 2, ..., dp m) is a sequence of dissimilarity entered.

  For example, the following matrix is obtained according to the input.

よって、以下のＰＣ_ｊが得られる。

Therefore, the following PC _j is obtained.

次に、分類確定手段（ＣＬ）１０７を説明する。

Next, the classification determination means (CL) 107 will be described.

Users despite the search request input belongs to a classification c _k, there is a part of the keywords extracted is tilted more classification c _q. Classification such as c _q is called noise classification.

These noise classifications are removed to increase the final accuracy of the search. Specifically, the classification with a value of pc smaller than tc _k shown below is removed.

ここで、Ｍａｘ（ＰＱ_ｋ’）は最大値であり、ｐｑ_ｉｋは目標文書の分類ｃ_ｋに出現する確率である。なお、ｐｑ_ｉｋは上の階級で計算される。ただ、現階級は第１階級であれば、ｐｑ＝１／ｎである。ここで、ｎは現在の分類の分類番号である。

Here, Max (PQ _k ′) is the maximum value, and pq _ik is the probability of appearing in the target document classification _ck . Note that pq _ik is calculated in the upper class. However, if the current class is the first class, pq = 1 / n. Here, n is the classification number of the current classification.

The inputs to the classification determination means (CL) 107 are the PQ column and the classification column, and the output is the selected classification C = (c ₁ , c ₂ ,..., C _q ).

  Next, the dissimilarity synthesis module (DIM) 106 will be described.

  The dissimilarity synthesis module (DIM) 106 synthesizes dissimilarities of different classes.

The input to the dissimilarity synthesis module (DIM) 106 is the dissimilarity and the probability in each classification of the target document, and the output is the dissimilarity tw _i after synthesis.

ここで、ｋ１、ｋ２、ｋ３はパラメータである。

Here, k1, k2, and k3 are parameters.

tw _i is the weight of the i-th keyword.

dp _i is the degree of difference in the classification of the first class of the i-th keyword.

dp _ti is the degree of difference in the Ct subclass of the i-th keyword.

dq _t is a probability that the target document belongs to the classification Ct.

dp _tut is the degree of difference in the Ct subclass of the i-th keyword.

dq _tu is the probability that the target document belongs to the sub-class of Ct.

  Next, the weight merging module 108 will be described.

  As described above, the classification of the target document can be easily specified based on the degree of difference. However, if the keyword is frequently used in the target document, it cannot be further searched with this keyword. In this case, it is necessary to use the method of the present invention in combination with another global weight calculation method.

  Inputs to the weight merging module 108 are keyword strings, corresponding dissimilarities, and weights calculated by a global weight calculation method. The output is the keyword weight. Here, it calculates using the following formula | equation.

以上、本発明の好ましい実施形態を説明したが、本発明はこれらの実施形態に限定されず、本発明の趣旨を離脱しない限り、本発明に対するあらゆる変更は本発明の範囲に属する。

As mentioned above, although preferable embodiment of this invention was described, this invention is not limited to these embodiment, Unless it leaves | separates the meaning of this invention, all the changes with respect to this invention belong to the scope of the present invention.

1 is a block diagram illustrating a configuration of a document search apparatus 100 according to an embodiment of the present invention. 6 is a flowchart illustrating a document search method according to an embodiment of the present invention. It is a flowchart which shows operation which searches the target document based on the Example of this invention. It is a flowchart which shows operation | movement of the difference calculator (DPC) 104 which concerns on the Example of this invention. It is a flowchart which shows operation | movement of the target document estimation means (PRE) 105 which concerns on embodiment of this invention.

Explanation of symbols

100 document retrieval apparatus 101 word extraction means (TE)
102 Keyword selection module (TSM)
103 Classification Selection Subdivision Module (CSM)
104 Dissimilarity calculator (DPC)
105 Target document estimation means (PRE)
106 Dissimilarity synthesis module (DIM)
107 Classification confirmation means (CL)
108 Weight Merge Module (TWC)
109 Search Engine 110 Digital Data Set

Claims (22)

A method for retrieving a predetermined target document from a digital data set having a hierarchical classification structure obtained by hierarchically classifying a set of digital data of a plurality of documents based on classification information of a predetermined number of layers,
(A) extracting a keyword string including at least one keyword from a search request sentence input by a user;
(B) obtaining classification information of the current hierarchy from the digital data set;
(C) calculating the degree of difference of each keyword in the keyword string for each classification belonging to the current hierarchy;
(D) calculating a probability that the target document exists in each classification belonging to the current hierarchy based on the degree of difference;
(E1) If the number of processed layers is smaller than the predetermined number, the next layer is set as the current layer, and the above steps (c), (d), and (e1) are executed again.
(E2) When the number of processed hierarchies is equal to or greater than the predetermined number, the calculated difference between each class belonging to each hierarchy and the probability that the target document exists in each class belonging to each hierarchy are combined And (f) searching for the target document in accordance with the composite difference. The minimum number of layers in the digital data set is 1, and the minimum number of classifications in each layer is 1.
The document search method according to claim 1. The document search method according to claim 1 or 2, wherein in step (e2), all the degrees of difference calculated in step (c) are combined. The document search method according to any one of claims 1 to 3, further comprising combining global dissimilarities in the step (e2). The document search method according to any one of claims 1 to 4, wherein in the step (c), the degree of difference is calculated based on a distinctiveness for each classification of the keyword. The document search method according to claim 5, wherein the distinctiveness for each classification of the keyword is calculated by the expressive power for each classification of the keyword. The document search method according to claim 6, wherein the power of expression for each classification of the keyword is calculated based on an appearance frequency in each classification of the keyword and an attribute of each classification. In step (e1), before making the next hierarchy the current hierarchy,
The document search method according to claim 1, wherein (g) a noise keyword is removed from the keyword string of the current hierarchy based on the keyword string of the current hierarchy, the corresponding dissimilarity, and the appearance frequency to obtain a keyword string of the next hierarchy. . The document search method according to claim 4, wherein the global dissimilarity is calculated by a statistical method. In step (e1), before making the next hierarchy the current hierarchy,
The document search method according to claim 1, wherein (h) noise classification is removed based on a probability existing in each classification belonging to the current hierarchy of the target document, and classification information of the next hierarchy is used. The document search method according to any one of claims 1 to 10, wherein the keyword includes a word or a phrase. An apparatus for retrieving a predetermined target document from a digital data set having a hierarchical classification structure obtained by hierarchically classifying a set of digital data of a plurality of documents, based on classification information of a predetermined number of layers,
Word extraction means for extracting a keyword string including at least one keyword from a search request sentence input by a user;
Classification selection subdivision means for obtaining classification information of the current hierarchy from the digital data set;
A dissimilarity calculating means for calculating a dissimilarity of each keyword in the keyword string for each classification belonging to the current hierarchy;
Target document estimation means for calculating a probability that the target document exists in each classification belonging to the current hierarchy based on the degree of difference;
A dissimilarity synthesizing unit that synthesizes the dissimilarity with respect to each classification and the probability that the target document exists in each classification belonging to each hierarchy, and obtains the synthetic dissimilarity of each keyword;
A search engine that searches the target document according to the composite dissimilarity, and
When the number of processed hierarchies is smaller than the predetermined number, the dissimilarity calculating means and the target document estimating means sequentially calculate the dissimilarity and the probability for each hierarchy,
When the number of processed hierarchies is equal to or greater than the predetermined number, the difference degree synthesizing unit obtains the degree of composition difference, and the search engine performs a search. The minimum number of layers in the digital data set is 1, and the minimum number of classifications in each layer is 1.
The document search device according to claim 12. The document search apparatus according to claim 12 or 13, wherein the dissimilarity synthesis unit synthesizes all the dissimilarities calculated by the dissimilarity calculation unit. The document search device according to any one of claims 12 to 14, wherein the dissimilarity synthesizing unit further synthesizes a global dissimilarity. The document search device according to any one of claims 12 to 15, wherein the dissimilarity calculation means calculates the dissimilarity based on a distinctiveness for each classification of the keyword. The document search apparatus according to claim 16, wherein the distinctiveness for each classification of the keyword is calculated by the expressive power for each classification of the keyword. The document search apparatus according to claim 17, wherein the expression power for each classification of the keyword is calculated based on an appearance frequency in each classification of the keyword and an attribute of each classification. The keyword selection unit according to claim 12, further comprising: a keyword selection unit that removes a noise keyword from the keyword string of the current hierarchy according to the keyword string of the current hierarchy, the corresponding difference degree, and the appearance frequency, and sets the keyword keyword of the next hierarchy. Document retrieval device. The document search apparatus according to claim 15, wherein the global dissimilarity is calculated by a statistical method. 13. The document search apparatus according to claim 12, further comprising: a class determination unit that removes a noise class based on a probability existing in each class belonging to the current hierarchy of the target document and sets it as class information of a next hierarchy. The document search device according to any one of claims 12 to 21, wherein the keyword includes a word or a phrase.

Images